Oct 31 01:01:14.139079 ip-10-0-133-197 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Oct 31 01:01:14.139116 ip-10-0-133-197 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Oct 31 01:01:14.139125 ip-10-0-133-197 systemd[1]: kubelet.service: Failed with result 'resources'.
Oct 31 01:01:14.139423 ip-10-0-133-197 systemd[1]: Failed to start Kubernetes Kubelet.
Oct 31 01:01:24.388377 ip-10-0-133-197 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Oct 31 01:01:24.388396 ip-10-0-133-197 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot 734896feba5a4878a9944b46528ea61c --
Oct 31 01:03:08.194570 ip-10-0-133-197 systemd[1]: Starting Kubernetes Kubelet...
Oct 31 01:03:08.619591 ip-10-0-133-197 kubenswrapper[2578]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 31 01:03:08.619591 ip-10-0-133-197 kubenswrapper[2578]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Oct 31 01:03:08.619591 ip-10-0-133-197 kubenswrapper[2578]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 31 01:03:08.619591 ip-10-0-133-197 kubenswrapper[2578]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 31 01:03:08.623168 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.623084    2578 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 01:03:08.623168 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.623169    2578 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 01:03:08.623242 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.623174    2578 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 01:03:08.623242 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.623178    2578 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 01:03:08.623242 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.623181    2578 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 01:03:08.623242 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.623184    2578 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 01:03:08.623242 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.623187    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 01:03:08.623242 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.623190    2578 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 01:03:08.623242 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.623193    2578 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 01:03:08.623242 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.623196    2578 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 01:03:08.623242 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.623198    2578 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 01:03:08.623242 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.623201    2578 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 01:03:08.623242 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.623204    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 01:03:08.623242 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.623206    2578 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 01:03:08.623242 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.623209    2578 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 01:03:08.623242 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.623212    2578 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 01:03:08.623242 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.623214    2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 01:03:08.623242 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.623217    2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 01:03:08.623242 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.623219    2578 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 01:03:08.623242 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.623222    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 01:03:08.623242 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.623224    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 01:03:08.623705 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.623227    2578 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 01:03:08.623705 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.623229    2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 01:03:08.623705 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.623232    2578 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 01:03:08.623705 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.623243    2578 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 01:03:08.623705 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.623246    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 01:03:08.623705 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.623249    2578 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 01:03:08.623705 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.623252    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 01:03:08.623705 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.623255    2578 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 01:03:08.623705 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.623258    2578 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 01:03:08.623705 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.623260    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 01:03:08.623705 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.623263    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 01:03:08.623705 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.623266    2578 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 01:03:08.623705 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.623268    2578 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 01:03:08.623705 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.623271    2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 01:03:08.623705 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.623273    2578 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 01:03:08.623705 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.623276    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 01:03:08.623705 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.623278    2578 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 01:03:08.623705 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.623281    2578 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 01:03:08.623705 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.623284    2578 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 01:03:08.624150 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.623287    2578 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 01:03:08.624150 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.623289    2578 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 01:03:08.624150 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.623292    2578 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 01:03:08.624150 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.623294    2578 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 01:03:08.624150 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.623297    2578 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 01:03:08.624150 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.623299    2578 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 01:03:08.624150 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.623302    2578 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 01:03:08.624150 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.623304    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 01:03:08.624150 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.623307    2578 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 01:03:08.624150 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.623309    2578 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 01:03:08.624150 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.623326    2578 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 01:03:08.624150 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.623329    2578 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 01:03:08.624150 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.623332    2578 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 01:03:08.624150 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.623334    2578 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 01:03:08.624150 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.623338    2578 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 01:03:08.624150 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.623341    2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 01:03:08.624150 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.623343    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 01:03:08.624150 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.623346    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 01:03:08.624150 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.623349    2578 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 01:03:08.624150 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.623351    2578 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 01:03:08.624634 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.623354    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 01:03:08.624634 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.623357    2578 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 01:03:08.624634 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.623359    2578 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 01:03:08.624634 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.623362    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 01:03:08.624634 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.623364    2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 01:03:08.624634 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.623367    2578 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 01:03:08.624634 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.623370    2578 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 01:03:08.624634 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.623372    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 01:03:08.624634 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.623375    2578 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 01:03:08.624634 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.623378    2578 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 01:03:08.624634 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.623380    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 01:03:08.624634 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.623383    2578 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 01:03:08.624634 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.623385    2578 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 01:03:08.624634 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.623390    2578 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 01:03:08.624634 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.623394    2578 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 01:03:08.624634 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.623397    2578 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 01:03:08.624634 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.623400    2578 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 01:03:08.624634 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.623403    2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 01:03:08.624634 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.623406    2578 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 01:03:08.625111 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.623408    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 01:03:08.625111 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.623411    2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 01:03:08.625111 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.623414    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 01:03:08.625111 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.623417    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 01:03:08.625111 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.623419    2578 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 01:03:08.625111 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.623423    2578 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 01:03:08.625111 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.623426    2578 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 01:03:08.625111 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.623429    2578 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 01:03:08.625111 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.623432    2578 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 01:03:08.625111 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.623435    2578 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 01:03:08.625111 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.623437    2578 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 01:03:08.625111 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.623440    2578 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 01:03:08.625111 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.623443    2578 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 01:03:08.625111 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.623445    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 01:03:08.625111 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.623448    2578 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 01:03:08.625111 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.623452    2578 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 01:03:08.625111 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.623457    2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 01:03:08.625111 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.623460    2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 01:03:08.625111 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.623463    2578 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 01:03:08.625587 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.623465    2578 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 01:03:08.625587 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.623468    2578 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 01:03:08.625587 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.623471    2578 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 01:03:08.625587 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.623473    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 01:03:08.625587 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.623476    2578 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 01:03:08.625587 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.623479    2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 01:03:08.625587 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.623481    2578 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 01:03:08.625587 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.623484    2578 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 01:03:08.625587 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.624284    2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 01:03:08.625587 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.624291    2578 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 01:03:08.625587 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.624294    2578 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 01:03:08.625587 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.624297    2578 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 01:03:08.625587 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.624300    2578 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 01:03:08.625587 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.624303    2578 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 01:03:08.625587 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.624306    2578 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 01:03:08.625587 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.624309    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 01:03:08.625587 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.624312    2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 01:03:08.625587 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.624330    2578 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 01:03:08.625587 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.624333    2578 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 01:03:08.625587 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.624336    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 01:03:08.627309 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.624339    2578 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 01:03:08.627309 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.624342    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 01:03:08.627309 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.624345    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 01:03:08.627309 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.624347    2578 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 01:03:08.627309 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.624350    2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 01:03:08.627309 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.624353    2578 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 01:03:08.627309 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.624356    2578 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 01:03:08.627309 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.624358    2578 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 01:03:08.627309 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.624361    2578 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 01:03:08.627309 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.624363    2578 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 01:03:08.627309 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.624366    2578 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 01:03:08.627309 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.624369    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 01:03:08.627309 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.624371    2578 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 01:03:08.627309 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.624374    2578 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 01:03:08.627309 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.624376    2578 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 01:03:08.627309 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.624379    2578 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 01:03:08.627309 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.624381    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 01:03:08.627309 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.624385    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 01:03:08.627309 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.624388    2578 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 01:03:08.627309 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.624390    2578 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 01:03:08.627799 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.624393    2578 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 01:03:08.627799 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.624395    2578 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 01:03:08.627799 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.624397    2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 01:03:08.627799 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.624400    2578 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 01:03:08.627799 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.624403    2578 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 01:03:08.627799 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.624406    2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 01:03:08.627799 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.624412    2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 01:03:08.627799 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.624415    2578 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 01:03:08.627799 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.624418    2578 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 01:03:08.627799 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.624421    2578 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 01:03:08.627799 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.624423    2578 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 01:03:08.627799 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.624426    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 01:03:08.627799 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.624429    2578 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 01:03:08.627799 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.624431    2578 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 01:03:08.627799 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.624434    2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 01:03:08.627799 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.624437    2578 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 01:03:08.627799 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.624439    2578 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 01:03:08.627799 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.624442    2578 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 01:03:08.627799 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.624444    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 01:03:08.628248 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.624447    2578 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 01:03:08.628248 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.624449    2578 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 01:03:08.628248 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.624452    2578 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 01:03:08.628248 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.624454    2578 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 01:03:08.628248 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.624458    2578 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 01:03:08.628248 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.624462    2578 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 01:03:08.628248 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.624466    2578 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 01:03:08.628248 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.624469    2578 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 01:03:08.628248 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.624472    2578 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 01:03:08.628248 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.624475    2578 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 01:03:08.628248 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.624478    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 01:03:08.628248 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.624481    2578 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 01:03:08.628248 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.624483    2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 01:03:08.628248 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.624486    2578 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 01:03:08.628248 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.624488    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 01:03:08.628248 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.624490    2578 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 01:03:08.628248 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.624493    2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 01:03:08.628248 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.624496    2578 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 01:03:08.628248 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.624499    2578 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 01:03:08.628248 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.624501    2578 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 01:03:08.628773 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.624505    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 01:03:08.628773 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.624507    2578 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 01:03:08.628773 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.624510    2578 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 01:03:08.628773 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.624512    2578 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 01:03:08.628773 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.624515    2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 01:03:08.628773 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.624518    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 01:03:08.628773 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.624520    2578 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 01:03:08.628773 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.624523    2578 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 01:03:08.628773 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.624525    2578 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 01:03:08.628773 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.624528    2578 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 01:03:08.628773 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.624530    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 01:03:08.628773 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.624533    2578 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 01:03:08.628773 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.624536    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 01:03:08.628773 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.624538    2578 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 01:03:08.628773 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.624542    2578 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 01:03:08.628773 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.624546    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 01:03:08.628773 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.624550    2578 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 01:03:08.628773 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.624553    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 01:03:08.628773 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.624555    2578 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 01:03:08.629214 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.624558    2578 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 01:03:08.629214 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.624561    2578 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 01:03:08.629214 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.624563    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 01:03:08.629214 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.624566    2578 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 01:03:08.629214 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.624569    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 01:03:08.629214 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.624572    2578 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 01:03:08.629214 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.624574    2578 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 01:03:08.629214 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.624577    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 01:03:08.629214 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.624579    2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 01:03:08.629214 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.624582    2578 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 01:03:08.629214 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.624585    2578 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 01:03:08.629214 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.624587    2578 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 01:03:08.629214 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.624590    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 01:03:08.629214 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.624592    2578 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 01:03:08.629214 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.624596    2578 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 01:03:08.629214 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.624598    2578 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 01:03:08.629214 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.625096    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 01:03:08.629214 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.625102    2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 01:03:08.629214 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.625104    2578 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 01:03:08.629678 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.625107    2578 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 01:03:08.629678 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.625112    2578 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 01:03:08.629678 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.625117    2578 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 01:03:08.629678 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.625120    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 01:03:08.629678 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.625123    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 01:03:08.629678 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.625126    2578 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 01:03:08.629678 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.625130    2578 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 01:03:08.629678 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.625133    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 01:03:08.629678 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.625136    2578 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 01:03:08.629678 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.625138    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 01:03:08.629678 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.625141    2578 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 01:03:08.629678 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.625144    2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 01:03:08.629678 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.625147    2578 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 01:03:08.629678 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.625149    2578 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 01:03:08.629678 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.625152    2578 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 01:03:08.629678 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.625154    2578 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 01:03:08.629678 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.625157    2578 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 01:03:08.629678 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.625160    2578 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 01:03:08.630111 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.625162    2578 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 01:03:08.630111 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.625165    2578 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 01:03:08.630111 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.625168    2578 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 01:03:08.630111 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.625171    2578 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 01:03:08.630111 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.625174    2578 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 01:03:08.630111 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.625176    2578 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 01:03:08.630111 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.625179    2578 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 01:03:08.630111 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.625182    2578 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 01:03:08.630111 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.625185    2578 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 01:03:08.630111 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.625188    2578 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 01:03:08.630111 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.625191    2578 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 01:03:08.630111 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.625194    2578 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 01:03:08.630111 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.625196    2578 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 01:03:08.630111 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.625199    2578 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 01:03:08.630111 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.625202    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 01:03:08.630111 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.625204    2578 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 01:03:08.630111 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.625207    2578 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 01:03:08.630111 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.625209    2578 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 01:03:08.630111 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.625212    2578 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 01:03:08.630574 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.625214    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 01:03:08.630574 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.625217    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 01:03:08.630574 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.625219    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 01:03:08.630574 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.625222    2578 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 01:03:08.630574 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.625224    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 01:03:08.630574 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.625227    2578 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 01:03:08.630574 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.625229    2578 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 01:03:08.630574 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.625232    2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 01:03:08.630574 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.625234    2578 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 01:03:08.630574 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.625237    2578 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 01:03:08.630574 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.625239    2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 01:03:08.630574 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.625244    2578 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 01:03:08.630574 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.625247    2578 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 01:03:08.630574 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.625250    2578 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 01:03:08.630574 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.625253    2578 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 01:03:08.630574 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.625255    2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 01:03:08.630574 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.625258    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 01:03:08.630574 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.625261    2578 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 01:03:08.630574 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.625263    2578 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 01:03:08.630574 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.625266    2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 01:03:08.631045 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.625268    2578 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 01:03:08.631045 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.625271    2578 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 01:03:08.631045 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.625273    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 01:03:08.631045 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.625276    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 01:03:08.631045 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.625279    2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 01:03:08.631045 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.625282    2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 01:03:08.631045 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.625284    2578 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 01:03:08.631045 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.625287    2578 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 01:03:08.631045 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.625289    2578 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 01:03:08.631045 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.625291    2578 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 01:03:08.631045 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.625294    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 01:03:08.631045 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.625297    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 01:03:08.631045 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.625299    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 01:03:08.631045 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.625302    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 01:03:08.631045 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.625304    2578 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 01:03:08.631045 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.625307    2578 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 01:03:08.631045 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.625309    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 01:03:08.631045 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.625327    2578 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 01:03:08.631045 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.625330    2578 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 01:03:08.631045 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.625333    2578 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 01:03:08.631530 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.625335    2578 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 01:03:08.631530 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.625338    2578 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 01:03:08.631530 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.625340    2578 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 01:03:08.631530 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.625344    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 01:03:08.631530 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.625347    2578 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 01:03:08.631530 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.625350    2578 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 01:03:08.631530 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.625352    2578 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 01:03:08.631530 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.625355    2578 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 01:03:08.631530 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.625357    2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 01:03:08.631530 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.625360    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 01:03:08.631530 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.625362    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 01:03:08.631530 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.625365    2578 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 01:03:08.631530 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.625368    2578 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 01:03:08.631530 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.625371    2578 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 01:03:08.631530 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.625373    2578 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 01:03:08.631530 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.625375    2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 01:03:08.631530 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.625379    2578 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 01:03:08.631530 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.625381    2578 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 01:03:08.631530 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.625384    2578 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 01:03:08.631530 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.625386    2578 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 01:03:08.632000 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.625388    2578 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 01:03:08.632000 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.625391    2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 01:03:08.632000 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.625393    2578 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 01:03:08.632000 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.625396    2578 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 01:03:08.632000 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.625398    2578 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 01:03:08.632000 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.625402    2578 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 01:03:08.632000 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:08.625490    2578 flags.go:64] FLAG: --address="0.0.0.0"
Oct 31 01:03:08.632000 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:08.625497    2578 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Oct 31 01:03:08.632000 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:08.625503    2578 flags.go:64] FLAG: --anonymous-auth="true"
Oct 31 01:03:08.632000 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:08.625508    2578 flags.go:64] FLAG: --application-metrics-count-limit="100"
Oct 31 01:03:08.632000 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:08.625512    2578 flags.go:64] FLAG: --authentication-token-webhook="false"
Oct 31 01:03:08.632000 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:08.625516    2578 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Oct 31 01:03:08.632000 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:08.625523    2578 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Oct 31 01:03:08.632000 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:08.625527    2578 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Oct 31 01:03:08.632000 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:08.625530    2578 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Oct 31 01:03:08.632000 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:08.625534    2578 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Oct 31 01:03:08.632000 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:08.625538    2578 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Oct 31 01:03:08.632000 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:08.625541    2578 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Oct 31 01:03:08.632000 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:08.625544    2578 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Oct 31 01:03:08.632000 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:08.625546    2578 flags.go:64] FLAG: --cgroup-root=""
Oct 31 01:03:08.632000 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:08.625549    2578 flags.go:64] FLAG: --cgroups-per-qos="true"
Oct 31 01:03:08.632000 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:08.625552    2578 flags.go:64] FLAG: --client-ca-file=""
Oct 31 01:03:08.632000 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:08.625555    2578 flags.go:64] FLAG: --cloud-provider="external"
Oct 31 01:03:08.632545 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:08.625558    2578 flags.go:64] FLAG: --cluster-dns="[]"
Oct 31 01:03:08.632545 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:08.625562    2578 flags.go:64] FLAG: --cluster-domain=""
Oct 31 01:03:08.632545 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:08.625564    2578 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Oct 31 01:03:08.632545 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:08.625567    2578 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Oct 31 01:03:08.632545 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:08.625570    2578 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Oct 31 01:03:08.632545 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:08.625573    2578 flags.go:64] FLAG: --container-log-max-files="5"
Oct 31 01:03:08.632545 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:08.625578    2578 flags.go:64] FLAG: --container-log-max-size="10Mi"
Oct 31 01:03:08.632545 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:08.625581    2578 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Oct 31 01:03:08.632545 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:08.625584    2578 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Oct 31 01:03:08.632545 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:08.625587    2578 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Oct 31 01:03:08.632545 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:08.625591    2578 flags.go:64] FLAG: --contention-profiling="false"
Oct 31 01:03:08.632545 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:08.625594    2578 flags.go:64] FLAG: --cpu-cfs-quota="true"
Oct 31 01:03:08.632545 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:08.625597    2578 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Oct 31 01:03:08.632545 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:08.625600    2578 flags.go:64] FLAG: --cpu-manager-policy="none"
Oct 31 01:03:08.632545 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:08.625603    2578 flags.go:64] FLAG: --cpu-manager-policy-options=""
Oct 31 01:03:08.632545 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:08.625607    2578 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Oct 31 01:03:08.632545 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:08.625610    2578 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Oct 31 01:03:08.632545 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:08.625613    2578 flags.go:64] FLAG: --enable-debugging-handlers="true"
Oct 31 01:03:08.632545 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:08.625615    2578 flags.go:64] FLAG: --enable-load-reader="false"
Oct 31 01:03:08.632545 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:08.625618    2578 flags.go:64] FLAG: --enable-server="true"
Oct 31 01:03:08.632545 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:08.625621    2578 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Oct 31 01:03:08.632545 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:08.625626    2578 flags.go:64] FLAG: --event-burst="100"
Oct 31 01:03:08.632545 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:08.625630    2578 flags.go:64] FLAG: --event-qps="50"
Oct 31 01:03:08.632545 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:08.625633    2578 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Oct 31 01:03:08.633149 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:08.625636    2578 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Oct 31 01:03:08.633149 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:08.625639    2578 flags.go:64] FLAG: --eviction-hard=""
Oct 31 01:03:08.633149 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:08.625644    2578 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Oct 31 01:03:08.633149 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:08.625647    2578 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Oct 31 01:03:08.633149 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:08.625650    2578 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Oct 31 01:03:08.633149 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:08.625652    2578 flags.go:64] FLAG: --eviction-soft=""
Oct 31 01:03:08.633149 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:08.625656    2578 flags.go:64] FLAG: --eviction-soft-grace-period=""
Oct 31 01:03:08.633149 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:08.625659    2578 flags.go:64] FLAG: --exit-on-lock-contention="false"
Oct 31 01:03:08.633149 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:08.625662    2578 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Oct 31 01:03:08.633149 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:08.625665    2578 flags.go:64] FLAG: --experimental-mounter-path=""
Oct 31 01:03:08.633149 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:08.625668    2578 flags.go:64] FLAG: --fail-cgroupv1="true"
Oct 31 01:03:08.633149 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:08.625671    2578 flags.go:64] FLAG: --fail-swap-on="true"
Oct 31 01:03:08.633149 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:08.625673    2578 flags.go:64] FLAG: --feature-gates=""
Oct 31 01:03:08.633149 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:08.625678    2578 flags.go:64] FLAG: --file-check-frequency="20s"
Oct 31 01:03:08.633149 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:08.625681    2578 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Oct 31 01:03:08.633149 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:08.625684    2578 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Oct 31 01:03:08.633149 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:08.625687    2578 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Oct 31 01:03:08.633149 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:08.625690    2578 flags.go:64] FLAG: --healthz-port="10248"
Oct 31 01:03:08.633149 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:08.625693    2578 flags.go:64] FLAG: --help="false"
Oct 31 01:03:08.633149 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:08.625696    2578 flags.go:64] FLAG: --hostname-override="ip-10-0-133-197.us-west-1.compute.internal"
Oct 31 01:03:08.633149 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:08.625699    2578 flags.go:64] FLAG: --housekeeping-interval="10s"
Oct 31 01:03:08.633149 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:08.625702    2578 flags.go:64] FLAG: --http-check-frequency="20s"
Oct 31 01:03:08.633149 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:08.625705    2578 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Oct 31 01:03:08.633708 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:08.625708    2578 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Oct 31 01:03:08.633708 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:08.625711    2578 flags.go:64] FLAG: --image-gc-high-threshold="85"
Oct 31 01:03:08.633708 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:08.625714    2578 flags.go:64] FLAG: --image-gc-low-threshold="80"
Oct 31 01:03:08.633708 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:08.625717    2578 flags.go:64] FLAG: --image-service-endpoint=""
Oct 31 01:03:08.633708 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:08.625727    2578 flags.go:64] FLAG: --kernel-memcg-notification="false"
Oct 31 01:03:08.633708 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:08.625731    2578 flags.go:64] FLAG: --kube-api-burst="100"
Oct 31 01:03:08.633708 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:08.625734    2578 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Oct 31 01:03:08.633708 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:08.625736    2578 flags.go:64] FLAG: --kube-api-qps="50"
Oct 31 01:03:08.633708 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:08.625740    2578 flags.go:64] FLAG: --kube-reserved=""
Oct 31 01:03:08.633708 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:08.625743    2578 flags.go:64] FLAG: --kube-reserved-cgroup=""
Oct 31 01:03:08.633708 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:08.625746    2578 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Oct 31 01:03:08.633708 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:08.625749    2578 flags.go:64] FLAG: --kubelet-cgroups=""
Oct 31 01:03:08.633708 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:08.625752    2578 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Oct 31 01:03:08.633708 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:08.625755    2578 flags.go:64] FLAG: --lock-file=""
Oct 31 01:03:08.633708 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:08.625757    2578 flags.go:64] FLAG: --log-cadvisor-usage="false"
Oct 31 01:03:08.633708 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:08.625760    2578 flags.go:64] FLAG: --log-flush-frequency="5s"
Oct 31 01:03:08.633708 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:08.625763    2578 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Oct 31 01:03:08.633708 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:08.625768    2578 flags.go:64] FLAG: --log-json-split-stream="false"
Oct 31 01:03:08.633708 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:08.625772    2578 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Oct 31 01:03:08.633708 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:08.625775    2578 flags.go:64] FLAG: --log-text-split-stream="false"
Oct 31 01:03:08.633708 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:08.625777    2578 flags.go:64] FLAG: --logging-format="text"
Oct 31 01:03:08.633708 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:08.625781    2578 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Oct 31 01:03:08.633708 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:08.625784    2578 flags.go:64] FLAG: --make-iptables-util-chains="true"
Oct 31 01:03:08.633708 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:08.625787    2578 flags.go:64] FLAG: --manifest-url=""
Oct 31 01:03:08.634286 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:08.625790    2578 flags.go:64] FLAG: --manifest-url-header=""
Oct 31 01:03:08.634286 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:08.625795    2578 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Oct 31 01:03:08.634286 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:08.625798    2578 flags.go:64] FLAG: --max-open-files="1000000"
Oct 31 01:03:08.634286 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:08.625802    2578 flags.go:64] FLAG: --max-pods="110"
Oct 31 01:03:08.634286 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:08.625805    2578 flags.go:64] FLAG: --maximum-dead-containers="-1"
Oct 31 01:03:08.634286 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:08.625808    2578 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Oct 31 01:03:08.634286 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:08.625811    2578 flags.go:64] FLAG: --memory-manager-policy="None"
Oct 31 01:03:08.634286 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:08.625814    2578 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Oct 31 01:03:08.634286 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:08.625816    2578 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Oct 31 01:03:08.634286 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:08.625819    2578 flags.go:64] FLAG: --node-ip="0.0.0.0"
Oct 31 01:03:08.634286 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:08.625822    2578 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Oct 31 01:03:08.634286 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:08.625828    2578 flags.go:64] FLAG: --node-status-max-images="50"
Oct 31 01:03:08.634286 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:08.625831    2578 flags.go:64] FLAG: --node-status-update-frequency="10s"
Oct 31 01:03:08.634286 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:08.625834    2578 flags.go:64] FLAG: --oom-score-adj="-999"
Oct 31 01:03:08.634286 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:08.625837    2578 flags.go:64] FLAG: --pod-cidr=""
Oct 31 01:03:08.634286 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:08.625840    2578 flags.go:64] FLAG: --pod-manifest-path=""
Oct 31 01:03:08.634286 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:08.625845    2578 flags.go:64] FLAG: --pod-max-pids="-1"
Oct 31 01:03:08.634286 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:08.625848    2578 flags.go:64] FLAG: --pods-per-core="0"
Oct 31 01:03:08.634286 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:08.625851    2578 flags.go:64] FLAG: --port="10250"
Oct 31 01:03:08.634286 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:08.625854    2578 flags.go:64] FLAG: --protect-kernel-defaults="false"
Oct 31 01:03:08.634286 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:08.625857    2578 flags.go:64] FLAG: --provider-id="aws:///us-west-1a/i-0540feba47811ea65"
Oct 31 01:03:08.634286 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:08.625859    2578 flags.go:64] FLAG: --qos-reserved=""
Oct 31 01:03:08.634286 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:08.625863    2578 flags.go:64] FLAG: --read-only-port="10255"
Oct 31 01:03:08.634286 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:08.625866    2578 flags.go:64] FLAG: --register-node="true"
Oct 31 01:03:08.634286 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:08.625868    2578 flags.go:64] FLAG: --register-with-taints=""
Oct 31 01:03:08.634286 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:08.625872    2578 flags.go:64] FLAG: --registry-burst="10"
Oct 31 01:03:08.634916 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:08.625875    2578 flags.go:64] FLAG: --registry-qps="5"
Oct 31 01:03:08.634916 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:08.625877    2578 flags.go:64] FLAG: --reserved-cpus=""
Oct 31 01:03:08.634916 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:08.625880    2578 flags.go:64] FLAG: --reserved-memory=""
Oct 31 01:03:08.634916 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:08.625884    2578 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Oct 31 01:03:08.634916 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:08.625887    2578 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Oct 31 01:03:08.634916 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:08.625890    2578 flags.go:64] FLAG: --rotate-certificates="false"
Oct 31 01:03:08.634916 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:08.625893    2578 flags.go:64] FLAG: --rotate-server-certificates="false"
Oct 31 01:03:08.634916 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:08.625896    2578 flags.go:64] FLAG: --runonce="false"
Oct 31 01:03:08.634916 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:08.625899    2578 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Oct 31 01:03:08.634916 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:08.625902    2578 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Oct 31 01:03:08.634916 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:08.625905    2578 flags.go:64] FLAG: --seccomp-default="false"
Oct 31 01:03:08.634916 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:08.625908    2578 flags.go:64] FLAG: --serialize-image-pulls="true"
Oct 31 01:03:08.634916 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:08.625911    2578 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Oct 31 01:03:08.634916 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:08.625914    2578 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Oct 31 01:03:08.634916 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:08.625917    2578 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Oct 31 01:03:08.634916 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:08.625919    2578 flags.go:64] FLAG: --storage-driver-password="root"
Oct 31 01:03:08.634916 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:08.625922    2578 flags.go:64] FLAG: --storage-driver-secure="false"
Oct 31 01:03:08.634916 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:08.625925    2578 flags.go:64] FLAG: --storage-driver-table="stats"
Oct 31 01:03:08.634916 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:08.625928    2578 flags.go:64] FLAG: --storage-driver-user="root"
Oct 31 01:03:08.634916 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:08.625930    2578 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Oct 31 01:03:08.634916 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:08.625933    2578 flags.go:64] FLAG: --sync-frequency="1m0s"
Oct 31 01:03:08.634916 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:08.625936    2578 flags.go:64] FLAG: --system-cgroups=""
Oct 31 01:03:08.634916 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:08.625940    2578 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Oct 31 01:03:08.634916 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:08.625945    2578 flags.go:64] FLAG: --system-reserved-cgroup=""
Oct 31 01:03:08.634916 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:08.625948    2578 flags.go:64] FLAG: --tls-cert-file=""
Oct 31 01:03:08.635523 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:08.625951    2578 flags.go:64] FLAG: --tls-cipher-suites="[]"
Oct 31 01:03:08.635523 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:08.625955    2578 flags.go:64] FLAG: --tls-min-version=""
Oct 31 01:03:08.635523 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:08.625957    2578 flags.go:64] FLAG: --tls-private-key-file=""
Oct 31 01:03:08.635523 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:08.625960    2578 flags.go:64] FLAG: --topology-manager-policy="none"
Oct 31 01:03:08.635523 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:08.625963    2578 flags.go:64] FLAG: --topology-manager-policy-options=""
Oct 31 01:03:08.635523 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:08.625966    2578 flags.go:64] FLAG: --topology-manager-scope="container"
Oct 31 01:03:08.635523 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:08.625969    2578 flags.go:64] FLAG: --v="2"
Oct 31 01:03:08.635523 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:08.625973    2578 flags.go:64] FLAG: --version="false"
Oct 31 01:03:08.635523 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:08.625977    2578 flags.go:64] FLAG: --vmodule=""
Oct 31 01:03:08.635523 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:08.625981    2578 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Oct 31 01:03:08.635523 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:08.625984    2578 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Oct 31 01:03:08.635523 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.626114    2578 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 01:03:08.635523 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.626118    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 01:03:08.635523 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.626122    2578 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 01:03:08.635523 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.626125    2578 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 01:03:08.635523 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.626128    2578 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 01:03:08.635523 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.626131    2578 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 01:03:08.635523 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.626133    2578 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 01:03:08.635523 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.626136    2578 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 01:03:08.635523 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.626139    2578 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 01:03:08.635523 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.626141    2578 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 01:03:08.635523 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.626144    2578 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 01:03:08.636045 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.626146    2578 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 01:03:08.636045 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.626149    2578 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 01:03:08.636045 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.626152    2578 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 01:03:08.636045 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.626154    2578 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 01:03:08.636045 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.626157    2578 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 01:03:08.636045 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.626159    2578 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 01:03:08.636045 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.626161    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 01:03:08.636045 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.626165    2578 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 01:03:08.636045 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.626169    2578 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 01:03:08.636045 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.626172    2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 01:03:08.636045 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.626175    2578 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 01:03:08.636045 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.626177    2578 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 01:03:08.636045 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.626180    2578 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 01:03:08.636045 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.626182    2578 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 01:03:08.636045 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.626185    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 01:03:08.636045 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.626189    2578 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 01:03:08.636045 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.626193    2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 01:03:08.636045 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.626197    2578 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 01:03:08.636045 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.626200    2578 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 01:03:08.636509 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.626202    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 01:03:08.636509 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.626205    2578 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 01:03:08.636509 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.626208    2578 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 01:03:08.636509 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.626210    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 01:03:08.636509 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.626213    2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 01:03:08.636509 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.626215    2578 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 01:03:08.636509 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.626217    2578 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 01:03:08.636509 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.626220    2578 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 01:03:08.636509 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.626223    2578 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 01:03:08.636509 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.626226    2578 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 01:03:08.636509 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.626228    2578 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 01:03:08.636509 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.626231    2578 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 01:03:08.636509 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.626234    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 01:03:08.636509 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.626236    2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 01:03:08.636509 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.626238    2578 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 01:03:08.636509 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.626241    2578 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 01:03:08.636509 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.626243    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 01:03:08.636509 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.626246    2578 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 01:03:08.636509 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.626248    2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 01:03:08.636954 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.626251    2578 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 01:03:08.636954 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.626254    2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 01:03:08.636954 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.626258    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 01:03:08.636954 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.626260    2578 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 01:03:08.636954 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.626263    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 01:03:08.636954 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.626265    2578 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 01:03:08.636954 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.626268    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 01:03:08.636954 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.626271    2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 01:03:08.636954 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.626273    2578 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 01:03:08.636954 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.626276    2578 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 01:03:08.636954 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.626279    2578 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 01:03:08.636954 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.626281    2578 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 01:03:08.636954 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.626284    2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 01:03:08.636954 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.626287    2578 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 01:03:08.636954 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.626289    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 01:03:08.636954 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.626292    2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 01:03:08.636954 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.626294    2578 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 01:03:08.636954 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.626297    2578 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 01:03:08.636954 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.626299    2578 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 01:03:08.636954 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.626302    2578 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 01:03:08.637473 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.626304    2578 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 01:03:08.637473 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.626307    2578 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 01:03:08.637473 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.626310    2578 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 01:03:08.637473 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.626329    2578 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 01:03:08.637473 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.626332    2578 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 01:03:08.637473 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.626334    2578 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 01:03:08.637473 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.626337    2578 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 01:03:08.637473 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.626340    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 01:03:08.637473 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.626343    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 01:03:08.637473 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.626345    2578 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 01:03:08.637473 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.626348    2578 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 01:03:08.637473 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.626350    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 01:03:08.637473 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.626353    2578 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 01:03:08.637473 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.626357    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 01:03:08.637473 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.626360    2578 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 01:03:08.637473 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.626363    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 01:03:08.637473 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.626365    2578 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 01:03:08.637473 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.626368    2578 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 01:03:08.637473 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.626370    2578 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 01:03:08.637473 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.626373    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 01:03:08.637990 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.626376    2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 01:03:08.637990 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.626378    2578 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 01:03:08.637990 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.626381    2578 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 01:03:08.637990 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.626383    2578 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 01:03:08.637990 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.626386    2578 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 01:03:08.637990 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.626388    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 01:03:08.637990 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.626390    2578 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 01:03:08.637990 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.626393    2578 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 01:03:08.637990 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.626396    2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 01:03:08.637990 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.626398    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 01:03:08.637990 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.626402    2578 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 01:03:08.637990 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.626406    2578 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 01:03:08.637990 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.626409    2578 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 01:03:08.637990 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.626412    2578 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 01:03:08.637990 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.626415    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 01:03:08.637990 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.626418    2578 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 01:03:08.637990 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.626420    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 01:03:08.638755 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:08.626446    2578 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 31 01:03:08.638755 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.626482    2578 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 01:03:08.638755 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.626486    2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 01:03:08.638755 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.626489    2578 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 01:03:08.638755 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.626492    2578 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 01:03:08.638755 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.626494    2578 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 01:03:08.638755 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.626497    2578 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 01:03:08.638755 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.626499    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 01:03:08.638755 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.626503    2578 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 01:03:08.638755 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.626510    2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 01:03:08.638755 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.626513    2578 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 01:03:08.638755 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.626516    2578 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 01:03:08.638755 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.626519    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 01:03:08.638755 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.626522    2578 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 01:03:08.638755 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.626525    2578 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 01:03:08.639202 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.626528    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 01:03:08.639202 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.626530    2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 01:03:08.639202 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.626533    2578 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 01:03:08.639202 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.626535    2578 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 01:03:08.639202 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.626538    2578 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 01:03:08.639202 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.626541    2578 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 01:03:08.639202 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.626543    2578 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 01:03:08.639202 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.626546    2578 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 01:03:08.639202 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.626548    2578 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 01:03:08.639202 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.626551    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 01:03:08.639202 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.626553    2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 01:03:08.639202 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.626556    2578 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 01:03:08.639202 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.626559    2578 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 01:03:08.639202 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.626561    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 01:03:08.639202 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.626564    2578 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 01:03:08.639202 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.626566    2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 01:03:08.639202 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.626574    2578 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 01:03:08.639202 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.626577    2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 01:03:08.639202 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.626580    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 01:03:08.639202 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.626582    2578 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 01:03:08.639750 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.626584    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 01:03:08.639750 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.626587    2578 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 01:03:08.639750 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.626589    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 01:03:08.639750 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.626592    2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 01:03:08.639750 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.626595    2578 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 01:03:08.639750 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.626597    2578 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 01:03:08.639750 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.626602    2578 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 01:03:08.639750 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.626604    2578 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 01:03:08.639750 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.626607    2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 01:03:08.639750 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.626610    2578 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 01:03:08.639750 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.626613    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 01:03:08.639750 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.626615    2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 01:03:08.639750 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.626618    2578 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 01:03:08.639750 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.626621    2578 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 01:03:08.639750 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.626623    2578 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 01:03:08.639750 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.626626    2578 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 01:03:08.639750 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.626629    2578 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 01:03:08.639750 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.626632    2578 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 01:03:08.639750 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.626634    2578 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 01:03:08.639750 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.626637    2578 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 01:03:08.640221 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.626639    2578 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 01:03:08.640221 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.626642    2578 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 01:03:08.640221 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.626644    2578 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 01:03:08.640221 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.626647    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 01:03:08.640221 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.626649    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 01:03:08.640221 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.626652    2578 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 01:03:08.640221 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.626654    2578 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 01:03:08.640221 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.626657    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 01:03:08.640221 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.626660    2578 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 01:03:08.640221 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.626662    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 01:03:08.640221 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.626665    2578 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 01:03:08.640221 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.626668    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 01:03:08.640221 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.626670    2578 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 01:03:08.640221 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.626673    2578 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 01:03:08.640221 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.626676    2578 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 01:03:08.640221 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.626678    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 01:03:08.640221 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.626682    2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 01:03:08.640221 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.626684    2578 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 01:03:08.640221 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.626689    2578 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 01:03:08.640221 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.626691    2578 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 01:03:08.640708 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.626694    2578 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 01:03:08.640708 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.626696    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 01:03:08.640708 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.626699    2578 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 01:03:08.640708 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.626701    2578 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 01:03:08.640708 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.626704    2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 01:03:08.640708 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.626707    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 01:03:08.640708 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.626710    2578 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 01:03:08.640708 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.626714    2578 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 01:03:08.640708 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.626717    2578 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 01:03:08.640708 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.626720    2578 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 01:03:08.640708 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.626722    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 01:03:08.640708 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.626725    2578 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 01:03:08.640708 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.626728    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 01:03:08.640708 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.626730    2578 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 01:03:08.640708 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.626732    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 01:03:08.640708 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.626735    2578 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 01:03:08.640708 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.626738    2578 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 01:03:08.640708 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.626740    2578 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 01:03:08.640708 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.626743    2578 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 01:03:08.640708 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.626745    2578 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 01:03:08.641165 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.626748    2578 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 01:03:08.641165 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.626750    2578 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 01:03:08.641165 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.626753    2578 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 01:03:08.641165 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.626756    2578 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 01:03:08.641165 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.626758    2578 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 01:03:08.641165 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.626760    2578 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 01:03:08.641165 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.626764    2578 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 01:03:08.641165 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.626767    2578 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 01:03:08.641165 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.626769    2578 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 01:03:08.641165 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.626772    2578 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 01:03:08.641165 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.626776    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 01:03:08.641165 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.626779    2578 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 01:03:08.641165 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:08.633604    2578 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Oct 31 01:03:08.641165 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:08.633621    2578 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Oct 31 01:03:08.641165 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:08.633636    2578 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Oct 31 01:03:08.641165 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:08.633651    2578 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Oct 31 01:03:08.641165 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.633717    2578 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 01:03:08.641165 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.633731    2578 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 01:03:08.641165 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.633734    2578 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 01:03:08.641634 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.633738    2578 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 01:03:08.641634 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.633741    2578 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 01:03:08.641634 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.633744    2578 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 01:03:08.641634 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.633746    2578 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 01:03:08.641634 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.633749    2578 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 01:03:08.641634 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.633752    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 01:03:08.641634 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.633754    2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 01:03:08.641634 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.633757    2578 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 01:03:08.641634 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.633760    2578 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 01:03:08.641634 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.633762    2578 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 01:03:08.641634 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.633765    2578 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 01:03:08.641634 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.633768    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 01:03:08.641634 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.633770    2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 01:03:08.641634 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.633773    2578 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 01:03:08.641634 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.633776    2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 01:03:08.641634 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.633778    2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 01:03:08.641634 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.633781    2578 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 01:03:08.641634 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.633783    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 01:03:08.641634 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.633786    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 01:03:08.642093 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.633788    2578 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 01:03:08.642093 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.633791    2578 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 01:03:08.642093 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.633793    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 01:03:08.642093 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.633796    2578 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 01:03:08.642093 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.633798    2578 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 01:03:08.642093 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.633801    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 01:03:08.642093 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.633804    2578 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 01:03:08.642093 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.633809    2578 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 01:03:08.642093 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.633814    2578 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 01:03:08.642093 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.633817    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 01:03:08.642093 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.633820    2578 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 01:03:08.642093 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.633823    2578 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 01:03:08.642093 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.633827    2578 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 01:03:08.642093 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.633829    2578 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 01:03:08.642093 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.633833    2578 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 01:03:08.642093 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.633837    2578 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 01:03:08.642093 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.633840    2578 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 01:03:08.642093 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.633843    2578 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 01:03:08.642093 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.633846    2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 01:03:08.642553 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.633848    2578 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 01:03:08.642553 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.633851    2578 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 01:03:08.642553 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.633854    2578 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 01:03:08.642553 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.633856    2578 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 01:03:08.642553 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.633859    2578 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 01:03:08.642553 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.633861    2578 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 01:03:08.642553 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.633864    2578 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 01:03:08.642553 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.633867    2578 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 01:03:08.642553 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.633869    2578 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 01:03:08.642553 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.633872    2578 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 01:03:08.642553 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.633874    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 01:03:08.642553 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.633877    2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 01:03:08.642553 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.633879    2578 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 01:03:08.642553 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.633882    2578 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 01:03:08.642553 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.633884    2578 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 01:03:08.642553 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.633887    2578 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 01:03:08.642553 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.633890    2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 01:03:08.642553 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.633893    2578 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 01:03:08.642553 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.633895    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 01:03:08.642553 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.633898    2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 01:03:08.643017 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.633901    2578 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 01:03:08.643017 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.633904    2578 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 01:03:08.643017 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.633906    2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 01:03:08.643017 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.633909    2578 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 01:03:08.643017 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.633911    2578 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 01:03:08.643017 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.633914    2578 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 01:03:08.643017 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.633916    2578 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 01:03:08.643017 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.633919    2578 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 01:03:08.643017 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.633921    2578 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 01:03:08.643017 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.633924    2578 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 01:03:08.643017 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.633926    2578 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 01:03:08.643017 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.633928    2578 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 01:03:08.643017 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.633931    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 01:03:08.643017 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.633933    2578 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 01:03:08.643017 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.633936    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 01:03:08.643017 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.633938    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 01:03:08.643017 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.633941    2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 01:03:08.643017 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.633944    2578 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 01:03:08.643017 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.633947    2578 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 01:03:08.643017 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.633949    2578 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 01:03:08.643535 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.633952    2578 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 01:03:08.643535 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.633955    2578 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 01:03:08.643535 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.633957    2578 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 01:03:08.643535 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.633960    2578 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 01:03:08.643535 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.633962    2578 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 01:03:08.643535 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.633965    2578 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 01:03:08.643535 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.633967    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 01:03:08.643535 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.633970    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 01:03:08.643535 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.633972    2578 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 01:03:08.643535 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.633975    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 01:03:08.643535 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.633977    2578 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 01:03:08.643535 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.633980    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 01:03:08.643535 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.633983    2578 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 01:03:08.643535 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.633986    2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 01:03:08.643535 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.633989    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 01:03:08.643535 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.633991    2578 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 01:03:08.643535 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.633994    2578 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 01:03:08.643535 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.633996    2578 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 01:03:08.643535 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.633998    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 01:03:08.643535 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.634001    2578 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 01:03:08.644003 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.634003    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 01:03:08.644003 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.634006    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 01:03:08.644003 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.634008    2578 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 01:03:08.644003 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.634010    2578 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 01:03:08.644003 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.634013    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 01:03:08.644003 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:08.634039    2578 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 31 01:03:08.644003 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.634202    2578 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 01:03:08.644003 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.634207    2578 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 01:03:08.644003 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.634210    2578 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 01:03:08.644003 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.634213    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 01:03:08.644003 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.634217    2578 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 01:03:08.644003 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.634220    2578 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 01:03:08.644003 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.634223    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 01:03:08.644003 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.634226    2578 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 01:03:08.644003 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.634229    2578 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 01:03:08.644354 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.634231    2578 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 01:03:08.644354 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.634234    2578 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 01:03:08.644354 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.634238    2578 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 01:03:08.644354 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.634242    2578 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 01:03:08.644354 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.634244    2578 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 01:03:08.644354 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.634247    2578 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 01:03:08.644354 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.634250    2578 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 01:03:08.644354 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.634252    2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 01:03:08.644354 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.634255    2578 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 01:03:08.644354 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.634258    2578 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 01:03:08.644354 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.634262    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 01:03:08.644354 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.634264    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 01:03:08.644354 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.634268    2578 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 01:03:08.644354 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.634271    2578 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 01:03:08.644354 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.634273    2578 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 01:03:08.644354 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.634276    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 01:03:08.644354 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.634278    2578 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 01:03:08.644354 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.634280    2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 01:03:08.644354 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.634283    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 01:03:08.644781 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.634286    2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 01:03:08.644781 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.634288    2578 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 01:03:08.644781 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.634291    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 01:03:08.644781 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.634294    2578 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 01:03:08.644781 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.634297    2578 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 01:03:08.644781 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.634299    2578 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 01:03:08.644781 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.634302    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 01:03:08.644781 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.634305    2578 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 01:03:08.644781 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.634307    2578 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 01:03:08.644781 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.634310    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 01:03:08.644781 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.634329    2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 01:03:08.644781 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.634332    2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 01:03:08.644781 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.634335    2578 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 01:03:08.644781 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.634338    2578 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 01:03:08.644781 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.634340    2578 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 01:03:08.644781 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.634343    2578 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 01:03:08.644781 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.634346    2578 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 01:03:08.644781 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.634348    2578 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 01:03:08.644781 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.634351    2578 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 01:03:08.645232 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.634353    2578 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 01:03:08.645232 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.634356    2578 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 01:03:08.645232 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.634358    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 01:03:08.645232 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.634361    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 01:03:08.645232 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.634363    2578 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 01:03:08.645232 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.634366    2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 01:03:08.645232 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.634368    2578 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 01:03:08.645232 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.634371    2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 01:03:08.645232 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.634374    2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 01:03:08.645232 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.634376    2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 01:03:08.645232 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.634379    2578 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 01:03:08.645232 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.634382    2578 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 01:03:08.645232 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.634385    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 01:03:08.645232 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.634387    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 01:03:08.645232 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.634390    2578 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 01:03:08.645232 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.634392    2578 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 01:03:08.645232 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.634394    2578 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 01:03:08.645232 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.634397    2578 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 01:03:08.645232 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.634399    2578 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 01:03:08.645232 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.634401    2578 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 01:03:08.645716 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.634406    2578 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 01:03:08.645716 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.634409    2578 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 01:03:08.645716 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.634412    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 01:03:08.645716 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.634414    2578 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 01:03:08.645716 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.634417    2578 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 01:03:08.645716 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.634419    2578 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 01:03:08.645716 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.634422    2578 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 01:03:08.645716 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.634424    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 01:03:08.645716 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.634427    2578 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 01:03:08.645716 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.634430    2578 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 01:03:08.645716 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.634432    2578 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 01:03:08.645716 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.634435    2578 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 01:03:08.645716 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.634437    2578 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 01:03:08.645716 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.634440    2578 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 01:03:08.645716 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.634442    2578 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 01:03:08.645716 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.634445    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 01:03:08.645716 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.634447    2578 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 01:03:08.645716 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.634450    2578 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 01:03:08.645716 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.634452    2578 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 01:03:08.645716 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.634455    2578 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 01:03:08.646198 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.634458    2578 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 01:03:08.646198 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.634460    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 01:03:08.646198 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.634463    2578 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 01:03:08.646198 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.634465    2578 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 01:03:08.646198 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.634468    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 01:03:08.646198 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.634470    2578 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 01:03:08.646198 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.634473    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 01:03:08.646198 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.634475    2578 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 01:03:08.646198 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.634478    2578 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 01:03:08.646198 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.634480    2578 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 01:03:08.646198 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.634483    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 01:03:08.646198 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.634486    2578 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 01:03:08.646198 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.634490    2578 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 01:03:08.646198 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.634492    2578 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 01:03:08.646198 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.634495    2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 01:03:08.646198 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.634497    2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 01:03:08.646198 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.634500    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 01:03:08.646198 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.634502    2578 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 01:03:08.646198 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.634505    2578 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 01:03:08.646777 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:08.634528    2578 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 31 01:03:08.646777 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.634560    2578 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 01:03:08.646777 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.634564    2578 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 01:03:08.646777 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.634567    2578 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 01:03:08.646777 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.634569    2578 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 01:03:08.646777 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.634573    2578 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 01:03:08.646777 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.634577    2578 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 01:03:08.646777 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.634580    2578 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 01:03:08.646777 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.634583    2578 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 01:03:08.646777 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.634586    2578 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 01:03:08.646777 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.634589    2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 01:03:08.646777 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.634591    2578 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 01:03:08.646777 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.634595    2578 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 01:03:08.646777 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.634598    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 01:03:08.646777 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.634602    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 01:03:08.647132 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.634605    2578 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 01:03:08.647132 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.634607    2578 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 01:03:08.647132 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.634610    2578 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 01:03:08.647132 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.634612    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 01:03:08.647132 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.634615    2578 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 01:03:08.647132 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.634617    2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 01:03:08.647132 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.634620    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 01:03:08.647132 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.634623    2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 01:03:08.647132 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.634625    2578 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 01:03:08.647132 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.634628    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 01:03:08.647132 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.634632    2578 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 01:03:08.647132 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.634635    2578 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 01:03:08.647132 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.634637    2578 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 01:03:08.647132 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.634640    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 01:03:08.647132 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.634642    2578 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 01:03:08.647132 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.634645    2578 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 01:03:08.647132 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.634647    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 01:03:08.647132 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.634650    2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 01:03:08.647132 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.634652    2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 01:03:08.647595 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.634655    2578 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 01:03:08.647595 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.634657    2578 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 01:03:08.647595 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.634660    2578 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 01:03:08.647595 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.634662    2578 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 01:03:08.647595 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.634665    2578 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 01:03:08.647595 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.634667    2578 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 01:03:08.647595 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.634670    2578 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 01:03:08.647595 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.634672    2578 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 01:03:08.647595 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.634675    2578 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 01:03:08.647595 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.634677    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 01:03:08.647595 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.634680    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 01:03:08.647595 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.634682    2578 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 01:03:08.647595 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.634684    2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 01:03:08.647595 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.634688    2578 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 01:03:08.647595 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.634690    2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 01:03:08.647595 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.634692    2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 01:03:08.647595 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.634695    2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 01:03:08.647595 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.634698    2578 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 01:03:08.647595 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.634701    2578 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 01:03:08.647595 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.634703    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 01:03:08.648056 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.634706    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 01:03:08.648056 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.634708    2578 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 01:03:08.648056 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.634711    2578 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 01:03:08.648056 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.634714    2578 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 01:03:08.648056 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.634717    2578 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 01:03:08.648056 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.634720    2578 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 01:03:08.648056 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.634722    2578 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 01:03:08.648056 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.634724    2578 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 01:03:08.648056 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.634727    2578 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 01:03:08.648056 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.634729    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 01:03:08.648056 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.634732    2578 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 01:03:08.648056 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.634734    2578 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 01:03:08.648056 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.634737    2578 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 01:03:08.648056 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.634739    2578 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 01:03:08.648056 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.634742    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 01:03:08.648056 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.634744    2578 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 01:03:08.648056 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.634746    2578 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 01:03:08.648056 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.634749    2578 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 01:03:08.648056 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.634751    2578 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 01:03:08.648056 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.634753    2578 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 01:03:08.648528 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.634756    2578 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 01:03:08.648528 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.634758    2578 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 01:03:08.648528 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.634761    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 01:03:08.648528 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.634763    2578 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 01:03:08.648528 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.634766    2578 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 01:03:08.648528 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.634771    2578 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 01:03:08.648528 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.634774    2578 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 01:03:08.648528 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.634777    2578 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 01:03:08.648528 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.634779    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 01:03:08.648528 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.634782    2578 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 01:03:08.648528 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.634784    2578 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 01:03:08.648528 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.634787    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 01:03:08.648528 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.634789    2578 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 01:03:08.648528 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.634792    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 01:03:08.648528 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.634794    2578 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 01:03:08.648528 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.634799    2578 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 01:03:08.648528 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.634802    2578 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 01:03:08.648528 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.634804    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 01:03:08.648528 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.634807    2578 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 01:03:08.648528 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.634809    2578 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 01:03:08.648996 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.634812    2578 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 01:03:08.648996 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.634814    2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 01:03:08.648996 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.634817    2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 01:03:08.648996 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.634819    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 01:03:08.648996 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.634822    2578 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 01:03:08.648996 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.634824    2578 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 01:03:08.648996 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.634827    2578 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 01:03:08.648996 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.634830    2578 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 01:03:08.648996 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.634832    2578 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 01:03:08.648996 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.634834    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 01:03:08.648996 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.634838    2578 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 01:03:08.648996 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.634841    2578 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 01:03:08.648996 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:08.634843    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 01:03:08.648996 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:08.634899    2578 server.go:957] "Client rotation is on, will bootstrap in background"
Oct 31 01:03:08.648996 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:08.636992    2578 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Oct 31 01:03:08.648996 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:08.638510    2578 server.go:1014] "Starting client certificate rotation"
Oct 31 01:03:08.648996 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:08.639009    2578 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Oct 31 01:03:08.648996 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:08.639048    2578 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Oct 31 01:03:08.649425 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:08.642746    2578 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 31 01:03:08.649425 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:08.643918    2578 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 31 01:03:08.657436 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:08.657421    2578 log.go:25] "Validated CRI v1 runtime API"
Oct 31 01:03:08.661654 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:08.661635    2578 log.go:25] "Validated CRI v1 image API"
Oct 31 01:03:08.662872 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:08.662844    2578 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Oct 31 01:03:08.666193 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:08.666169    2578 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 894c0bb4-949e-41b2-a0ad-fe010d3501de:/dev/nvme0n1p3 f763df3b-a4a8-4bc9-bc11-a55871c3415d:/dev/nvme0n1p4]
Oct 31 01:03:08.666243 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:08.666194    2578 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}]
Oct 31 01:03:08.671680 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:08.671538    2578 manager.go:217] Machine: {Timestamp:2026-10-31 01:03:08.670140819 +0000 UTC m=+0.355620223 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3097412 MemoryCapacity:32812019712 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec26fc72061aacb4e05cbcf025a13103 SystemUUID:ec26fc72-061a-acb4-e05c-bcf025a13103 BootID:734896fe-ba5a-4878-a994-4b46528ea61c Filesystems:[{Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16406011904 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16406007808 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6562406400 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:32:9d:23:68:b5 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:32:9d:23:68:b5 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:ea:76:2d:1d:e3:7a Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812019712 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Oct 31 01:03:08.672328 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:08.672307    2578 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Oct 31 01:03:08.672420 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:08.672401    2578 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Oct 31 01:03:08.676774 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:08.676753    2578 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Oct 31 01:03:08.676914 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:08.676776    2578 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-133-197.us-west-1.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Oct 31 01:03:08.676955 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:08.676916    2578 topology_manager.go:143] "Creating topology manager with none policy"
Oct 31 01:03:08.676955 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:08.676925    2578 container_manager_linux.go:311] "Creating device plugin manager"
Oct 31 01:03:08.677014 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:08.676963    2578 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 31 01:03:08.677014 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:08.676980    2578 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 31 01:03:08.678219 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:08.678208    2578 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Oct 31 01:03:08.682255 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:08.682244    2578 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Oct 31 01:03:08.682388 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:08.682379    2578 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Oct 31 01:03:08.683119 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:08.683110    2578 kubelet.go:493] "Attempting to sync node with API server"
Oct 31 01:03:08.683157 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:08.683127    2578 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Oct 31 01:03:08.683157 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:08.683146    2578 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Oct 31 01:03:08.683206 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:08.683158    2578 kubelet.go:405] "Adding apiserver pod source"
Oct 31 01:03:08.683206 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:08.683171    2578 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Oct 31 01:03:08.683562 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:08.683551    2578 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 31 01:03:08.686063 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:08.686047    2578 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Oct 31 01:03:08.693029 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:08.693011    2578 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 31 01:03:08.693927 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:08.693912    2578 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Oct 31 01:03:08.693974 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:08.693945    2578 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Oct 31 01:03:08.694102 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:08.694092    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Oct 31 01:03:08.694131 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:08.694109    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Oct 31 01:03:08.694131 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:08.694115    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Oct 31 01:03:08.694131 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:08.694120    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Oct 31 01:03:08.694131 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:08.694126    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Oct 31 01:03:08.694131 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:08.694131    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Oct 31 01:03:08.694899 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:08.694890    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Oct 31 01:03:08.694930 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:08.694899    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Oct 31 01:03:08.694930 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:08.694907    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Oct 31 01:03:08.694930 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:08.694913    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Oct 31 01:03:08.694930 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:08.694921    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Oct 31 01:03:08.694930 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:08.694931    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Oct 31 01:03:08.695797 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:08.695786    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Oct 31 01:03:08.695836 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:08.695799    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Oct 31 01:03:08.699616 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:08.699602    2578 server.go:1263] "Started kubelet"
Oct 31 01:03:08.699770 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:08.699705    2578 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Oct 31 01:03:08.699866 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:08.699789    2578 server_v1.go:49] "podresources" method="list" useActivePods=true
Oct 31 01:03:08.700000 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:08.699979    2578 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Oct 31 01:03:08.700258 ip-10-0-133-197 systemd[1]: Started Kubernetes Kubelet.
Oct 31 01:03:08.701099 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:08.701079    2578 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Oct 31 01:03:08.701839 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:08.701822    2578 server.go:317] "Adding debug handlers to kubelet server"
Oct 31 01:03:08.704667 ip-10-0-133-197 kubenswrapper[2578]: E1031 01:03:08.704649    2578 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Oct 31 01:03:08.708335 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:08.708300    2578 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Oct 31 01:03:08.708335 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:08.708312    2578 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Oct 31 01:03:08.708487 ip-10-0-133-197 kubenswrapper[2578]: E1031 01:03:08.708383    2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-197.us-west-1.compute.internal\" not found"
Oct 31 01:03:08.708935 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:08.708916    2578 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Oct 31 01:03:08.709026 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:08.708917    2578 volume_manager.go:309] "The desired_state_of_world populator starts"
Oct 31 01:03:08.709026 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:08.708958    2578 volume_manager.go:311] "Starting Kubelet Volume Manager"
Oct 31 01:03:08.709026 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:08.709011    2578 reconstruct.go:97] "Volume reconstruction finished"
Oct 31 01:03:08.709026 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:08.709018    2578 reconciler.go:29] "Reconciler: start to sync state"
Oct 31 01:03:08.709402 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:08.709105    2578 factory.go:55] Registering systemd factory
Oct 31 01:03:08.709402 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:08.709117    2578 factory.go:223] Registration of the systemd container factory successfully
Oct 31 01:03:08.709402 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:08.709341    2578 factory.go:153] Registering CRI-O factory
Oct 31 01:03:08.709402 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:08.709356    2578 factory.go:223] Registration of the crio container factory successfully
Oct 31 01:03:08.709584 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:08.709409    2578 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Oct 31 01:03:08.709584 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:08.709430    2578 factory.go:103] Registering Raw factory
Oct 31 01:03:08.709584 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:08.709443    2578 manager.go:1199] Started watching for new ooms in manager
Oct 31 01:03:08.710466 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:08.710451    2578 manager.go:319] Starting recovery of all containers
Oct 31 01:03:08.721970 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:08.721955    2578 manager.go:324] Recovery completed
Oct 31 01:03:08.727255 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:08.727240    2578 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 31 01:03:08.729914 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:08.729889    2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-197.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Oct 31 01:03:08.729974 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:08.729926    2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-197.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Oct 31 01:03:08.729974 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:08.729937    2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-197.us-west-1.compute.internal" event="NodeHasSufficientPID"
Oct 31 01:03:08.730475 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:08.730462    2578 cpu_manager.go:226] "Starting" policy="none"
Oct 31 01:03:08.730475 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:08.730475    2578 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 31 01:03:08.730580 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:08.730474    2578 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Oct 31 01:03:08.730580 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:08.730530    2578 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Oct 31 01:03:08.731147 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:08.731133    2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-197.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Oct 31 01:03:08.731200 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:08.731157    2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-197.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Oct 31 01:03:08.731200 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:08.731173    2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-197.us-west-1.compute.internal" event="NodeHasSufficientPID"
Oct 31 01:03:08.732765 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:08.732749    2578 policy_none.go:50] "Start"
Oct 31 01:03:08.732821 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:08.732773    2578 memory_manager.go:187] "Starting memorymanager" policy="None"
Oct 31 01:03:08.732821 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:08.732791    2578 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Oct 31 01:03:08.734917 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:08.734906    2578 policy_none.go:44] "Start"
Oct 31 01:03:08.789776 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:08.778904    2578 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Oct 31 01:03:08.789776 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:08.780689    2578 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Oct 31 01:03:08.789776 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:08.782092    2578 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Oct 31 01:03:08.789776 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:08.782117    2578 status_manager.go:249] "Starting to sync pod status with apiserver"
Oct 31 01:03:08.789776 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:08.782134    2578 kubelet.go:2521] "Starting kubelet main sync loop"
Oct 31 01:03:08.789776 ip-10-0-133-197 kubenswrapper[2578]: E1031 01:03:08.782166    2578 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]"
Oct 31 01:03:08.789776 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:08.783366    2578 manager.go:341] "Starting Device Plugin manager"
Oct 31 01:03:08.789776 ip-10-0-133-197 kubenswrapper[2578]: E1031 01:03:08.783399    2578 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Oct 31 01:03:08.789776 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:08.783411    2578 server.go:87] "Starting device plugin registration server"
Oct 31 01:03:08.789776 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:08.783661    2578 eviction_manager.go:194] "Eviction manager: starting control loop"
Oct 31 01:03:08.789776 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:08.783673    2578 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Oct 31 01:03:08.789776 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:08.783767    2578 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Oct 31 01:03:08.789776 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:08.783865    2578 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Oct 31 01:03:08.789776 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:08.783877    2578 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Oct 31 01:03:08.789776 ip-10-0-133-197 kubenswrapper[2578]: E1031 01:03:08.784546    2578 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Oct 31 01:03:08.789776 ip-10-0-133-197 kubenswrapper[2578]: E1031 01:03:08.784586    2578 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-133-197.us-west-1.compute.internal\" not found"
Oct 31 01:03:08.843246 ip-10-0-133-197 kubenswrapper[2578]: E1031 01:03:08.843216    2578 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-133-197.us-west-1.compute.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Oct 31 01:03:08.843246 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:08.843229    2578 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-133-197.us-west-1.compute.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Oct 31 01:03:08.844671 ip-10-0-133-197 kubenswrapper[2578]: E1031 01:03:08.843224    2578 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-133-197.us-west-1.compute.internal.18e3779c0a9294ce  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-133-197.us-west-1.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-133-197.us-west-1.compute.internal,},FirstTimestamp:2026-10-31 01:03:08.699571406 +0000 UTC m=+0.385050812,LastTimestamp:2026-10-31 01:03:08.699571406 +0000 UTC m=+0.385050812,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-133-197.us-west-1.compute.internal,}"
Oct 31 01:03:08.882810 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:08.882752    2578 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-197.us-west-1.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-133-197.us-west-1.compute.internal"]
Oct 31 01:03:08.882894 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:08.882834    2578 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 31 01:03:08.883644 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:08.883629    2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-197.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Oct 31 01:03:08.883709 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:08.883658    2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-197.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Oct 31 01:03:08.883709 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:08.883668    2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-197.us-west-1.compute.internal" event="NodeHasSufficientPID"
Oct 31 01:03:08.883788 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:08.883768    2578 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 31 01:03:08.884677 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:08.884663    2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-197.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Oct 31 01:03:08.884768 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:08.884693    2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-197.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Oct 31 01:03:08.884768 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:08.884710    2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-197.us-west-1.compute.internal" event="NodeHasSufficientPID"
Oct 31 01:03:08.884768 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:08.884738    2578 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-133-197.us-west-1.compute.internal"
Oct 31 01:03:08.885538 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:08.885524    2578 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 31 01:03:08.885687 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:08.885674    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-197.us-west-1.compute.internal"
Oct 31 01:03:08.885747 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:08.885704    2578 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 31 01:03:08.886208 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:08.886193    2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-197.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Oct 31 01:03:08.886208 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:08.886202    2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-197.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Oct 31 01:03:08.886354 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:08.886222    2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-197.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Oct 31 01:03:08.886354 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:08.886239    2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-197.us-west-1.compute.internal" event="NodeHasSufficientPID"
Oct 31 01:03:08.886354 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:08.886223    2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-197.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Oct 31 01:03:08.886354 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:08.886343    2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-197.us-west-1.compute.internal" event="NodeHasSufficientPID"
Oct 31 01:03:08.888059 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:08.888042    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-197.us-west-1.compute.internal"
Oct 31 01:03:08.888145 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:08.888071    2578 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 31 01:03:08.888760 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:08.888745    2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-197.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Oct 31 01:03:08.888850 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:08.888776    2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-197.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Oct 31 01:03:08.888850 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:08.888790    2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-197.us-west-1.compute.internal" event="NodeHasSufficientPID"
Oct 31 01:03:08.910942 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:08.910918    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/107fc5ddbeb783edeed66dfa339956e5-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-197.us-west-1.compute.internal\" (UID: \"107fc5ddbeb783edeed66dfa339956e5\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-197.us-west-1.compute.internal"
Oct 31 01:03:08.911020 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:08.910944    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/107fc5ddbeb783edeed66dfa339956e5-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-197.us-west-1.compute.internal\" (UID: \"107fc5ddbeb783edeed66dfa339956e5\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-197.us-west-1.compute.internal"
Oct 31 01:03:08.911020 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:08.910961    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/8dea7006d41de627c8e06bf18f518df6-config\") pod \"kube-apiserver-proxy-ip-10-0-133-197.us-west-1.compute.internal\" (UID: \"8dea7006d41de627c8e06bf18f518df6\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-197.us-west-1.compute.internal"
Oct 31 01:03:08.914417 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:08.914402    2578 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Oct 31 01:03:08.914474 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:08.914449    2578 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-w7vbc"
Oct 31 01:03:08.921169 ip-10-0-133-197 kubenswrapper[2578]: E1031 01:03:08.921151    2578 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-133-197.us-west-1.compute.internal\" not found" node="ip-10-0-133-197.us-west-1.compute.internal"
Oct 31 01:03:08.923951 ip-10-0-133-197 kubenswrapper[2578]: E1031 01:03:08.923935    2578 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-133-197.us-west-1.compute.internal\" not found" node="ip-10-0-133-197.us-west-1.compute.internal"
Oct 31 01:03:08.943063 ip-10-0-133-197 kubenswrapper[2578]: E1031 01:03:08.943036    2578 reflector.go:204] "Failed to watch" err="failed to list *v1.RuntimeClass: runtimeclasses.node.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"runtimeclasses\" in API group \"node.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass"
Oct 31 01:03:08.943063 ip-10-0-133-197 kubenswrapper[2578]: E1031 01:03:08.943041    2578 reflector.go:204] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-133-197.us-west-1.compute.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node"
Oct 31 01:03:08.991601 ip-10-0-133-197 kubenswrapper[2578]: E1031 01:03:08.991573    2578 reflector.go:204] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver"
Oct 31 01:03:08.991601 ip-10-0-133-197 kubenswrapper[2578]: E1031 01:03:08.991524    2578 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-133-197.us-west-1.compute.internal.18e3779c0c618ff0  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-133-197.us-west-1.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-133-197.us-west-1.compute.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-133-197.us-west-1.compute.internal,},FirstTimestamp:2026-10-31 01:03:08.729913328 +0000 UTC m=+0.415392733,LastTimestamp:2026-10-31 01:03:08.729913328 +0000 UTC m=+0.415392733,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-133-197.us-west-1.compute.internal,}"
Oct 31 01:03:08.991760 ip-10-0-133-197 kubenswrapper[2578]: E1031 01:03:08.991607    2578 reflector.go:204] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service"
Oct 31 01:03:08.998487 ip-10-0-133-197 kubenswrapper[2578]: E1031 01:03:08.998466    2578 kubelet_node_status.go:115] "Unable to register node with API server, error getting existing node" err="nodes \"ip-10-0-133-197.us-west-1.compute.internal\" not found" node="ip-10-0-133-197.us-west-1.compute.internal"
Oct 31 01:03:09.011854 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:09.011832    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/107fc5ddbeb783edeed66dfa339956e5-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-197.us-west-1.compute.internal\" (UID: \"107fc5ddbeb783edeed66dfa339956e5\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-197.us-west-1.compute.internal"
Oct 31 01:03:09.011933 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:09.011866    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/107fc5ddbeb783edeed66dfa339956e5-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-197.us-west-1.compute.internal\" (UID: \"107fc5ddbeb783edeed66dfa339956e5\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-197.us-west-1.compute.internal"
Oct 31 01:03:09.011933 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:09.011892    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/8dea7006d41de627c8e06bf18f518df6-config\") pod \"kube-apiserver-proxy-ip-10-0-133-197.us-west-1.compute.internal\" (UID: \"8dea7006d41de627c8e06bf18f518df6\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-197.us-west-1.compute.internal"
Oct 31 01:03:09.012020 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:09.011942    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/107fc5ddbeb783edeed66dfa339956e5-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-197.us-west-1.compute.internal\" (UID: \"107fc5ddbeb783edeed66dfa339956e5\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-197.us-west-1.compute.internal"
Oct 31 01:03:09.012020 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:09.011947    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/8dea7006d41de627c8e06bf18f518df6-config\") pod \"kube-apiserver-proxy-ip-10-0-133-197.us-west-1.compute.internal\" (UID: \"8dea7006d41de627c8e06bf18f518df6\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-197.us-west-1.compute.internal"
Oct 31 01:03:09.012020 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:09.011942    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/107fc5ddbeb783edeed66dfa339956e5-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-197.us-west-1.compute.internal\" (UID: \"107fc5ddbeb783edeed66dfa339956e5\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-197.us-west-1.compute.internal"
Oct 31 01:03:09.146089 ip-10-0-133-197 kubenswrapper[2578]: E1031 01:03:09.146024    2578 nodelease.go:50] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-133-197.us-west-1.compute.internal\" not found" node="ip-10-0-133-197.us-west-1.compute.internal"
Oct 31 01:03:09.198696 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:09.198667    2578 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 31 01:03:09.203410 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:09.203389    2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-197.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Oct 31 01:03:09.203474 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:09.203430    2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-197.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Oct 31 01:03:09.203474 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:09.203441    2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-197.us-west-1.compute.internal" event="NodeHasSufficientPID"
Oct 31 01:03:09.203474 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:09.203469    2578 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-133-197.us-west-1.compute.internal"
Oct 31 01:03:09.224982 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:09.224963    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-197.us-west-1.compute.internal"
Oct 31 01:03:09.227974 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:09.227958    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-197.us-west-1.compute.internal"
Oct 31 01:03:09.260823 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:09.260803    2578 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-133-197.us-west-1.compute.internal"
Oct 31 01:03:09.260904 ip-10-0-133-197 kubenswrapper[2578]: E1031 01:03:09.260825    2578 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-133-197.us-west-1.compute.internal\": node \"ip-10-0-133-197.us-west-1.compute.internal\" not found"
Oct 31 01:03:09.369907 ip-10-0-133-197 kubenswrapper[2578]: E1031 01:03:09.369884    2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-197.us-west-1.compute.internal\" not found"
Oct 31 01:03:09.470670 ip-10-0-133-197 kubenswrapper[2578]: E1031 01:03:09.470610    2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-197.us-west-1.compute.internal\" not found"
Oct 31 01:03:09.571728 ip-10-0-133-197 kubenswrapper[2578]: E1031 01:03:09.571706    2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-197.us-west-1.compute.internal\" not found"
Oct 31 01:03:09.638711 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:09.638688    2578 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Oct 31 01:03:09.639100 ip-10-0-133-197 kubenswrapper[2578]: E1031 01:03:09.638833    2578 event.go:368] "Unable to write event (may retry after sleeping)" err="Patch \"https://a53752eb2738248268d838fbfc908485-8edbb764ce6e4113.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/default/events/ip-10-0-133-197.us-west-1.compute.internal.18e3779c0c618ff0\": read tcp 10.0.133.197:44578->3.151.241.160:6443: use of closed network connection" event="&Event{ObjectMeta:{ip-10-0-133-197.us-west-1.compute.internal.18e3779c0c618ff0  default   4774 0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-133-197.us-west-1.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-133-197.us-west-1.compute.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-133-197.us-west-1.compute.internal,},FirstTimestamp:2026-10-31 01:03:08 +0000 UTC,LastTimestamp:2026-10-31 01:03:08.884677076 +0000 UTC m=+0.570156488,Count:4,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-133-197.us-west-1.compute.internal,}"
Oct 31 01:03:09.671917 ip-10-0-133-197 kubenswrapper[2578]: E1031 01:03:09.671895    2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-197.us-west-1.compute.internal\" not found"
Oct 31 01:03:09.772501 ip-10-0-133-197 kubenswrapper[2578]: E1031 01:03:09.772453    2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-197.us-west-1.compute.internal\" not found"
Oct 31 01:03:09.860445 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:09.860429    2578 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 31 01:03:09.861202 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:09.861189    2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-197.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Oct 31 01:03:09.861262 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:09.861218    2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-197.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Oct 31 01:03:09.861262 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:09.861227    2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-197.us-west-1.compute.internal" event="NodeHasSufficientPID"
Oct 31 01:03:09.872802 ip-10-0-133-197 kubenswrapper[2578]: E1031 01:03:09.872788    2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-197.us-west-1.compute.internal\" not found"
Oct 31 01:03:09.918764 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:09.918741    2578 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-30 00:58:09 +0000 UTC" deadline="2028-07-26 03:06:07.426693034 +0000 UTC"
Oct 31 01:03:09.918819 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:09.918763    2578 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15218h2m57.507933002s"
Oct 31 01:03:09.973752 ip-10-0-133-197 kubenswrapper[2578]: E1031 01:03:09.973729    2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-197.us-west-1.compute.internal\" not found"
Oct 31 01:03:10.000522 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:10.000502    2578 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 31 01:03:10.074204 ip-10-0-133-197 kubenswrapper[2578]: E1031 01:03:10.074154    2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-197.us-west-1.compute.internal\" not found"
Oct 31 01:03:10.114037 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:10.114012    2578 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 31 01:03:10.122968 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:10.122953    2578 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 31 01:03:10.209084 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:10.209061    2578 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-197.us-west-1.compute.internal"
Oct 31 01:03:10.216821 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:10.216803    2578 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 31 01:03:10.265833 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:10.265809    2578 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 31 01:03:10.266794 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:10.266778    2578 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-197.us-west-1.compute.internal"
Oct 31 01:03:10.328016 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:10.327964    2578 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 31 01:03:10.421586 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:10.421560    2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8dea7006d41de627c8e06bf18f518df6.slice/crio-e5ff0a702058f6c4a964d51275a39426fdf7de02bd4756e66c20fde0c8b3a1d3 WatchSource:0}: Error finding container e5ff0a702058f6c4a964d51275a39426fdf7de02bd4756e66c20fde0c8b3a1d3: Status 404 returned error can't find the container with id e5ff0a702058f6c4a964d51275a39426fdf7de02bd4756e66c20fde0c8b3a1d3
Oct 31 01:03:10.422028 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:10.422008    2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod107fc5ddbeb783edeed66dfa339956e5.slice/crio-45b221fc6e0aeea83b1d8e8c88b6e9db2827f1a4aa5b5c74bc6b6ea9efc46143 WatchSource:0}: Error finding container 45b221fc6e0aeea83b1d8e8c88b6e9db2827f1a4aa5b5c74bc6b6ea9efc46143: Status 404 returned error can't find the container with id 45b221fc6e0aeea83b1d8e8c88b6e9db2827f1a4aa5b5c74bc6b6ea9efc46143
Oct 31 01:03:10.428003 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:10.427987    2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 31 01:03:10.684084 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:10.684065    2578 apiserver.go:52] "Watching apiserver"
Oct 31 01:03:10.709769 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:10.709741    2578 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Oct 31 01:03:10.787161 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:10.787122    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-197.us-west-1.compute.internal" event={"ID":"107fc5ddbeb783edeed66dfa339956e5","Type":"ContainerStarted","Data":"45b221fc6e0aeea83b1d8e8c88b6e9db2827f1a4aa5b5c74bc6b6ea9efc46143"}
Oct 31 01:03:10.788275 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:10.788254    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-197.us-west-1.compute.internal" event={"ID":"8dea7006d41de627c8e06bf18f518df6","Type":"ContainerStarted","Data":"e5ff0a702058f6c4a964d51275a39426fdf7de02bd4756e66c20fde0c8b3a1d3"}
Oct 31 01:03:10.796614 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:10.796597    2578 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Oct 31 01:03:10.796895 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:10.796879    2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-197.us-west-1.compute.internal","openshift-multus/network-metrics-daemon-wdsmn","kube-system/global-pull-secret-syncer-zs5g7","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-snxz4","openshift-multus/multus-additional-cni-plugins-jwthj","kube-system/konnectivity-agent-78szg","openshift-cluster-node-tuning-operator/tuned-45bvr","openshift-multus/multus-hxh6j","openshift-dns/node-resolver-j46hv","openshift-network-diagnostics/network-check-target-8mqbd","openshift-network-operator/iptables-alerter-ssw8w","openshift-ovn-kubernetes/ovnkube-node-pghw2","kube-system/kube-apiserver-proxy-ip-10-0-133-197.us-west-1.compute.internal","openshift-image-registry/node-ca-rlxt2"]
Oct 31 01:03:10.800817 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:10.800800    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8mqbd"
Oct 31 01:03:10.800892 ip-10-0-133-197 kubenswrapper[2578]: E1031 01:03:10.800876    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8mqbd" podUID="565fefaf-82aa-42a8-9cc7-6517c4bddf1e"
Oct 31 01:03:10.802988 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:10.802968    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-ssw8w"
Oct 31 01:03:10.805943 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:10.805910    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zs5g7"
Oct 31 01:03:10.806024 ip-10-0-133-197 kubenswrapper[2578]: E1031 01:03:10.805998    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zs5g7" podUID="aa4836a5-c71a-485d-82fc-f8b53dae7717"
Oct 31 01:03:10.808866 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:10.808688    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-rlxt2"
Oct 31 01:03:10.810984 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:10.810967    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-jwthj"
Oct 31 01:03:10.813324 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:10.813305    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-78szg"
Oct 31 01:03:10.815596 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:10.815575    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-j46hv"
Oct 31 01:03:10.817833 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:10.817814    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-hxh6j"
Oct 31 01:03:10.820092 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:10.820074    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wdsmn"
Oct 31 01:03:10.820168 ip-10-0-133-197 kubenswrapper[2578]: E1031 01:03:10.820149    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wdsmn" podUID="8952fc79-c5a6-42ba-a296-8cb7a776f718"
Oct 31 01:03:10.820800 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:10.820780    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/9cd1266e-c630-4259-b1f7-1b88dbde7385-iptables-alerter-script\") pod \"iptables-alerter-ssw8w\" (UID: \"9cd1266e-c630-4259-b1f7-1b88dbde7385\") " pod="openshift-network-operator/iptables-alerter-ssw8w"
Oct 31 01:03:10.820867 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:10.820816    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9cd1266e-c630-4259-b1f7-1b88dbde7385-host-slash\") pod \"iptables-alerter-ssw8w\" (UID: \"9cd1266e-c630-4259-b1f7-1b88dbde7385\") " pod="openshift-network-operator/iptables-alerter-ssw8w"
Oct 31 01:03:10.820905 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:10.820889    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jxmsl\" (UniqueName: \"kubernetes.io/projected/9cd1266e-c630-4259-b1f7-1b88dbde7385-kube-api-access-jxmsl\") pod \"iptables-alerter-ssw8w\" (UID: \"9cd1266e-c630-4259-b1f7-1b88dbde7385\") " pod="openshift-network-operator/iptables-alerter-ssw8w"
Oct 31 01:03:10.820943 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:10.820914    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/aa4836a5-c71a-485d-82fc-f8b53dae7717-dbus\") pod \"global-pull-secret-syncer-zs5g7\" (UID: \"aa4836a5-c71a-485d-82fc-f8b53dae7717\") " pod="kube-system/global-pull-secret-syncer-zs5g7"
Oct 31 01:03:10.820943 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:10.820932    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/dd8134f9-b2a6-4542-82ec-c10edc3fcd72-serviceca\") pod \"node-ca-rlxt2\" (UID: \"dd8134f9-b2a6-4542-82ec-c10edc3fcd72\") " pod="openshift-image-registry/node-ca-rlxt2"
Oct 31 01:03:10.821027 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:10.820969    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/fa085dd8-ca1e-4ec5-acb2-9b9dc878bf8a-cnibin\") pod \"multus-additional-cni-plugins-jwthj\" (UID: \"fa085dd8-ca1e-4ec5-acb2-9b9dc878bf8a\") " pod="openshift-multus/multus-additional-cni-plugins-jwthj"
Oct 31 01:03:10.821066 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:10.821037    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/dd8134f9-b2a6-4542-82ec-c10edc3fcd72-host\") pod \"node-ca-rlxt2\" (UID: \"dd8134f9-b2a6-4542-82ec-c10edc3fcd72\") " pod="openshift-image-registry/node-ca-rlxt2"
Oct 31 01:03:10.821115 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:10.821098    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/fa085dd8-ca1e-4ec5-acb2-9b9dc878bf8a-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-jwthj\" (UID: \"fa085dd8-ca1e-4ec5-acb2-9b9dc878bf8a\") " pod="openshift-multus/multus-additional-cni-plugins-jwthj"
Oct 31 01:03:10.821149 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:10.821126    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r6zx9\" (UniqueName: \"kubernetes.io/projected/565fefaf-82aa-42a8-9cc7-6517c4bddf1e-kube-api-access-r6zx9\") pod \"network-check-target-8mqbd\" (UID: \"565fefaf-82aa-42a8-9cc7-6517c4bddf1e\") " pod="openshift-network-diagnostics/network-check-target-8mqbd"
Oct 31 01:03:10.821182 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:10.821154    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/aa4836a5-c71a-485d-82fc-f8b53dae7717-kubelet-config\") pod \"global-pull-secret-syncer-zs5g7\" (UID: \"aa4836a5-c71a-485d-82fc-f8b53dae7717\") " pod="kube-system/global-pull-secret-syncer-zs5g7"
Oct 31 01:03:10.821182 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:10.821174    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fa085dd8-ca1e-4ec5-acb2-9b9dc878bf8a-system-cni-dir\") pod \"multus-additional-cni-plugins-jwthj\" (UID: \"fa085dd8-ca1e-4ec5-acb2-9b9dc878bf8a\") " pod="openshift-multus/multus-additional-cni-plugins-jwthj"
Oct 31 01:03:10.821246 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:10.821190    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/fa085dd8-ca1e-4ec5-acb2-9b9dc878bf8a-cni-binary-copy\") pod \"multus-additional-cni-plugins-jwthj\" (UID: \"fa085dd8-ca1e-4ec5-acb2-9b9dc878bf8a\") " pod="openshift-multus/multus-additional-cni-plugins-jwthj"
Oct 31 01:03:10.821246 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:10.821205    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/fa085dd8-ca1e-4ec5-acb2-9b9dc878bf8a-tuning-conf-dir\") pod \"multus-additional-cni-plugins-jwthj\" (UID: \"fa085dd8-ca1e-4ec5-acb2-9b9dc878bf8a\") " pod="openshift-multus/multus-additional-cni-plugins-jwthj"
Oct 31 01:03:10.821246 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:10.821220    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/fa085dd8-ca1e-4ec5-acb2-9b9dc878bf8a-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-jwthj\" (UID: \"fa085dd8-ca1e-4ec5-acb2-9b9dc878bf8a\") " pod="openshift-multus/multus-additional-cni-plugins-jwthj"
Oct 31 01:03:10.821246 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:10.821238    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/aa4836a5-c71a-485d-82fc-f8b53dae7717-original-pull-secret\") pod \"global-pull-secret-syncer-zs5g7\" (UID: \"aa4836a5-c71a-485d-82fc-f8b53dae7717\") " pod="kube-system/global-pull-secret-syncer-zs5g7"
Oct 31 01:03:10.821380 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:10.821268    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wsrb6\" (UniqueName: \"kubernetes.io/projected/dd8134f9-b2a6-4542-82ec-c10edc3fcd72-kube-api-access-wsrb6\") pod \"node-ca-rlxt2\" (UID: \"dd8134f9-b2a6-4542-82ec-c10edc3fcd72\") " pod="openshift-image-registry/node-ca-rlxt2"
Oct 31 01:03:10.821380 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:10.821315    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nmlm7\" (UniqueName: \"kubernetes.io/projected/fa085dd8-ca1e-4ec5-acb2-9b9dc878bf8a-kube-api-access-nmlm7\") pod \"multus-additional-cni-plugins-jwthj\" (UID: \"fa085dd8-ca1e-4ec5-acb2-9b9dc878bf8a\") " pod="openshift-multus/multus-additional-cni-plugins-jwthj"
Oct 31 01:03:10.822690 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:10.822667    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-pghw2"
Oct 31 01:03:10.825191 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:10.825169    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-45bvr"
Oct 31 01:03:10.827789 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:10.827772    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-snxz4"
Oct 31 01:03:10.859365 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:10.859201    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Oct 31 01:03:10.859365 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:10.859305    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-42fft\""
Oct 31 01:03:10.859507 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:10.859449    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Oct 31 01:03:10.859651 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:10.859632    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Oct 31 01:03:10.864455 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:10.864434    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Oct 31 01:03:10.864552 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:10.864504    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-kk499\""
Oct 31 01:03:10.864552 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:10.864523    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Oct 31 01:03:10.864552 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:10.864524    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Oct 31 01:03:10.866439 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:10.866424    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Oct 31 01:03:10.898521 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:10.898499    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Oct 31 01:03:10.898665 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:10.898650    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Oct 31 01:03:10.898773 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:10.898760    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Oct 31 01:03:10.898831 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:10.898772    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Oct 31 01:03:10.898831 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:10.898790    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-grw5x\""
Oct 31 01:03:10.902937 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:10.902924    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-55dd6\""
Oct 31 01:03:10.907015 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:10.907000    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Oct 31 01:03:10.907069 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:10.907044    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Oct 31 01:03:10.909852 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:10.909840    2578 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Oct 31 01:03:10.910729 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:10.910714    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Oct 31 01:03:10.913272 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:10.913259    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Oct 31 01:03:10.913344 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:10.913281    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-zzhf8\""
Oct 31 01:03:10.913425 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:10.913380    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Oct 31 01:03:10.913465 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:10.913456    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-7g746\""
Oct 31 01:03:10.918501 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:10.918488    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Oct 31 01:03:10.921734 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:10.921716    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/318b3bc6-6e59-4211-af63-b116526a325d-registration-dir\") pod \"aws-ebs-csi-driver-node-snxz4\" (UID: \"318b3bc6-6e59-4211-af63-b116526a325d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-snxz4"
Oct 31 01:03:10.921813 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:10.921748    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/57c2f196-0d8a-4bce-a5bc-3b6b5c0d808c-hosts-file\") pod \"node-resolver-j46hv\" (UID: \"57c2f196-0d8a-4bce-a5bc-3b6b5c0d808c\") " pod="openshift-dns/node-resolver-j46hv"
Oct 31 01:03:10.921813 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:10.921774    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/d55eb70a-f1a8-4346-b8df-ccd2c598dd6b-run-ovn\") pod \"ovnkube-node-pghw2\" (UID: \"d55eb70a-f1a8-4346-b8df-ccd2c598dd6b\") " pod="openshift-ovn-kubernetes/ovnkube-node-pghw2"
Oct 31 01:03:10.921813 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:10.921798    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/08490837-b3b6-4c2b-a498-568af0d7086d-etc-tuned\") pod \"tuned-45bvr\" (UID: \"08490837-b3b6-4c2b-a498-568af0d7086d\") " pod="openshift-cluster-node-tuning-operator/tuned-45bvr"
Oct 31 01:03:10.921947 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:10.921848    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8952fc79-c5a6-42ba-a296-8cb7a776f718-metrics-certs\") pod \"network-metrics-daemon-wdsmn\" (UID: \"8952fc79-c5a6-42ba-a296-8cb7a776f718\") " pod="openshift-multus/network-metrics-daemon-wdsmn"
Oct 31 01:03:10.921947 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:10.921870    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/58b0b75a-1568-4703-ac55-fa87f29225ad-multus-daemon-config\") pod \"multus-hxh6j\" (UID: \"58b0b75a-1568-4703-ac55-fa87f29225ad\") " pod="openshift-multus/multus-hxh6j"
Oct 31 01:03:10.921947 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:10.921888    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/08490837-b3b6-4c2b-a498-568af0d7086d-etc-kubernetes\") pod \"tuned-45bvr\" (UID: \"08490837-b3b6-4c2b-a498-568af0d7086d\") " pod="openshift-cluster-node-tuning-operator/tuned-45bvr"
Oct 31 01:03:10.921947 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:10.921907    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fa085dd8-ca1e-4ec5-acb2-9b9dc878bf8a-system-cni-dir\") pod \"multus-additional-cni-plugins-jwthj\" (UID: \"fa085dd8-ca1e-4ec5-acb2-9b9dc878bf8a\") " pod="openshift-multus/multus-additional-cni-plugins-jwthj"
Oct 31 01:03:10.921947 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:10.921945    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fa085dd8-ca1e-4ec5-acb2-9b9dc878bf8a-system-cni-dir\") pod \"multus-additional-cni-plugins-jwthj\" (UID: \"fa085dd8-ca1e-4ec5-acb2-9b9dc878bf8a\") " pod="openshift-multus/multus-additional-cni-plugins-jwthj"
Oct 31 01:03:10.922201 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:10.921959    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/fa085dd8-ca1e-4ec5-acb2-9b9dc878bf8a-cni-binary-copy\") pod \"multus-additional-cni-plugins-jwthj\" (UID: \"fa085dd8-ca1e-4ec5-acb2-9b9dc878bf8a\") " pod="openshift-multus/multus-additional-cni-plugins-jwthj"
Oct 31 01:03:10.922201 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:10.922026    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/318b3bc6-6e59-4211-af63-b116526a325d-sys-fs\") pod \"aws-ebs-csi-driver-node-snxz4\" (UID: \"318b3bc6-6e59-4211-af63-b116526a325d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-snxz4"
Oct 31 01:03:10.922201 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:10.922061    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/58b0b75a-1568-4703-ac55-fa87f29225ad-host-var-lib-cni-bin\") pod \"multus-hxh6j\" (UID: \"58b0b75a-1568-4703-ac55-fa87f29225ad\") " pod="openshift-multus/multus-hxh6j"
Oct 31 01:03:10.922201 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:10.922084    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d55eb70a-f1a8-4346-b8df-ccd2c598dd6b-host-slash\") pod \"ovnkube-node-pghw2\" (UID: \"d55eb70a-f1a8-4346-b8df-ccd2c598dd6b\") " pod="openshift-ovn-kubernetes/ovnkube-node-pghw2"
Oct 31 01:03:10.922201 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:10.922102    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d55eb70a-f1a8-4346-b8df-ccd2c598dd6b-etc-openvswitch\") pod \"ovnkube-node-pghw2\" (UID: \"d55eb70a-f1a8-4346-b8df-ccd2c598dd6b\") " pod="openshift-ovn-kubernetes/ovnkube-node-pghw2"
Oct 31 01:03:10.922201 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:10.922126    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/08490837-b3b6-4c2b-a498-568af0d7086d-sys\") pod \"tuned-45bvr\" (UID: \"08490837-b3b6-4c2b-a498-568af0d7086d\") " pod="openshift-cluster-node-tuning-operator/tuned-45bvr"
Oct 31 01:03:10.922201 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:10.922150    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/58b0b75a-1568-4703-ac55-fa87f29225ad-cni-binary-copy\") pod \"multus-hxh6j\" (UID: \"58b0b75a-1568-4703-ac55-fa87f29225ad\") " pod="openshift-multus/multus-hxh6j"
Oct 31 01:03:10.922201 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:10.922182    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/58b0b75a-1568-4703-ac55-fa87f29225ad-host-run-k8s-cni-cncf-io\") pod \"multus-hxh6j\" (UID: \"58b0b75a-1568-4703-ac55-fa87f29225ad\") " pod="openshift-multus/multus-hxh6j"
Oct 31 01:03:10.922630 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:10.922224    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/58b0b75a-1568-4703-ac55-fa87f29225ad-multus-conf-dir\") pod \"multus-hxh6j\" (UID: \"58b0b75a-1568-4703-ac55-fa87f29225ad\") " pod="openshift-multus/multus-hxh6j"
Oct 31 01:03:10.922630 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:10.922263    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d55eb70a-f1a8-4346-b8df-ccd2c598dd6b-run-openvswitch\") pod \"ovnkube-node-pghw2\" (UID: \"d55eb70a-f1a8-4346-b8df-ccd2c598dd6b\") " pod="openshift-ovn-kubernetes/ovnkube-node-pghw2"
Oct 31 01:03:10.922630 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:10.922294    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d55eb70a-f1a8-4346-b8df-ccd2c598dd6b-host-run-ovn-kubernetes\") pod \"ovnkube-node-pghw2\" (UID: \"d55eb70a-f1a8-4346-b8df-ccd2c598dd6b\") " pod="openshift-ovn-kubernetes/ovnkube-node-pghw2"
Oct 31 01:03:10.922630 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:10.922317    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/08490837-b3b6-4c2b-a498-568af0d7086d-etc-sysctl-conf\") pod \"tuned-45bvr\" (UID: \"08490837-b3b6-4c2b-a498-568af0d7086d\") " pod="openshift-cluster-node-tuning-operator/tuned-45bvr"
Oct 31 01:03:10.922630 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:10.922353    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/08490837-b3b6-4c2b-a498-568af0d7086d-host\") pod \"tuned-45bvr\" (UID: \"08490837-b3b6-4c2b-a498-568af0d7086d\") " pod="openshift-cluster-node-tuning-operator/tuned-45bvr"
Oct 31 01:03:10.922630 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:10.922376    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/dd8134f9-b2a6-4542-82ec-c10edc3fcd72-serviceca\") pod \"node-ca-rlxt2\" (UID: \"dd8134f9-b2a6-4542-82ec-c10edc3fcd72\") " pod="openshift-image-registry/node-ca-rlxt2"
Oct 31 01:03:10.922630 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:10.922406    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/d55eb70a-f1a8-4346-b8df-ccd2c598dd6b-systemd-units\") pod \"ovnkube-node-pghw2\" (UID: \"d55eb70a-f1a8-4346-b8df-ccd2c598dd6b\") " pod="openshift-ovn-kubernetes/ovnkube-node-pghw2"
Oct 31 01:03:10.922630 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:10.922447    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/d55eb70a-f1a8-4346-b8df-ccd2c598dd6b-run-systemd\") pod \"ovnkube-node-pghw2\" (UID: \"d55eb70a-f1a8-4346-b8df-ccd2c598dd6b\") " pod="openshift-ovn-kubernetes/ovnkube-node-pghw2"
Oct 31 01:03:10.922630 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:10.922469    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/fa085dd8-ca1e-4ec5-acb2-9b9dc878bf8a-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-jwthj\" (UID: \"fa085dd8-ca1e-4ec5-acb2-9b9dc878bf8a\") " pod="openshift-multus/multus-additional-cni-plugins-jwthj"
Oct 31 01:03:10.922630 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:10.922508    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/318b3bc6-6e59-4211-af63-b116526a325d-socket-dir\") pod \"aws-ebs-csi-driver-node-snxz4\" (UID: \"318b3bc6-6e59-4211-af63-b116526a325d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-snxz4"
Oct 31 01:03:10.922630 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:10.922532    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/58b0b75a-1568-4703-ac55-fa87f29225ad-cnibin\") pod \"multus-hxh6j\" (UID: \"58b0b75a-1568-4703-ac55-fa87f29225ad\") " pod="openshift-multus/multus-hxh6j"
Oct 31 01:03:10.922630 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:10.922580    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/58b0b75a-1568-4703-ac55-fa87f29225ad-host-run-multus-certs\") pod \"multus-hxh6j\" (UID: \"58b0b75a-1568-4703-ac55-fa87f29225ad\") " pod="openshift-multus/multus-hxh6j"
Oct 31 01:03:10.922630 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:10.922608    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t228s\" (UniqueName: \"kubernetes.io/projected/58b0b75a-1568-4703-ac55-fa87f29225ad-kube-api-access-t228s\") pod \"multus-hxh6j\" (UID: \"58b0b75a-1568-4703-ac55-fa87f29225ad\") " pod="openshift-multus/multus-hxh6j"
Oct 31 01:03:10.922630 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:10.922633    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/d55eb70a-f1a8-4346-b8df-ccd2c598dd6b-log-socket\") pod \"ovnkube-node-pghw2\" (UID: \"d55eb70a-f1a8-4346-b8df-ccd2c598dd6b\") " pod="openshift-ovn-kubernetes/ovnkube-node-pghw2"
Oct 31 01:03:10.923249 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:10.922653    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/fa085dd8-ca1e-4ec5-acb2-9b9dc878bf8a-cni-binary-copy\") pod \"multus-additional-cni-plugins-jwthj\" (UID: \"fa085dd8-ca1e-4ec5-acb2-9b9dc878bf8a\") " pod="openshift-multus/multus-additional-cni-plugins-jwthj"
Oct 31 01:03:10.923249 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:10.922656    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/d55eb70a-f1a8-4346-b8df-ccd2c598dd6b-env-overrides\") pod \"ovnkube-node-pghw2\" (UID: \"d55eb70a-f1a8-4346-b8df-ccd2c598dd6b\") " pod="openshift-ovn-kubernetes/ovnkube-node-pghw2"
Oct 31 01:03:10.923249 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:10.922726    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/fa085dd8-ca1e-4ec5-acb2-9b9dc878bf8a-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-jwthj\" (UID: \"fa085dd8-ca1e-4ec5-acb2-9b9dc878bf8a\") " pod="openshift-multus/multus-additional-cni-plugins-jwthj"
Oct 31 01:03:10.923249 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:10.922758    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/dd8134f9-b2a6-4542-82ec-c10edc3fcd72-serviceca\") pod \"node-ca-rlxt2\" (UID: \"dd8134f9-b2a6-4542-82ec-c10edc3fcd72\") " pod="openshift-image-registry/node-ca-rlxt2"
Oct 31 01:03:10.923249 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:10.922759    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/318b3bc6-6e59-4211-af63-b116526a325d-device-dir\") pod \"aws-ebs-csi-driver-node-snxz4\" (UID: \"318b3bc6-6e59-4211-af63-b116526a325d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-snxz4"
Oct 31 01:03:10.923249 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:10.922801    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/f8855c2c-87b4-4b9e-b28a-227981205b11-agent-certs\") pod \"konnectivity-agent-78szg\" (UID: \"f8855c2c-87b4-4b9e-b28a-227981205b11\") " pod="kube-system/konnectivity-agent-78szg"
Oct 31 01:03:10.923249 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:10.922824    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/58b0b75a-1568-4703-ac55-fa87f29225ad-host-var-lib-kubelet\") pod \"multus-hxh6j\" (UID: \"58b0b75a-1568-4703-ac55-fa87f29225ad\") " pod="openshift-multus/multus-hxh6j"
Oct 31 01:03:10.923249 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:10.922854    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/d55eb70a-f1a8-4346-b8df-ccd2c598dd6b-node-log\") pod \"ovnkube-node-pghw2\" (UID: \"d55eb70a-f1a8-4346-b8df-ccd2c598dd6b\") " pod="openshift-ovn-kubernetes/ovnkube-node-pghw2"
Oct 31 01:03:10.923249 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:10.922886    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/08490837-b3b6-4c2b-a498-568af0d7086d-etc-sysconfig\") pod \"tuned-45bvr\" (UID: \"08490837-b3b6-4c2b-a498-568af0d7086d\") " pod="openshift-cluster-node-tuning-operator/tuned-45bvr"
Oct 31 01:03:10.923249 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:10.922919    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/aa4836a5-c71a-485d-82fc-f8b53dae7717-original-pull-secret\") pod \"global-pull-secret-syncer-zs5g7\" (UID: \"aa4836a5-c71a-485d-82fc-f8b53dae7717\") " pod="kube-system/global-pull-secret-syncer-zs5g7"
Oct 31 01:03:10.923249 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:10.922943    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wsrb6\" (UniqueName: \"kubernetes.io/projected/dd8134f9-b2a6-4542-82ec-c10edc3fcd72-kube-api-access-wsrb6\") pod \"node-ca-rlxt2\" (UID: \"dd8134f9-b2a6-4542-82ec-c10edc3fcd72\") " pod="openshift-image-registry/node-ca-rlxt2"
Oct 31 01:03:10.923249 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:10.922950    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/fa085dd8-ca1e-4ec5-acb2-9b9dc878bf8a-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-jwthj\" (UID: \"fa085dd8-ca1e-4ec5-acb2-9b9dc878bf8a\") " pod="openshift-multus/multus-additional-cni-plugins-jwthj"
Oct 31 01:03:10.923249 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:10.922976    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nmlm7\" (UniqueName: \"kubernetes.io/projected/fa085dd8-ca1e-4ec5-acb2-9b9dc878bf8a-kube-api-access-nmlm7\") pod \"multus-additional-cni-plugins-jwthj\" (UID: \"fa085dd8-ca1e-4ec5-acb2-9b9dc878bf8a\") " pod="openshift-multus/multus-additional-cni-plugins-jwthj"
Oct 31 01:03:10.923249 ip-10-0-133-197 kubenswrapper[2578]: E1031 01:03:10.923072    2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 31 01:03:10.923249 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:10.923100    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/d55eb70a-f1a8-4346-b8df-ccd2c598dd6b-ovn-node-metrics-cert\") pod \"ovnkube-node-pghw2\" (UID: \"d55eb70a-f1a8-4346-b8df-ccd2c598dd6b\") " pod="openshift-ovn-kubernetes/ovnkube-node-pghw2"
Oct 31 01:03:10.923249 ip-10-0-133-197 kubenswrapper[2578]: E1031 01:03:10.923131    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/aa4836a5-c71a-485d-82fc-f8b53dae7717-original-pull-secret podName:aa4836a5-c71a-485d-82fc-f8b53dae7717 nodeName:}" failed. No retries permitted until 2026-10-31 01:03:11.423111665 +0000 UTC m=+3.108591078 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/aa4836a5-c71a-485d-82fc-f8b53dae7717-original-pull-secret") pod "global-pull-secret-syncer-zs5g7" (UID: "aa4836a5-c71a-485d-82fc-f8b53dae7717") : object "kube-system"/"original-pull-secret" not registered
Oct 31 01:03:10.923843 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:10.923143    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/fa085dd8-ca1e-4ec5-acb2-9b9dc878bf8a-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-jwthj\" (UID: \"fa085dd8-ca1e-4ec5-acb2-9b9dc878bf8a\") " pod="openshift-multus/multus-additional-cni-plugins-jwthj"
Oct 31 01:03:10.923843 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:10.923160    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/d55eb70a-f1a8-4346-b8df-ccd2c598dd6b-ovnkube-script-lib\") pod \"ovnkube-node-pghw2\" (UID: \"d55eb70a-f1a8-4346-b8df-ccd2c598dd6b\") " pod="openshift-ovn-kubernetes/ovnkube-node-pghw2"
Oct 31 01:03:10.923843 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:10.923182    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/08490837-b3b6-4c2b-a498-568af0d7086d-etc-modprobe-d\") pod \"tuned-45bvr\" (UID: \"08490837-b3b6-4c2b-a498-568af0d7086d\") " pod="openshift-cluster-node-tuning-operator/tuned-45bvr"
Oct 31 01:03:10.923843 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:10.923230    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/fa085dd8-ca1e-4ec5-acb2-9b9dc878bf8a-cnibin\") pod \"multus-additional-cni-plugins-jwthj\" (UID: \"fa085dd8-ca1e-4ec5-acb2-9b9dc878bf8a\") " pod="openshift-multus/multus-additional-cni-plugins-jwthj"
Oct 31 01:03:10.923843 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:10.923249    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/58b0b75a-1568-4703-ac55-fa87f29225ad-multus-cni-dir\") pod \"multus-hxh6j\" (UID: \"58b0b75a-1568-4703-ac55-fa87f29225ad\") " pod="openshift-multus/multus-hxh6j"
Oct 31 01:03:10.923843 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:10.923291    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/58b0b75a-1568-4703-ac55-fa87f29225ad-host-var-lib-cni-multus\") pod \"multus-hxh6j\" (UID: \"58b0b75a-1568-4703-ac55-fa87f29225ad\") " pod="openshift-multus/multus-hxh6j"
Oct 31 01:03:10.923843 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:10.923302    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/fa085dd8-ca1e-4ec5-acb2-9b9dc878bf8a-cnibin\") pod \"multus-additional-cni-plugins-jwthj\" (UID: \"fa085dd8-ca1e-4ec5-acb2-9b9dc878bf8a\") " pod="openshift-multus/multus-additional-cni-plugins-jwthj"
Oct 31 01:03:10.923843 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:10.923339    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d55eb70a-f1a8-4346-b8df-ccd2c598dd6b-var-lib-openvswitch\") pod \"ovnkube-node-pghw2\" (UID: \"d55eb70a-f1a8-4346-b8df-ccd2c598dd6b\") " pod="openshift-ovn-kubernetes/ovnkube-node-pghw2"
Oct 31 01:03:10.923843 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:10.923363    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/318b3bc6-6e59-4211-af63-b116526a325d-etc-selinux\") pod \"aws-ebs-csi-driver-node-snxz4\" (UID: \"318b3bc6-6e59-4211-af63-b116526a325d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-snxz4"
Oct 31 01:03:10.923843 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:10.923384    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2dx97\" (UniqueName: \"kubernetes.io/projected/57c2f196-0d8a-4bce-a5bc-3b6b5c0d808c-kube-api-access-2dx97\") pod \"node-resolver-j46hv\" (UID: \"57c2f196-0d8a-4bce-a5bc-3b6b5c0d808c\") " pod="openshift-dns/node-resolver-j46hv"
Oct 31 01:03:10.923843 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:10.923408    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/08490837-b3b6-4c2b-a498-568af0d7086d-etc-sysctl-d\") pod \"tuned-45bvr\" (UID: \"08490837-b3b6-4c2b-a498-568af0d7086d\") " pod="openshift-cluster-node-tuning-operator/tuned-45bvr"
Oct 31 01:03:10.923843 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:10.923427    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/aa4836a5-c71a-485d-82fc-f8b53dae7717-kubelet-config\") pod \"global-pull-secret-syncer-zs5g7\" (UID: \"aa4836a5-c71a-485d-82fc-f8b53dae7717\") " pod="kube-system/global-pull-secret-syncer-zs5g7"
Oct 31 01:03:10.923843 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:10.923443    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/58b0b75a-1568-4703-ac55-fa87f29225ad-multus-socket-dir-parent\") pod \"multus-hxh6j\" (UID: \"58b0b75a-1568-4703-ac55-fa87f29225ad\") " pod="openshift-multus/multus-hxh6j"
Oct 31 01:03:10.923843 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:10.923462    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/d55eb70a-f1a8-4346-b8df-ccd2c598dd6b-host-kubelet\") pod \"ovnkube-node-pghw2\" (UID: \"d55eb70a-f1a8-4346-b8df-ccd2c598dd6b\") " pod="openshift-ovn-kubernetes/ovnkube-node-pghw2"
Oct 31 01:03:10.923843 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:10.923480    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/aa4836a5-c71a-485d-82fc-f8b53dae7717-kubelet-config\") pod \"global-pull-secret-syncer-zs5g7\" (UID: \"aa4836a5-c71a-485d-82fc-f8b53dae7717\") " pod="kube-system/global-pull-secret-syncer-zs5g7"
Oct 31 01:03:10.923843 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:10.923488    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d55eb70a-f1a8-4346-b8df-ccd2c598dd6b-host-run-netns\") pod \"ovnkube-node-pghw2\" (UID: \"d55eb70a-f1a8-4346-b8df-ccd2c598dd6b\") " pod="openshift-ovn-kubernetes/ovnkube-node-pghw2"
Oct 31 01:03:10.924424 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:10.923518    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ftdwt\" (UniqueName: \"kubernetes.io/projected/d55eb70a-f1a8-4346-b8df-ccd2c598dd6b-kube-api-access-ftdwt\") pod \"ovnkube-node-pghw2\" (UID: \"d55eb70a-f1a8-4346-b8df-ccd2c598dd6b\") " pod="openshift-ovn-kubernetes/ovnkube-node-pghw2"
Oct 31 01:03:10.924424 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:10.923536    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rms4s\" (UniqueName: \"kubernetes.io/projected/318b3bc6-6e59-4211-af63-b116526a325d-kube-api-access-rms4s\") pod \"aws-ebs-csi-driver-node-snxz4\" (UID: \"318b3bc6-6e59-4211-af63-b116526a325d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-snxz4"
Oct 31 01:03:10.924424 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:10.923555    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/58b0b75a-1568-4703-ac55-fa87f29225ad-hostroot\") pod \"multus-hxh6j\" (UID: \"58b0b75a-1568-4703-ac55-fa87f29225ad\") " pod="openshift-multus/multus-hxh6j"
Oct 31 01:03:10.924424 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:10.923571    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/08490837-b3b6-4c2b-a498-568af0d7086d-tmp\") pod \"tuned-45bvr\" (UID: \"08490837-b3b6-4c2b-a498-568af0d7086d\") " pod="openshift-cluster-node-tuning-operator/tuned-45bvr"
Oct 31 01:03:10.924424 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:10.923632    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/9cd1266e-c630-4259-b1f7-1b88dbde7385-iptables-alerter-script\") pod \"iptables-alerter-ssw8w\" (UID: \"9cd1266e-c630-4259-b1f7-1b88dbde7385\") " pod="openshift-network-operator/iptables-alerter-ssw8w"
Oct 31 01:03:10.924424 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:10.923666    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9cd1266e-c630-4259-b1f7-1b88dbde7385-host-slash\") pod \"iptables-alerter-ssw8w\" (UID: \"9cd1266e-c630-4259-b1f7-1b88dbde7385\") " pod="openshift-network-operator/iptables-alerter-ssw8w"
Oct 31 01:03:10.924424 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:10.923744    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9cd1266e-c630-4259-b1f7-1b88dbde7385-host-slash\") pod \"iptables-alerter-ssw8w\" (UID: \"9cd1266e-c630-4259-b1f7-1b88dbde7385\") " pod="openshift-network-operator/iptables-alerter-ssw8w"
Oct 31 01:03:10.924424 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:10.923755    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jxmsl\" (UniqueName: \"kubernetes.io/projected/9cd1266e-c630-4259-b1f7-1b88dbde7385-kube-api-access-jxmsl\") pod \"iptables-alerter-ssw8w\" (UID: \"9cd1266e-c630-4259-b1f7-1b88dbde7385\") " pod="openshift-network-operator/iptables-alerter-ssw8w"
Oct 31 01:03:10.924424 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:10.923915    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/aa4836a5-c71a-485d-82fc-f8b53dae7717-dbus\") pod \"global-pull-secret-syncer-zs5g7\" (UID: \"aa4836a5-c71a-485d-82fc-f8b53dae7717\") " pod="kube-system/global-pull-secret-syncer-zs5g7"
Oct 31 01:03:10.924424 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:10.923955    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-srkbh\" (UniqueName: \"kubernetes.io/projected/8952fc79-c5a6-42ba-a296-8cb7a776f718-kube-api-access-srkbh\") pod \"network-metrics-daemon-wdsmn\" (UID: \"8952fc79-c5a6-42ba-a296-8cb7a776f718\") " pod="openshift-multus/network-metrics-daemon-wdsmn"
Oct 31 01:03:10.924424 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:10.923981    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/57c2f196-0d8a-4bce-a5bc-3b6b5c0d808c-tmp-dir\") pod \"node-resolver-j46hv\" (UID: \"57c2f196-0d8a-4bce-a5bc-3b6b5c0d808c\") " pod="openshift-dns/node-resolver-j46hv"
Oct 31 01:03:10.924424 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:10.924008    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d55eb70a-f1a8-4346-b8df-ccd2c598dd6b-host-cni-bin\") pod \"ovnkube-node-pghw2\" (UID: \"d55eb70a-f1a8-4346-b8df-ccd2c598dd6b\") " pod="openshift-ovn-kubernetes/ovnkube-node-pghw2"
Oct 31 01:03:10.924424 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:10.924080    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/aa4836a5-c71a-485d-82fc-f8b53dae7717-dbus\") pod \"global-pull-secret-syncer-zs5g7\" (UID: \"aa4836a5-c71a-485d-82fc-f8b53dae7717\") " pod="kube-system/global-pull-secret-syncer-zs5g7"
Oct 31 01:03:10.924424 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:10.924157    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/dd8134f9-b2a6-4542-82ec-c10edc3fcd72-host\") pod \"node-ca-rlxt2\" (UID: \"dd8134f9-b2a6-4542-82ec-c10edc3fcd72\") " pod="openshift-image-registry/node-ca-rlxt2"
Oct 31 01:03:10.924424 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:10.924186    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r6zx9\" (UniqueName: \"kubernetes.io/projected/565fefaf-82aa-42a8-9cc7-6517c4bddf1e-kube-api-access-r6zx9\") pod \"network-check-target-8mqbd\" (UID: \"565fefaf-82aa-42a8-9cc7-6517c4bddf1e\") " pod="openshift-network-diagnostics/network-check-target-8mqbd"
Oct 31 01:03:10.924424 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:10.924194    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/9cd1266e-c630-4259-b1f7-1b88dbde7385-iptables-alerter-script\") pod \"iptables-alerter-ssw8w\" (UID: \"9cd1266e-c630-4259-b1f7-1b88dbde7385\") " pod="openshift-network-operator/iptables-alerter-ssw8w"
Oct 31 01:03:10.924424 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:10.924213    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/f8855c2c-87b4-4b9e-b28a-227981205b11-konnectivity-ca\") pod \"konnectivity-agent-78szg\" (UID: \"f8855c2c-87b4-4b9e-b28a-227981205b11\") " pod="kube-system/konnectivity-agent-78szg"
Oct 31 01:03:10.925029 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:10.924237    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/58b0b75a-1568-4703-ac55-fa87f29225ad-host-run-netns\") pod \"multus-hxh6j\" (UID: \"58b0b75a-1568-4703-ac55-fa87f29225ad\") " pod="openshift-multus/multus-hxh6j"
Oct 31 01:03:10.925029 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:10.924353    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/08490837-b3b6-4c2b-a498-568af0d7086d-lib-modules\") pod \"tuned-45bvr\" (UID: \"08490837-b3b6-4c2b-a498-568af0d7086d\") " pod="openshift-cluster-node-tuning-operator/tuned-45bvr"
Oct 31 01:03:10.925029 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:10.924364    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/dd8134f9-b2a6-4542-82ec-c10edc3fcd72-host\") pod \"node-ca-rlxt2\" (UID: \"dd8134f9-b2a6-4542-82ec-c10edc3fcd72\") " pod="openshift-image-registry/node-ca-rlxt2"
Oct 31 01:03:10.925029 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:10.924422    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/08490837-b3b6-4c2b-a498-568af0d7086d-var-lib-kubelet\") pod \"tuned-45bvr\" (UID: \"08490837-b3b6-4c2b-a498-568af0d7086d\") " pod="openshift-cluster-node-tuning-operator/tuned-45bvr"
Oct 31 01:03:10.925029 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:10.924447    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/fa085dd8-ca1e-4ec5-acb2-9b9dc878bf8a-tuning-conf-dir\") pod \"multus-additional-cni-plugins-jwthj\" (UID: \"fa085dd8-ca1e-4ec5-acb2-9b9dc878bf8a\") " pod="openshift-multus/multus-additional-cni-plugins-jwthj"
Oct 31 01:03:10.925029 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:10.924532    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/58b0b75a-1568-4703-ac55-fa87f29225ad-etc-kubernetes\") pod \"multus-hxh6j\" (UID: \"58b0b75a-1568-4703-ac55-fa87f29225ad\") " pod="openshift-multus/multus-hxh6j"
Oct 31 01:03:10.925029 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:10.924583    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/d55eb70a-f1a8-4346-b8df-ccd2c598dd6b-host-cni-netd\") pod \"ovnkube-node-pghw2\" (UID: \"d55eb70a-f1a8-4346-b8df-ccd2c598dd6b\") " pod="openshift-ovn-kubernetes/ovnkube-node-pghw2"
Oct 31 01:03:10.925029 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:10.924590    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/fa085dd8-ca1e-4ec5-acb2-9b9dc878bf8a-tuning-conf-dir\") pod \"multus-additional-cni-plugins-jwthj\" (UID: \"fa085dd8-ca1e-4ec5-acb2-9b9dc878bf8a\") " pod="openshift-multus/multus-additional-cni-plugins-jwthj"
Oct 31 01:03:10.925029 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:10.924672    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d55eb70a-f1a8-4346-b8df-ccd2c598dd6b-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-pghw2\" (UID: \"d55eb70a-f1a8-4346-b8df-ccd2c598dd6b\") " pod="openshift-ovn-kubernetes/ovnkube-node-pghw2"
Oct 31 01:03:10.925029 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:10.924694    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/08490837-b3b6-4c2b-a498-568af0d7086d-etc-systemd\") pod \"tuned-45bvr\" (UID: \"08490837-b3b6-4c2b-a498-568af0d7086d\") " pod="openshift-cluster-node-tuning-operator/tuned-45bvr"
Oct 31 01:03:10.925029 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:10.924709    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w9tzc\" (UniqueName: \"kubernetes.io/projected/08490837-b3b6-4c2b-a498-568af0d7086d-kube-api-access-w9tzc\") pod \"tuned-45bvr\" (UID: \"08490837-b3b6-4c2b-a498-568af0d7086d\") " pod="openshift-cluster-node-tuning-operator/tuned-45bvr"
Oct 31 01:03:10.925029 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:10.924730    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/318b3bc6-6e59-4211-af63-b116526a325d-kubelet-dir\") pod \"aws-ebs-csi-driver-node-snxz4\" (UID: \"318b3bc6-6e59-4211-af63-b116526a325d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-snxz4"
Oct 31 01:03:10.925029 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:10.924756    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/58b0b75a-1568-4703-ac55-fa87f29225ad-system-cni-dir\") pod \"multus-hxh6j\" (UID: \"58b0b75a-1568-4703-ac55-fa87f29225ad\") " pod="openshift-multus/multus-hxh6j"
Oct 31 01:03:10.925029 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:10.924793    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/d55eb70a-f1a8-4346-b8df-ccd2c598dd6b-ovnkube-config\") pod \"ovnkube-node-pghw2\" (UID: \"d55eb70a-f1a8-4346-b8df-ccd2c598dd6b\") " pod="openshift-ovn-kubernetes/ovnkube-node-pghw2"
Oct 31 01:03:10.925029 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:10.924817    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/08490837-b3b6-4c2b-a498-568af0d7086d-run\") pod \"tuned-45bvr\" (UID: \"08490837-b3b6-4c2b-a498-568af0d7086d\") " pod="openshift-cluster-node-tuning-operator/tuned-45bvr"
Oct 31 01:03:10.952898 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:10.952875    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Oct 31 01:03:10.952898 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:10.952879    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Oct 31 01:03:10.953038 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:10.952963    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Oct 31 01:03:10.953038 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:10.952997    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-t8rwd\""
Oct 31 01:03:10.953124 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:10.953115    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-rnpg9\""
Oct 31 01:03:10.957060 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:10.957045    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Oct 31 01:03:10.957124 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:10.957086    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Oct 31 01:03:10.957124 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:10.957113    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Oct 31 01:03:10.961102 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:10.961086    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Oct 31 01:03:10.961178 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:10.961137    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Oct 31 01:03:10.961178 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:10.961135    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Oct 31 01:03:10.961286 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:10.961172    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Oct 31 01:03:10.961286 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:10.961192    2578 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Oct 31 01:03:10.961286 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:10.961226    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-h9z86\""
Oct 31 01:03:10.961286 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:10.961252    2578 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-bvqx9"
Oct 31 01:03:10.982253 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:10.982226    2578 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Oct 31 01:03:10.982365 ip-10-0-133-197 kubenswrapper[2578]: E1031 01:03:10.982287    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 31 01:03:10.982365 ip-10-0-133-197 kubenswrapper[2578]: E1031 01:03:10.982300    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 31 01:03:10.982365 ip-10-0-133-197 kubenswrapper[2578]: E1031 01:03:10.982309    2578 projected.go:196] Error preparing data for projected volume kube-api-access-r6zx9 for pod openshift-network-diagnostics/network-check-target-8mqbd: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 31 01:03:10.982530 ip-10-0-133-197 kubenswrapper[2578]: E1031 01:03:10.982374    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/565fefaf-82aa-42a8-9cc7-6517c4bddf1e-kube-api-access-r6zx9 podName:565fefaf-82aa-42a8-9cc7-6517c4bddf1e nodeName:}" failed. No retries permitted until 2026-10-31 01:03:11.482360962 +0000 UTC m=+3.167840374 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-r6zx9" (UniqueName: "kubernetes.io/projected/565fefaf-82aa-42a8-9cc7-6517c4bddf1e-kube-api-access-r6zx9") pod "network-check-target-8mqbd" (UID: "565fefaf-82aa-42a8-9cc7-6517c4bddf1e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 31 01:03:10.985757 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:10.985736    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jxmsl\" (UniqueName: \"kubernetes.io/projected/9cd1266e-c630-4259-b1f7-1b88dbde7385-kube-api-access-jxmsl\") pod \"iptables-alerter-ssw8w\" (UID: \"9cd1266e-c630-4259-b1f7-1b88dbde7385\") " pod="openshift-network-operator/iptables-alerter-ssw8w"
Oct 31 01:03:10.985843 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:10.985739    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nmlm7\" (UniqueName: \"kubernetes.io/projected/fa085dd8-ca1e-4ec5-acb2-9b9dc878bf8a-kube-api-access-nmlm7\") pod \"multus-additional-cni-plugins-jwthj\" (UID: \"fa085dd8-ca1e-4ec5-acb2-9b9dc878bf8a\") " pod="openshift-multus/multus-additional-cni-plugins-jwthj"
Oct 31 01:03:10.985843 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:10.985741    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wsrb6\" (UniqueName: \"kubernetes.io/projected/dd8134f9-b2a6-4542-82ec-c10edc3fcd72-kube-api-access-wsrb6\") pod \"node-ca-rlxt2\" (UID: \"dd8134f9-b2a6-4542-82ec-c10edc3fcd72\") " pod="openshift-image-registry/node-ca-rlxt2"
Oct 31 01:03:11.025249 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:11.025222    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/d55eb70a-f1a8-4346-b8df-ccd2c598dd6b-run-systemd\") pod \"ovnkube-node-pghw2\" (UID: \"d55eb70a-f1a8-4346-b8df-ccd2c598dd6b\") " pod="openshift-ovn-kubernetes/ovnkube-node-pghw2"
Oct 31 01:03:11.025358 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:11.025261    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/318b3bc6-6e59-4211-af63-b116526a325d-socket-dir\") pod \"aws-ebs-csi-driver-node-snxz4\" (UID: \"318b3bc6-6e59-4211-af63-b116526a325d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-snxz4"
Oct 31 01:03:11.025358 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:11.025285    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/58b0b75a-1568-4703-ac55-fa87f29225ad-cnibin\") pod \"multus-hxh6j\" (UID: \"58b0b75a-1568-4703-ac55-fa87f29225ad\") " pod="openshift-multus/multus-hxh6j"
Oct 31 01:03:11.025358 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:11.025301    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/58b0b75a-1568-4703-ac55-fa87f29225ad-host-run-multus-certs\") pod \"multus-hxh6j\" (UID: \"58b0b75a-1568-4703-ac55-fa87f29225ad\") " pod="openshift-multus/multus-hxh6j"
Oct 31 01:03:11.025358 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:11.025312    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/d55eb70a-f1a8-4346-b8df-ccd2c598dd6b-run-systemd\") pod \"ovnkube-node-pghw2\" (UID: \"d55eb70a-f1a8-4346-b8df-ccd2c598dd6b\") " pod="openshift-ovn-kubernetes/ovnkube-node-pghw2"
Oct 31 01:03:11.025358 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:11.025326    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t228s\" (UniqueName: \"kubernetes.io/projected/58b0b75a-1568-4703-ac55-fa87f29225ad-kube-api-access-t228s\") pod \"multus-hxh6j\" (UID: \"58b0b75a-1568-4703-ac55-fa87f29225ad\") " pod="openshift-multus/multus-hxh6j"
Oct 31 01:03:11.025609 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:11.025368    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/d55eb70a-f1a8-4346-b8df-ccd2c598dd6b-log-socket\") pod \"ovnkube-node-pghw2\" (UID: \"d55eb70a-f1a8-4346-b8df-ccd2c598dd6b\") " pod="openshift-ovn-kubernetes/ovnkube-node-pghw2"
Oct 31 01:03:11.025609 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:11.025375    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/58b0b75a-1568-4703-ac55-fa87f29225ad-cnibin\") pod \"multus-hxh6j\" (UID: \"58b0b75a-1568-4703-ac55-fa87f29225ad\") " pod="openshift-multus/multus-hxh6j"
Oct 31 01:03:11.025609 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:11.025375    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/58b0b75a-1568-4703-ac55-fa87f29225ad-host-run-multus-certs\") pod \"multus-hxh6j\" (UID: \"58b0b75a-1568-4703-ac55-fa87f29225ad\") " pod="openshift-multus/multus-hxh6j"
Oct 31 01:03:11.025609 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:11.025396    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/318b3bc6-6e59-4211-af63-b116526a325d-socket-dir\") pod \"aws-ebs-csi-driver-node-snxz4\" (UID: \"318b3bc6-6e59-4211-af63-b116526a325d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-snxz4"
Oct 31 01:03:11.025609 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:11.025394    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/d55eb70a-f1a8-4346-b8df-ccd2c598dd6b-env-overrides\") pod \"ovnkube-node-pghw2\" (UID: \"d55eb70a-f1a8-4346-b8df-ccd2c598dd6b\") " pod="openshift-ovn-kubernetes/ovnkube-node-pghw2"
Oct 31 01:03:11.025609 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:11.025448    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/d55eb70a-f1a8-4346-b8df-ccd2c598dd6b-log-socket\") pod \"ovnkube-node-pghw2\" (UID: \"d55eb70a-f1a8-4346-b8df-ccd2c598dd6b\") " pod="openshift-ovn-kubernetes/ovnkube-node-pghw2"
Oct 31 01:03:11.025609 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:11.025448    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/318b3bc6-6e59-4211-af63-b116526a325d-device-dir\") pod \"aws-ebs-csi-driver-node-snxz4\" (UID: \"318b3bc6-6e59-4211-af63-b116526a325d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-snxz4"
Oct 31 01:03:11.025609 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:11.025483    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/f8855c2c-87b4-4b9e-b28a-227981205b11-agent-certs\") pod \"konnectivity-agent-78szg\" (UID: \"f8855c2c-87b4-4b9e-b28a-227981205b11\") " pod="kube-system/konnectivity-agent-78szg"
Oct 31 01:03:11.025609 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:11.025498    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/58b0b75a-1568-4703-ac55-fa87f29225ad-host-var-lib-kubelet\") pod \"multus-hxh6j\" (UID: \"58b0b75a-1568-4703-ac55-fa87f29225ad\") " pod="openshift-multus/multus-hxh6j"
Oct 31 01:03:11.025609 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:11.025520    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/d55eb70a-f1a8-4346-b8df-ccd2c598dd6b-node-log\") pod \"ovnkube-node-pghw2\" (UID: \"d55eb70a-f1a8-4346-b8df-ccd2c598dd6b\") " pod="openshift-ovn-kubernetes/ovnkube-node-pghw2"
Oct 31 01:03:11.025609 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:11.025541    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/08490837-b3b6-4c2b-a498-568af0d7086d-etc-sysconfig\") pod \"tuned-45bvr\" (UID: \"08490837-b3b6-4c2b-a498-568af0d7086d\") " pod="openshift-cluster-node-tuning-operator/tuned-45bvr"
Oct 31 01:03:11.025609 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:11.025571    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/d55eb70a-f1a8-4346-b8df-ccd2c598dd6b-ovn-node-metrics-cert\") pod \"ovnkube-node-pghw2\" (UID: \"d55eb70a-f1a8-4346-b8df-ccd2c598dd6b\") " pod="openshift-ovn-kubernetes/ovnkube-node-pghw2"
Oct 31 01:03:11.025609 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:11.025587    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/58b0b75a-1568-4703-ac55-fa87f29225ad-host-var-lib-kubelet\") pod \"multus-hxh6j\" (UID: \"58b0b75a-1568-4703-ac55-fa87f29225ad\") " pod="openshift-multus/multus-hxh6j"
Oct 31 01:03:11.025609 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:11.025606    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/d55eb70a-f1a8-4346-b8df-ccd2c598dd6b-node-log\") pod \"ovnkube-node-pghw2\" (UID: \"d55eb70a-f1a8-4346-b8df-ccd2c598dd6b\") " pod="openshift-ovn-kubernetes/ovnkube-node-pghw2"
Oct 31 01:03:11.026228 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:11.025636    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/318b3bc6-6e59-4211-af63-b116526a325d-device-dir\") pod \"aws-ebs-csi-driver-node-snxz4\" (UID: \"318b3bc6-6e59-4211-af63-b116526a325d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-snxz4"
Oct 31 01:03:11.026228 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:11.025653    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/08490837-b3b6-4c2b-a498-568af0d7086d-etc-sysconfig\") pod \"tuned-45bvr\" (UID: \"08490837-b3b6-4c2b-a498-568af0d7086d\") " pod="openshift-cluster-node-tuning-operator/tuned-45bvr"
Oct 31 01:03:11.026228 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:11.025660    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/d55eb70a-f1a8-4346-b8df-ccd2c598dd6b-ovnkube-script-lib\") pod \"ovnkube-node-pghw2\" (UID: \"d55eb70a-f1a8-4346-b8df-ccd2c598dd6b\") " pod="openshift-ovn-kubernetes/ovnkube-node-pghw2"
Oct 31 01:03:11.026228 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:11.025734    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/08490837-b3b6-4c2b-a498-568af0d7086d-etc-modprobe-d\") pod \"tuned-45bvr\" (UID: \"08490837-b3b6-4c2b-a498-568af0d7086d\") " pod="openshift-cluster-node-tuning-operator/tuned-45bvr"
Oct 31 01:03:11.026228 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:11.025765    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/58b0b75a-1568-4703-ac55-fa87f29225ad-multus-cni-dir\") pod \"multus-hxh6j\" (UID: \"58b0b75a-1568-4703-ac55-fa87f29225ad\") " pod="openshift-multus/multus-hxh6j"
Oct 31 01:03:11.026228 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:11.025788    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/58b0b75a-1568-4703-ac55-fa87f29225ad-host-var-lib-cni-multus\") pod \"multus-hxh6j\" (UID: \"58b0b75a-1568-4703-ac55-fa87f29225ad\") " pod="openshift-multus/multus-hxh6j"
Oct 31 01:03:11.026228 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:11.025813    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d55eb70a-f1a8-4346-b8df-ccd2c598dd6b-var-lib-openvswitch\") pod \"ovnkube-node-pghw2\" (UID: \"d55eb70a-f1a8-4346-b8df-ccd2c598dd6b\") " pod="openshift-ovn-kubernetes/ovnkube-node-pghw2"
Oct 31 01:03:11.026228 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:11.025836    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/d55eb70a-f1a8-4346-b8df-ccd2c598dd6b-env-overrides\") pod \"ovnkube-node-pghw2\" (UID: \"d55eb70a-f1a8-4346-b8df-ccd2c598dd6b\") " pod="openshift-ovn-kubernetes/ovnkube-node-pghw2"
Oct 31 01:03:11.026228 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:11.025852    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/58b0b75a-1568-4703-ac55-fa87f29225ad-host-var-lib-cni-multus\") pod \"multus-hxh6j\" (UID: \"58b0b75a-1568-4703-ac55-fa87f29225ad\") " pod="openshift-multus/multus-hxh6j"
Oct 31 01:03:11.026228 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:11.025874    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d55eb70a-f1a8-4346-b8df-ccd2c598dd6b-var-lib-openvswitch\") pod \"ovnkube-node-pghw2\" (UID: \"d55eb70a-f1a8-4346-b8df-ccd2c598dd6b\") " pod="openshift-ovn-kubernetes/ovnkube-node-pghw2"
Oct 31 01:03:11.026228 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:11.025872    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/58b0b75a-1568-4703-ac55-fa87f29225ad-multus-cni-dir\") pod \"multus-hxh6j\" (UID: \"58b0b75a-1568-4703-ac55-fa87f29225ad\") " pod="openshift-multus/multus-hxh6j"
Oct 31 01:03:11.026228 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:11.025893    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/318b3bc6-6e59-4211-af63-b116526a325d-etc-selinux\") pod \"aws-ebs-csi-driver-node-snxz4\" (UID: \"318b3bc6-6e59-4211-af63-b116526a325d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-snxz4"
Oct 31 01:03:11.026228 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:11.025859    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/08490837-b3b6-4c2b-a498-568af0d7086d-etc-modprobe-d\") pod \"tuned-45bvr\" (UID: \"08490837-b3b6-4c2b-a498-568af0d7086d\") " pod="openshift-cluster-node-tuning-operator/tuned-45bvr"
Oct 31 01:03:11.026228 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:11.025838    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/318b3bc6-6e59-4211-af63-b116526a325d-etc-selinux\") pod \"aws-ebs-csi-driver-node-snxz4\" (UID: \"318b3bc6-6e59-4211-af63-b116526a325d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-snxz4"
Oct 31 01:03:11.026228 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:11.025941    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2dx97\" (UniqueName: \"kubernetes.io/projected/57c2f196-0d8a-4bce-a5bc-3b6b5c0d808c-kube-api-access-2dx97\") pod \"node-resolver-j46hv\" (UID: \"57c2f196-0d8a-4bce-a5bc-3b6b5c0d808c\") " pod="openshift-dns/node-resolver-j46hv"
Oct 31 01:03:11.026228 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:11.025981    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/08490837-b3b6-4c2b-a498-568af0d7086d-etc-sysctl-d\") pod \"tuned-45bvr\" (UID: \"08490837-b3b6-4c2b-a498-568af0d7086d\") " pod="openshift-cluster-node-tuning-operator/tuned-45bvr"
Oct 31 01:03:11.026228 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:11.026011    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/58b0b75a-1568-4703-ac55-fa87f29225ad-multus-socket-dir-parent\") pod \"multus-hxh6j\" (UID: \"58b0b75a-1568-4703-ac55-fa87f29225ad\") " pod="openshift-multus/multus-hxh6j"
Oct 31 01:03:11.027135 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:11.026039    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/d55eb70a-f1a8-4346-b8df-ccd2c598dd6b-host-kubelet\") pod \"ovnkube-node-pghw2\" (UID: \"d55eb70a-f1a8-4346-b8df-ccd2c598dd6b\") " pod="openshift-ovn-kubernetes/ovnkube-node-pghw2"
Oct 31 01:03:11.027135 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:11.026064    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d55eb70a-f1a8-4346-b8df-ccd2c598dd6b-host-run-netns\") pod \"ovnkube-node-pghw2\" (UID: \"d55eb70a-f1a8-4346-b8df-ccd2c598dd6b\") " pod="openshift-ovn-kubernetes/ovnkube-node-pghw2"
Oct 31 01:03:11.027135 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:11.026094    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ftdwt\" (UniqueName: \"kubernetes.io/projected/d55eb70a-f1a8-4346-b8df-ccd2c598dd6b-kube-api-access-ftdwt\") pod \"ovnkube-node-pghw2\" (UID: \"d55eb70a-f1a8-4346-b8df-ccd2c598dd6b\") " pod="openshift-ovn-kubernetes/ovnkube-node-pghw2"
Oct 31 01:03:11.027135 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:11.026112    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/58b0b75a-1568-4703-ac55-fa87f29225ad-multus-socket-dir-parent\") pod \"multus-hxh6j\" (UID: \"58b0b75a-1568-4703-ac55-fa87f29225ad\") " pod="openshift-multus/multus-hxh6j"
Oct 31 01:03:11.027135 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:11.026132    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rms4s\" (UniqueName: \"kubernetes.io/projected/318b3bc6-6e59-4211-af63-b116526a325d-kube-api-access-rms4s\") pod \"aws-ebs-csi-driver-node-snxz4\" (UID: \"318b3bc6-6e59-4211-af63-b116526a325d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-snxz4"
Oct 31 01:03:11.027135 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:11.026139    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/d55eb70a-f1a8-4346-b8df-ccd2c598dd6b-host-kubelet\") pod \"ovnkube-node-pghw2\" (UID: \"d55eb70a-f1a8-4346-b8df-ccd2c598dd6b\") " pod="openshift-ovn-kubernetes/ovnkube-node-pghw2"
Oct 31 01:03:11.027135 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:11.026160    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/58b0b75a-1568-4703-ac55-fa87f29225ad-hostroot\") pod \"multus-hxh6j\" (UID: \"58b0b75a-1568-4703-ac55-fa87f29225ad\") " pod="openshift-multus/multus-hxh6j"
Oct 31 01:03:11.027135 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:11.026171    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/d55eb70a-f1a8-4346-b8df-ccd2c598dd6b-ovnkube-script-lib\") pod \"ovnkube-node-pghw2\" (UID: \"d55eb70a-f1a8-4346-b8df-ccd2c598dd6b\") " pod="openshift-ovn-kubernetes/ovnkube-node-pghw2"
Oct 31 01:03:11.027135 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:11.026192    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/08490837-b3b6-4c2b-a498-568af0d7086d-tmp\") pod \"tuned-45bvr\" (UID: \"08490837-b3b6-4c2b-a498-568af0d7086d\") " pod="openshift-cluster-node-tuning-operator/tuned-45bvr"
Oct 31 01:03:11.027135 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:11.026196    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/08490837-b3b6-4c2b-a498-568af0d7086d-etc-sysctl-d\") pod \"tuned-45bvr\" (UID: \"08490837-b3b6-4c2b-a498-568af0d7086d\") " pod="openshift-cluster-node-tuning-operator/tuned-45bvr"
Oct 31 01:03:11.027135 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:11.026246    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/58b0b75a-1568-4703-ac55-fa87f29225ad-hostroot\") pod \"multus-hxh6j\" (UID: \"58b0b75a-1568-4703-ac55-fa87f29225ad\") " pod="openshift-multus/multus-hxh6j"
Oct 31 01:03:11.027135 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:11.026300    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-srkbh\" (UniqueName: \"kubernetes.io/projected/8952fc79-c5a6-42ba-a296-8cb7a776f718-kube-api-access-srkbh\") pod \"network-metrics-daemon-wdsmn\" (UID: \"8952fc79-c5a6-42ba-a296-8cb7a776f718\") " pod="openshift-multus/network-metrics-daemon-wdsmn"
Oct 31 01:03:11.027135 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:11.026331    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/57c2f196-0d8a-4bce-a5bc-3b6b5c0d808c-tmp-dir\") pod \"node-resolver-j46hv\" (UID: \"57c2f196-0d8a-4bce-a5bc-3b6b5c0d808c\") " pod="openshift-dns/node-resolver-j46hv"
Oct 31 01:03:11.027135 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:11.026356    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d55eb70a-f1a8-4346-b8df-ccd2c598dd6b-host-cni-bin\") pod \"ovnkube-node-pghw2\" (UID: \"d55eb70a-f1a8-4346-b8df-ccd2c598dd6b\") " pod="openshift-ovn-kubernetes/ovnkube-node-pghw2"
Oct 31 01:03:11.027135 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:11.026387    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d55eb70a-f1a8-4346-b8df-ccd2c598dd6b-host-run-netns\") pod \"ovnkube-node-pghw2\" (UID: \"d55eb70a-f1a8-4346-b8df-ccd2c598dd6b\") " pod="openshift-ovn-kubernetes/ovnkube-node-pghw2"
Oct 31 01:03:11.027135 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:11.026413    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/f8855c2c-87b4-4b9e-b28a-227981205b11-konnectivity-ca\") pod \"konnectivity-agent-78szg\" (UID: \"f8855c2c-87b4-4b9e-b28a-227981205b11\") " pod="kube-system/konnectivity-agent-78szg"
Oct 31 01:03:11.027135 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:11.026424    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d55eb70a-f1a8-4346-b8df-ccd2c598dd6b-host-cni-bin\") pod \"ovnkube-node-pghw2\" (UID: \"d55eb70a-f1a8-4346-b8df-ccd2c598dd6b\") " pod="openshift-ovn-kubernetes/ovnkube-node-pghw2"
Oct 31 01:03:11.027135 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:11.026439    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/58b0b75a-1568-4703-ac55-fa87f29225ad-host-run-netns\") pod \"multus-hxh6j\" (UID: \"58b0b75a-1568-4703-ac55-fa87f29225ad\") " pod="openshift-multus/multus-hxh6j"
Oct 31 01:03:11.028277 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:11.026464    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/08490837-b3b6-4c2b-a498-568af0d7086d-lib-modules\") pod \"tuned-45bvr\" (UID: \"08490837-b3b6-4c2b-a498-568af0d7086d\") " pod="openshift-cluster-node-tuning-operator/tuned-45bvr"
Oct 31 01:03:11.028277 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:11.026490    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/08490837-b3b6-4c2b-a498-568af0d7086d-var-lib-kubelet\") pod \"tuned-45bvr\" (UID: \"08490837-b3b6-4c2b-a498-568af0d7086d\") " pod="openshift-cluster-node-tuning-operator/tuned-45bvr"
Oct 31 01:03:11.028277 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:11.026516    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/58b0b75a-1568-4703-ac55-fa87f29225ad-etc-kubernetes\") pod \"multus-hxh6j\" (UID: \"58b0b75a-1568-4703-ac55-fa87f29225ad\") " pod="openshift-multus/multus-hxh6j"
Oct 31 01:03:11.028277 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:11.026540    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/d55eb70a-f1a8-4346-b8df-ccd2c598dd6b-host-cni-netd\") pod \"ovnkube-node-pghw2\" (UID: \"d55eb70a-f1a8-4346-b8df-ccd2c598dd6b\") " pod="openshift-ovn-kubernetes/ovnkube-node-pghw2"
Oct 31 01:03:11.028277 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:11.026674    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/57c2f196-0d8a-4bce-a5bc-3b6b5c0d808c-tmp-dir\") pod \"node-resolver-j46hv\" (UID: \"57c2f196-0d8a-4bce-a5bc-3b6b5c0d808c\") " pod="openshift-dns/node-resolver-j46hv"
Oct 31 01:03:11.028277 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:11.026691    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/08490837-b3b6-4c2b-a498-568af0d7086d-lib-modules\") pod \"tuned-45bvr\" (UID: \"08490837-b3b6-4c2b-a498-568af0d7086d\") " pod="openshift-cluster-node-tuning-operator/tuned-45bvr"
Oct 31 01:03:11.028277 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:11.026738    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/58b0b75a-1568-4703-ac55-fa87f29225ad-host-run-netns\") pod \"multus-hxh6j\" (UID: \"58b0b75a-1568-4703-ac55-fa87f29225ad\") " pod="openshift-multus/multus-hxh6j"
Oct 31 01:03:11.028277 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:11.026745    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d55eb70a-f1a8-4346-b8df-ccd2c598dd6b-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-pghw2\" (UID: \"d55eb70a-f1a8-4346-b8df-ccd2c598dd6b\") " pod="openshift-ovn-kubernetes/ovnkube-node-pghw2"
Oct 31 01:03:11.028277 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:11.026768    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/d55eb70a-f1a8-4346-b8df-ccd2c598dd6b-host-cni-netd\") pod \"ovnkube-node-pghw2\" (UID: \"d55eb70a-f1a8-4346-b8df-ccd2c598dd6b\") " pod="openshift-ovn-kubernetes/ovnkube-node-pghw2"
Oct 31 01:03:11.028277 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:11.026776    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/08490837-b3b6-4c2b-a498-568af0d7086d-etc-systemd\") pod \"tuned-45bvr\" (UID: \"08490837-b3b6-4c2b-a498-568af0d7086d\") " pod="openshift-cluster-node-tuning-operator/tuned-45bvr"
Oct 31 01:03:11.028277 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:11.026797    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/08490837-b3b6-4c2b-a498-568af0d7086d-var-lib-kubelet\") pod \"tuned-45bvr\" (UID: \"08490837-b3b6-4c2b-a498-568af0d7086d\") " pod="openshift-cluster-node-tuning-operator/tuned-45bvr"
Oct 31 01:03:11.028277 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:11.026806    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w9tzc\" (UniqueName: \"kubernetes.io/projected/08490837-b3b6-4c2b-a498-568af0d7086d-kube-api-access-w9tzc\") pod \"tuned-45bvr\" (UID: \"08490837-b3b6-4c2b-a498-568af0d7086d\") " pod="openshift-cluster-node-tuning-operator/tuned-45bvr"
Oct 31 01:03:11.028277 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:11.026809    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d55eb70a-f1a8-4346-b8df-ccd2c598dd6b-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-pghw2\" (UID: \"d55eb70a-f1a8-4346-b8df-ccd2c598dd6b\") " pod="openshift-ovn-kubernetes/ovnkube-node-pghw2"
Oct 31 01:03:11.028277 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:11.026833    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/58b0b75a-1568-4703-ac55-fa87f29225ad-etc-kubernetes\") pod \"multus-hxh6j\" (UID: \"58b0b75a-1568-4703-ac55-fa87f29225ad\") " pod="openshift-multus/multus-hxh6j"
Oct 31 01:03:11.028277 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:11.026847    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/318b3bc6-6e59-4211-af63-b116526a325d-kubelet-dir\") pod \"aws-ebs-csi-driver-node-snxz4\" (UID: \"318b3bc6-6e59-4211-af63-b116526a325d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-snxz4"
Oct 31 01:03:11.028277 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:11.026875    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/58b0b75a-1568-4703-ac55-fa87f29225ad-system-cni-dir\") pod \"multus-hxh6j\" (UID: \"58b0b75a-1568-4703-ac55-fa87f29225ad\") " pod="openshift-multus/multus-hxh6j"
Oct 31 01:03:11.028277 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:11.026902    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/d55eb70a-f1a8-4346-b8df-ccd2c598dd6b-ovnkube-config\") pod \"ovnkube-node-pghw2\" (UID: \"d55eb70a-f1a8-4346-b8df-ccd2c598dd6b\") " pod="openshift-ovn-kubernetes/ovnkube-node-pghw2"
Oct 31 01:03:11.029169 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:11.026922    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/f8855c2c-87b4-4b9e-b28a-227981205b11-konnectivity-ca\") pod \"konnectivity-agent-78szg\" (UID: \"f8855c2c-87b4-4b9e-b28a-227981205b11\") " pod="kube-system/konnectivity-agent-78szg"
Oct 31 01:03:11.029169 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:11.026928    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/08490837-b3b6-4c2b-a498-568af0d7086d-run\") pod \"tuned-45bvr\" (UID: \"08490837-b3b6-4c2b-a498-568af0d7086d\") " pod="openshift-cluster-node-tuning-operator/tuned-45bvr"
Oct 31 01:03:11.029169 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:11.026957    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/318b3bc6-6e59-4211-af63-b116526a325d-registration-dir\") pod \"aws-ebs-csi-driver-node-snxz4\" (UID: \"318b3bc6-6e59-4211-af63-b116526a325d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-snxz4"
Oct 31 01:03:11.029169 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:11.026974    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/58b0b75a-1568-4703-ac55-fa87f29225ad-system-cni-dir\") pod \"multus-hxh6j\" (UID: \"58b0b75a-1568-4703-ac55-fa87f29225ad\") " pod="openshift-multus/multus-hxh6j"
Oct 31 01:03:11.029169 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:11.026983    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/57c2f196-0d8a-4bce-a5bc-3b6b5c0d808c-hosts-file\") pod \"node-resolver-j46hv\" (UID: \"57c2f196-0d8a-4bce-a5bc-3b6b5c0d808c\") " pod="openshift-dns/node-resolver-j46hv"
Oct 31 01:03:11.029169 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:11.027008    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/318b3bc6-6e59-4211-af63-b116526a325d-kubelet-dir\") pod \"aws-ebs-csi-driver-node-snxz4\" (UID: \"318b3bc6-6e59-4211-af63-b116526a325d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-snxz4"
Oct 31 01:03:11.029169 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:11.027011    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/d55eb70a-f1a8-4346-b8df-ccd2c598dd6b-run-ovn\") pod \"ovnkube-node-pghw2\" (UID: \"d55eb70a-f1a8-4346-b8df-ccd2c598dd6b\") " pod="openshift-ovn-kubernetes/ovnkube-node-pghw2"
Oct 31 01:03:11.029169 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:11.027045    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/08490837-b3b6-4c2b-a498-568af0d7086d-etc-tuned\") pod \"tuned-45bvr\" (UID: \"08490837-b3b6-4c2b-a498-568af0d7086d\") " pod="openshift-cluster-node-tuning-operator/tuned-45bvr"
Oct 31 01:03:11.029169 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:11.027051    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/d55eb70a-f1a8-4346-b8df-ccd2c598dd6b-run-ovn\") pod \"ovnkube-node-pghw2\" (UID: \"d55eb70a-f1a8-4346-b8df-ccd2c598dd6b\") " pod="openshift-ovn-kubernetes/ovnkube-node-pghw2"
Oct 31 01:03:11.029169 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:11.027097    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8952fc79-c5a6-42ba-a296-8cb7a776f718-metrics-certs\") pod \"network-metrics-daemon-wdsmn\" (UID: \"8952fc79-c5a6-42ba-a296-8cb7a776f718\") " pod="openshift-multus/network-metrics-daemon-wdsmn"
Oct 31 01:03:11.029169 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:11.027106    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/08490837-b3b6-4c2b-a498-568af0d7086d-run\") pod \"tuned-45bvr\" (UID: \"08490837-b3b6-4c2b-a498-568af0d7086d\") " pod="openshift-cluster-node-tuning-operator/tuned-45bvr"
Oct 31 01:03:11.029169 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:11.027126    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/58b0b75a-1568-4703-ac55-fa87f29225ad-multus-daemon-config\") pod \"multus-hxh6j\" (UID: \"58b0b75a-1568-4703-ac55-fa87f29225ad\") " pod="openshift-multus/multus-hxh6j"
Oct 31 01:03:11.029169 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:11.027167    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/57c2f196-0d8a-4bce-a5bc-3b6b5c0d808c-hosts-file\") pod \"node-resolver-j46hv\" (UID: \"57c2f196-0d8a-4bce-a5bc-3b6b5c0d808c\") " pod="openshift-dns/node-resolver-j46hv"
Oct 31 01:03:11.029169 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:11.027155    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/08490837-b3b6-4c2b-a498-568af0d7086d-etc-kubernetes\") pod \"tuned-45bvr\" (UID: \"08490837-b3b6-4c2b-a498-568af0d7086d\") " pod="openshift-cluster-node-tuning-operator/tuned-45bvr"
Oct 31 01:03:11.029169 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:11.027213    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/318b3bc6-6e59-4211-af63-b116526a325d-sys-fs\") pod \"aws-ebs-csi-driver-node-snxz4\" (UID: \"318b3bc6-6e59-4211-af63-b116526a325d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-snxz4"
Oct 31 01:03:11.029169 ip-10-0-133-197 kubenswrapper[2578]: E1031 01:03:11.027235    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 31 01:03:11.029169 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:11.027259    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/58b0b75a-1568-4703-ac55-fa87f29225ad-host-var-lib-cni-bin\") pod \"multus-hxh6j\" (UID: \"58b0b75a-1568-4703-ac55-fa87f29225ad\") " pod="openshift-multus/multus-hxh6j"
Oct 31 01:03:11.029731 ip-10-0-133-197 kubenswrapper[2578]: E1031 01:03:11.027285    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8952fc79-c5a6-42ba-a296-8cb7a776f718-metrics-certs podName:8952fc79-c5a6-42ba-a296-8cb7a776f718 nodeName:}" failed. No retries permitted until 2026-10-31 01:03:11.527268049 +0000 UTC m=+3.212747457 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8952fc79-c5a6-42ba-a296-8cb7a776f718-metrics-certs") pod "network-metrics-daemon-wdsmn" (UID: "8952fc79-c5a6-42ba-a296-8cb7a776f718") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 31 01:03:11.029731 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:11.027296    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/318b3bc6-6e59-4211-af63-b116526a325d-sys-fs\") pod \"aws-ebs-csi-driver-node-snxz4\" (UID: \"318b3bc6-6e59-4211-af63-b116526a325d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-snxz4"
Oct 31 01:03:11.029731 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:11.027238    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/58b0b75a-1568-4703-ac55-fa87f29225ad-host-var-lib-cni-bin\") pod \"multus-hxh6j\" (UID: \"58b0b75a-1568-4703-ac55-fa87f29225ad\") " pod="openshift-multus/multus-hxh6j"
Oct 31 01:03:11.029731 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:11.027333    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d55eb70a-f1a8-4346-b8df-ccd2c598dd6b-host-slash\") pod \"ovnkube-node-pghw2\" (UID: \"d55eb70a-f1a8-4346-b8df-ccd2c598dd6b\") " pod="openshift-ovn-kubernetes/ovnkube-node-pghw2"
Oct 31 01:03:11.029731 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:11.027378    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d55eb70a-f1a8-4346-b8df-ccd2c598dd6b-etc-openvswitch\") pod \"ovnkube-node-pghw2\" (UID: \"d55eb70a-f1a8-4346-b8df-ccd2c598dd6b\") " pod="openshift-ovn-kubernetes/ovnkube-node-pghw2"
Oct 31 01:03:11.029731 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:11.027459    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/08490837-b3b6-4c2b-a498-568af0d7086d-sys\") pod \"tuned-45bvr\" (UID: \"08490837-b3b6-4c2b-a498-568af0d7086d\") " pod="openshift-cluster-node-tuning-operator/tuned-45bvr"
Oct 31 01:03:11.029731 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:11.027485    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/58b0b75a-1568-4703-ac55-fa87f29225ad-cni-binary-copy\") pod \"multus-hxh6j\" (UID: \"58b0b75a-1568-4703-ac55-fa87f29225ad\") " pod="openshift-multus/multus-hxh6j"
Oct 31 01:03:11.029731 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:11.027510    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/58b0b75a-1568-4703-ac55-fa87f29225ad-host-run-k8s-cni-cncf-io\") pod \"multus-hxh6j\" (UID: \"58b0b75a-1568-4703-ac55-fa87f29225ad\") " pod="openshift-multus/multus-hxh6j"
Oct 31 01:03:11.029731 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:11.027539    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/58b0b75a-1568-4703-ac55-fa87f29225ad-multus-conf-dir\") pod \"multus-hxh6j\" (UID: \"58b0b75a-1568-4703-ac55-fa87f29225ad\") " pod="openshift-multus/multus-hxh6j"
Oct 31 01:03:11.029731 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:11.027567    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d55eb70a-f1a8-4346-b8df-ccd2c598dd6b-run-openvswitch\") pod \"ovnkube-node-pghw2\" (UID: \"d55eb70a-f1a8-4346-b8df-ccd2c598dd6b\") " pod="openshift-ovn-kubernetes/ovnkube-node-pghw2"
Oct 31 01:03:11.029731 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:11.027591    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d55eb70a-f1a8-4346-b8df-ccd2c598dd6b-host-run-ovn-kubernetes\") pod \"ovnkube-node-pghw2\" (UID: \"d55eb70a-f1a8-4346-b8df-ccd2c598dd6b\") " pod="openshift-ovn-kubernetes/ovnkube-node-pghw2"
Oct 31 01:03:11.029731 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:11.027616    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/08490837-b3b6-4c2b-a498-568af0d7086d-etc-sysctl-conf\") pod \"tuned-45bvr\" (UID: \"08490837-b3b6-4c2b-a498-568af0d7086d\") " pod="openshift-cluster-node-tuning-operator/tuned-45bvr"
Oct 31 01:03:11.029731 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:11.027639    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/08490837-b3b6-4c2b-a498-568af0d7086d-host\") pod \"tuned-45bvr\" (UID: \"08490837-b3b6-4c2b-a498-568af0d7086d\") " pod="openshift-cluster-node-tuning-operator/tuned-45bvr"
Oct 31 01:03:11.029731 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:11.027670    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/d55eb70a-f1a8-4346-b8df-ccd2c598dd6b-systemd-units\") pod \"ovnkube-node-pghw2\" (UID: \"d55eb70a-f1a8-4346-b8df-ccd2c598dd6b\") " pod="openshift-ovn-kubernetes/ovnkube-node-pghw2"
Oct 31 01:03:11.029731 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:11.027679    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/d55eb70a-f1a8-4346-b8df-ccd2c598dd6b-ovnkube-config\") pod \"ovnkube-node-pghw2\" (UID: \"d55eb70a-f1a8-4346-b8df-ccd2c598dd6b\") " pod="openshift-ovn-kubernetes/ovnkube-node-pghw2"
Oct 31 01:03:11.029731 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:11.027744    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/d55eb70a-f1a8-4346-b8df-ccd2c598dd6b-systemd-units\") pod \"ovnkube-node-pghw2\" (UID: \"d55eb70a-f1a8-4346-b8df-ccd2c598dd6b\") " pod="openshift-ovn-kubernetes/ovnkube-node-pghw2"
Oct 31 01:03:11.029731 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:11.027777    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/58b0b75a-1568-4703-ac55-fa87f29225ad-host-run-k8s-cni-cncf-io\") pod \"multus-hxh6j\" (UID: \"58b0b75a-1568-4703-ac55-fa87f29225ad\") " pod="openshift-multus/multus-hxh6j"
Oct 31 01:03:11.030208 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:11.027799    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/58b0b75a-1568-4703-ac55-fa87f29225ad-multus-conf-dir\") pod \"multus-hxh6j\" (UID: \"58b0b75a-1568-4703-ac55-fa87f29225ad\") " pod="openshift-multus/multus-hxh6j"
Oct 31 01:03:11.030208 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:11.027819    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/08490837-b3b6-4c2b-a498-568af0d7086d-etc-kubernetes\") pod \"tuned-45bvr\" (UID: \"08490837-b3b6-4c2b-a498-568af0d7086d\") " pod="openshift-cluster-node-tuning-operator/tuned-45bvr"
Oct 31 01:03:11.030208 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:11.027834    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/58b0b75a-1568-4703-ac55-fa87f29225ad-multus-daemon-config\") pod \"multus-hxh6j\" (UID: \"58b0b75a-1568-4703-ac55-fa87f29225ad\") " pod="openshift-multus/multus-hxh6j"
Oct 31 01:03:11.030208 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:11.027845    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d55eb70a-f1a8-4346-b8df-ccd2c598dd6b-run-openvswitch\") pod \"ovnkube-node-pghw2\" (UID: \"d55eb70a-f1a8-4346-b8df-ccd2c598dd6b\") " pod="openshift-ovn-kubernetes/ovnkube-node-pghw2"
Oct 31 01:03:11.030208 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:11.027851    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d55eb70a-f1a8-4346-b8df-ccd2c598dd6b-host-slash\") pod \"ovnkube-node-pghw2\" (UID: \"d55eb70a-f1a8-4346-b8df-ccd2c598dd6b\") " pod="openshift-ovn-kubernetes/ovnkube-node-pghw2"
Oct 31 01:03:11.030208 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:11.027882    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d55eb70a-f1a8-4346-b8df-ccd2c598dd6b-etc-openvswitch\") pod \"ovnkube-node-pghw2\" (UID: \"d55eb70a-f1a8-4346-b8df-ccd2c598dd6b\") " pod="openshift-ovn-kubernetes/ovnkube-node-pghw2"
Oct 31 01:03:11.030208 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:11.027914    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/08490837-b3b6-4c2b-a498-568af0d7086d-host\") pod \"tuned-45bvr\" (UID: \"08490837-b3b6-4c2b-a498-568af0d7086d\") " pod="openshift-cluster-node-tuning-operator/tuned-45bvr"
Oct 31 01:03:11.030208 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:11.027934    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/08490837-b3b6-4c2b-a498-568af0d7086d-sys\") pod \"tuned-45bvr\" (UID: \"08490837-b3b6-4c2b-a498-568af0d7086d\") " pod="openshift-cluster-node-tuning-operator/tuned-45bvr"
Oct 31 01:03:11.030208 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:11.027954    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/08490837-b3b6-4c2b-a498-568af0d7086d-etc-sysctl-conf\") pod \"tuned-45bvr\" (UID: \"08490837-b3b6-4c2b-a498-568af0d7086d\") " pod="openshift-cluster-node-tuning-operator/tuned-45bvr"
Oct 31 01:03:11.030208 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:11.027983    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d55eb70a-f1a8-4346-b8df-ccd2c598dd6b-host-run-ovn-kubernetes\") pod \"ovnkube-node-pghw2\" (UID: \"d55eb70a-f1a8-4346-b8df-ccd2c598dd6b\") " pod="openshift-ovn-kubernetes/ovnkube-node-pghw2"
Oct 31 01:03:11.030208 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:11.028037    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/08490837-b3b6-4c2b-a498-568af0d7086d-etc-systemd\") pod \"tuned-45bvr\" (UID: \"08490837-b3b6-4c2b-a498-568af0d7086d\") " pod="openshift-cluster-node-tuning-operator/tuned-45bvr"
Oct 31 01:03:11.030208 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:11.028084    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/318b3bc6-6e59-4211-af63-b116526a325d-registration-dir\") pod \"aws-ebs-csi-driver-node-snxz4\" (UID: \"318b3bc6-6e59-4211-af63-b116526a325d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-snxz4"
Oct 31 01:03:11.030208 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:11.028244    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/d55eb70a-f1a8-4346-b8df-ccd2c598dd6b-ovn-node-metrics-cert\") pod \"ovnkube-node-pghw2\" (UID: \"d55eb70a-f1a8-4346-b8df-ccd2c598dd6b\") " pod="openshift-ovn-kubernetes/ovnkube-node-pghw2"
Oct 31 01:03:11.030208 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:11.028300    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/58b0b75a-1568-4703-ac55-fa87f29225ad-cni-binary-copy\") pod \"multus-hxh6j\" (UID: \"58b0b75a-1568-4703-ac55-fa87f29225ad\") " pod="openshift-multus/multus-hxh6j"
Oct 31 01:03:11.030208 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:11.028734    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/f8855c2c-87b4-4b9e-b28a-227981205b11-agent-certs\") pod \"konnectivity-agent-78szg\" (UID: \"f8855c2c-87b4-4b9e-b28a-227981205b11\") " pod="kube-system/konnectivity-agent-78szg"
Oct 31 01:03:11.030208 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:11.029341    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/08490837-b3b6-4c2b-a498-568af0d7086d-etc-tuned\") pod \"tuned-45bvr\" (UID: \"08490837-b3b6-4c2b-a498-568af0d7086d\") " pod="openshift-cluster-node-tuning-operator/tuned-45bvr"
Oct 31 01:03:11.030208 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:11.029384    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/08490837-b3b6-4c2b-a498-568af0d7086d-tmp\") pod \"tuned-45bvr\" (UID: \"08490837-b3b6-4c2b-a498-568af0d7086d\") " pod="openshift-cluster-node-tuning-operator/tuned-45bvr"
Oct 31 01:03:11.088988 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:11.088926    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rms4s\" (UniqueName: \"kubernetes.io/projected/318b3bc6-6e59-4211-af63-b116526a325d-kube-api-access-rms4s\") pod \"aws-ebs-csi-driver-node-snxz4\" (UID: \"318b3bc6-6e59-4211-af63-b116526a325d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-snxz4"
Oct 31 01:03:11.089579 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:11.089557    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w9tzc\" (UniqueName: \"kubernetes.io/projected/08490837-b3b6-4c2b-a498-568af0d7086d-kube-api-access-w9tzc\") pod \"tuned-45bvr\" (UID: \"08490837-b3b6-4c2b-a498-568af0d7086d\") " pod="openshift-cluster-node-tuning-operator/tuned-45bvr"
Oct 31 01:03:11.090000 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:11.089983    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t228s\" (UniqueName: \"kubernetes.io/projected/58b0b75a-1568-4703-ac55-fa87f29225ad-kube-api-access-t228s\") pod \"multus-hxh6j\" (UID: \"58b0b75a-1568-4703-ac55-fa87f29225ad\") " pod="openshift-multus/multus-hxh6j"
Oct 31 01:03:11.090255 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:11.090235    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-srkbh\" (UniqueName: \"kubernetes.io/projected/8952fc79-c5a6-42ba-a296-8cb7a776f718-kube-api-access-srkbh\") pod \"network-metrics-daemon-wdsmn\" (UID: \"8952fc79-c5a6-42ba-a296-8cb7a776f718\") " pod="openshift-multus/network-metrics-daemon-wdsmn"
Oct 31 01:03:11.090381 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:11.090363    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ftdwt\" (UniqueName: \"kubernetes.io/projected/d55eb70a-f1a8-4346-b8df-ccd2c598dd6b-kube-api-access-ftdwt\") pod \"ovnkube-node-pghw2\" (UID: \"d55eb70a-f1a8-4346-b8df-ccd2c598dd6b\") " pod="openshift-ovn-kubernetes/ovnkube-node-pghw2"
Oct 31 01:03:11.090483 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:11.090466    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2dx97\" (UniqueName: \"kubernetes.io/projected/57c2f196-0d8a-4bce-a5bc-3b6b5c0d808c-kube-api-access-2dx97\") pod \"node-resolver-j46hv\" (UID: \"57c2f196-0d8a-4bce-a5bc-3b6b5c0d808c\") " pod="openshift-dns/node-resolver-j46hv"
Oct 31 01:03:11.114684 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:11.114668    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-ssw8w"
Oct 31 01:03:11.122102 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:11.122085    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-rlxt2"
Oct 31 01:03:11.124898 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:11.124877    2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9cd1266e_c630_4259_b1f7_1b88dbde7385.slice/crio-74e94b63179e44eb28db4bd8dd65d786876ea1da8d701522187f36b22647470a WatchSource:0}: Error finding container 74e94b63179e44eb28db4bd8dd65d786876ea1da8d701522187f36b22647470a: Status 404 returned error can't find the container with id 74e94b63179e44eb28db4bd8dd65d786876ea1da8d701522187f36b22647470a
Oct 31 01:03:11.131076 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:11.131054    2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd8134f9_b2a6_4542_82ec_c10edc3fcd72.slice/crio-1bfb2f27afea8897b43d2d990b1e208f70c168e0be7c46e2c3a03b21c99b28b0 WatchSource:0}: Error finding container 1bfb2f27afea8897b43d2d990b1e208f70c168e0be7c46e2c3a03b21c99b28b0: Status 404 returned error can't find the container with id 1bfb2f27afea8897b43d2d990b1e208f70c168e0be7c46e2c3a03b21c99b28b0
Oct 31 01:03:11.139418 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:11.139381    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-78szg"
Oct 31 01:03:11.139678 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:11.139665    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-j46hv"
Oct 31 01:03:11.145386 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:11.145360    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-jwthj"
Oct 31 01:03:11.147648 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:11.147624    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-hxh6j"
Oct 31 01:03:11.150281 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:11.150244    2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8855c2c_87b4_4b9e_b28a_227981205b11.slice/crio-5ad1aa46bf6437794ab135946309ee2068f696296572304aee0cbf7dfc6ea469 WatchSource:0}: Error finding container 5ad1aa46bf6437794ab135946309ee2068f696296572304aee0cbf7dfc6ea469: Status 404 returned error can't find the container with id 5ad1aa46bf6437794ab135946309ee2068f696296572304aee0cbf7dfc6ea469
Oct 31 01:03:11.153135 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:11.153104    2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod57c2f196_0d8a_4bce_a5bc_3b6b5c0d808c.slice/crio-66674f7724debd180e9b8be4d79be7b77877df53417946aa834a4897afec5876 WatchSource:0}: Error finding container 66674f7724debd180e9b8be4d79be7b77877df53417946aa834a4897afec5876: Status 404 returned error can't find the container with id 66674f7724debd180e9b8be4d79be7b77877df53417946aa834a4897afec5876
Oct 31 01:03:11.156385 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:11.156366    2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa085dd8_ca1e_4ec5_acb2_9b9dc878bf8a.slice/crio-0b3559dbcb56809941ca496ac90b08c271ef0b225c141267043d900f6e7d7564 WatchSource:0}: Error finding container 0b3559dbcb56809941ca496ac90b08c271ef0b225c141267043d900f6e7d7564: Status 404 returned error can't find the container with id 0b3559dbcb56809941ca496ac90b08c271ef0b225c141267043d900f6e7d7564
Oct 31 01:03:11.159447 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:11.159422    2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58b0b75a_1568_4703_ac55_fa87f29225ad.slice/crio-105e79f277b5491b41149e986f955c0083288f78b77703e37bbf1cd561db332c WatchSource:0}: Error finding container 105e79f277b5491b41149e986f955c0083288f78b77703e37bbf1cd561db332c: Status 404 returned error can't find the container with id 105e79f277b5491b41149e986f955c0083288f78b77703e37bbf1cd561db332c
Oct 31 01:03:11.164682 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:11.164666    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-45bvr"
Oct 31 01:03:11.173381 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:11.173357    2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod08490837_b3b6_4c2b_a498_568af0d7086d.slice/crio-a2d6c402bc09e03d1054dc0c70d7fbb00ef937fe8be849c8373a001969de0230 WatchSource:0}: Error finding container a2d6c402bc09e03d1054dc0c70d7fbb00ef937fe8be849c8373a001969de0230: Status 404 returned error can't find the container with id a2d6c402bc09e03d1054dc0c70d7fbb00ef937fe8be849c8373a001969de0230
Oct 31 01:03:11.176884 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:11.175725    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-snxz4"
Oct 31 01:03:11.177635 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:11.177616    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-pghw2"
Oct 31 01:03:11.185631 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:11.185605    2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod318b3bc6_6e59_4211_af63_b116526a325d.slice/crio-ddade6cb4a39d5c69647385ce4c28b127550d12a98dfe2caade4862845ce5388 WatchSource:0}: Error finding container ddade6cb4a39d5c69647385ce4c28b127550d12a98dfe2caade4862845ce5388: Status 404 returned error can't find the container with id ddade6cb4a39d5c69647385ce4c28b127550d12a98dfe2caade4862845ce5388
Oct 31 01:03:11.188686 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:11.188667    2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd55eb70a_f1a8_4346_b8df_ccd2c598dd6b.slice/crio-a2d98ed6ecf479a4141b67729bc58f0489ca18f41040a2da19154c4317bcaac5 WatchSource:0}: Error finding container a2d98ed6ecf479a4141b67729bc58f0489ca18f41040a2da19154c4317bcaac5: Status 404 returned error can't find the container with id a2d98ed6ecf479a4141b67729bc58f0489ca18f41040a2da19154c4317bcaac5
Oct 31 01:03:11.430723 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:11.430650    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/aa4836a5-c71a-485d-82fc-f8b53dae7717-original-pull-secret\") pod \"global-pull-secret-syncer-zs5g7\" (UID: \"aa4836a5-c71a-485d-82fc-f8b53dae7717\") " pod="kube-system/global-pull-secret-syncer-zs5g7"
Oct 31 01:03:11.430849 ip-10-0-133-197 kubenswrapper[2578]: E1031 01:03:11.430788    2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 31 01:03:11.430901 ip-10-0-133-197 kubenswrapper[2578]: E1031 01:03:11.430851    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/aa4836a5-c71a-485d-82fc-f8b53dae7717-original-pull-secret podName:aa4836a5-c71a-485d-82fc-f8b53dae7717 nodeName:}" failed. No retries permitted until 2026-10-31 01:03:12.430832237 +0000 UTC m=+4.116311650 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/aa4836a5-c71a-485d-82fc-f8b53dae7717-original-pull-secret") pod "global-pull-secret-syncer-zs5g7" (UID: "aa4836a5-c71a-485d-82fc-f8b53dae7717") : object "kube-system"/"original-pull-secret" not registered
Oct 31 01:03:11.531836 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:11.531798    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8952fc79-c5a6-42ba-a296-8cb7a776f718-metrics-certs\") pod \"network-metrics-daemon-wdsmn\" (UID: \"8952fc79-c5a6-42ba-a296-8cb7a776f718\") " pod="openshift-multus/network-metrics-daemon-wdsmn"
Oct 31 01:03:11.531983 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:11.531896    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r6zx9\" (UniqueName: \"kubernetes.io/projected/565fefaf-82aa-42a8-9cc7-6517c4bddf1e-kube-api-access-r6zx9\") pod \"network-check-target-8mqbd\" (UID: \"565fefaf-82aa-42a8-9cc7-6517c4bddf1e\") " pod="openshift-network-diagnostics/network-check-target-8mqbd"
Oct 31 01:03:11.532059 ip-10-0-133-197 kubenswrapper[2578]: E1031 01:03:11.532034    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 31 01:03:11.532059 ip-10-0-133-197 kubenswrapper[2578]: E1031 01:03:11.532058    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 31 01:03:11.532237 ip-10-0-133-197 kubenswrapper[2578]: E1031 01:03:11.532071    2578 projected.go:196] Error preparing data for projected volume kube-api-access-r6zx9 for pod openshift-network-diagnostics/network-check-target-8mqbd: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 31 01:03:11.532237 ip-10-0-133-197 kubenswrapper[2578]: E1031 01:03:11.532129    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/565fefaf-82aa-42a8-9cc7-6517c4bddf1e-kube-api-access-r6zx9 podName:565fefaf-82aa-42a8-9cc7-6517c4bddf1e nodeName:}" failed. No retries permitted until 2026-10-31 01:03:12.532108762 +0000 UTC m=+4.217588202 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-r6zx9" (UniqueName: "kubernetes.io/projected/565fefaf-82aa-42a8-9cc7-6517c4bddf1e-kube-api-access-r6zx9") pod "network-check-target-8mqbd" (UID: "565fefaf-82aa-42a8-9cc7-6517c4bddf1e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 31 01:03:11.532237 ip-10-0-133-197 kubenswrapper[2578]: E1031 01:03:11.532204    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 31 01:03:11.532440 ip-10-0-133-197 kubenswrapper[2578]: E1031 01:03:11.532240    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8952fc79-c5a6-42ba-a296-8cb7a776f718-metrics-certs podName:8952fc79-c5a6-42ba-a296-8cb7a776f718 nodeName:}" failed. No retries permitted until 2026-10-31 01:03:12.532228225 +0000 UTC m=+4.217707621 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8952fc79-c5a6-42ba-a296-8cb7a776f718-metrics-certs") pod "network-metrics-daemon-wdsmn" (UID: "8952fc79-c5a6-42ba-a296-8cb7a776f718") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 31 01:03:11.791606 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:11.791517    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-hxh6j" event={"ID":"58b0b75a-1568-4703-ac55-fa87f29225ad","Type":"ContainerStarted","Data":"105e79f277b5491b41149e986f955c0083288f78b77703e37bbf1cd561db332c"}
Oct 31 01:03:11.793698 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:11.793635    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-j46hv" event={"ID":"57c2f196-0d8a-4bce-a5bc-3b6b5c0d808c","Type":"ContainerStarted","Data":"66674f7724debd180e9b8be4d79be7b77877df53417946aa834a4897afec5876"}
Oct 31 01:03:11.795034 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:11.795003    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-rlxt2" event={"ID":"dd8134f9-b2a6-4542-82ec-c10edc3fcd72","Type":"ContainerStarted","Data":"1bfb2f27afea8897b43d2d990b1e208f70c168e0be7c46e2c3a03b21c99b28b0"}
Oct 31 01:03:11.796413 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:11.796387    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-pghw2" event={"ID":"d55eb70a-f1a8-4346-b8df-ccd2c598dd6b","Type":"ContainerStarted","Data":"a2d98ed6ecf479a4141b67729bc58f0489ca18f41040a2da19154c4317bcaac5"}
Oct 31 01:03:11.797763 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:11.797732    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-45bvr" event={"ID":"08490837-b3b6-4c2b-a498-568af0d7086d","Type":"ContainerStarted","Data":"a2d6c402bc09e03d1054dc0c70d7fbb00ef937fe8be849c8373a001969de0230"}
Oct 31 01:03:11.799054 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:11.799015    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jwthj" event={"ID":"fa085dd8-ca1e-4ec5-acb2-9b9dc878bf8a","Type":"ContainerStarted","Data":"0b3559dbcb56809941ca496ac90b08c271ef0b225c141267043d900f6e7d7564"}
Oct 31 01:03:11.800555 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:11.800527    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-78szg" event={"ID":"f8855c2c-87b4-4b9e-b28a-227981205b11","Type":"ContainerStarted","Data":"5ad1aa46bf6437794ab135946309ee2068f696296572304aee0cbf7dfc6ea469"}
Oct 31 01:03:11.801876 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:11.801850    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-ssw8w" event={"ID":"9cd1266e-c630-4259-b1f7-1b88dbde7385","Type":"ContainerStarted","Data":"74e94b63179e44eb28db4bd8dd65d786876ea1da8d701522187f36b22647470a"}
Oct 31 01:03:11.803874 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:11.803847    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-snxz4" event={"ID":"318b3bc6-6e59-4211-af63-b116526a325d","Type":"ContainerStarted","Data":"ddade6cb4a39d5c69647385ce4c28b127550d12a98dfe2caade4862845ce5388"}
Oct 31 01:03:11.965504 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:11.965433    2578 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-30 00:58:11 +0000 UTC" deadline="2028-03-31 03:53:39.394690329 +0000 UTC"
Oct 31 01:03:11.965504 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:11.965503    2578 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12410h50m27.42919763s"
Oct 31 01:03:12.439757 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:12.439715    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/aa4836a5-c71a-485d-82fc-f8b53dae7717-original-pull-secret\") pod \"global-pull-secret-syncer-zs5g7\" (UID: \"aa4836a5-c71a-485d-82fc-f8b53dae7717\") " pod="kube-system/global-pull-secret-syncer-zs5g7"
Oct 31 01:03:12.439944 ip-10-0-133-197 kubenswrapper[2578]: E1031 01:03:12.439925    2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 31 01:03:12.439999 ip-10-0-133-197 kubenswrapper[2578]: E1031 01:03:12.439992    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/aa4836a5-c71a-485d-82fc-f8b53dae7717-original-pull-secret podName:aa4836a5-c71a-485d-82fc-f8b53dae7717 nodeName:}" failed. No retries permitted until 2026-10-31 01:03:14.439973253 +0000 UTC m=+6.125452652 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/aa4836a5-c71a-485d-82fc-f8b53dae7717-original-pull-secret") pod "global-pull-secret-syncer-zs5g7" (UID: "aa4836a5-c71a-485d-82fc-f8b53dae7717") : object "kube-system"/"original-pull-secret" not registered
Oct 31 01:03:12.540095 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:12.540058    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r6zx9\" (UniqueName: \"kubernetes.io/projected/565fefaf-82aa-42a8-9cc7-6517c4bddf1e-kube-api-access-r6zx9\") pod \"network-check-target-8mqbd\" (UID: \"565fefaf-82aa-42a8-9cc7-6517c4bddf1e\") " pod="openshift-network-diagnostics/network-check-target-8mqbd"
Oct 31 01:03:12.540274 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:12.540121    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8952fc79-c5a6-42ba-a296-8cb7a776f718-metrics-certs\") pod \"network-metrics-daemon-wdsmn\" (UID: \"8952fc79-c5a6-42ba-a296-8cb7a776f718\") " pod="openshift-multus/network-metrics-daemon-wdsmn"
Oct 31 01:03:12.540274 ip-10-0-133-197 kubenswrapper[2578]: E1031 01:03:12.540251    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 31 01:03:12.540417 ip-10-0-133-197 kubenswrapper[2578]: E1031 01:03:12.540310    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8952fc79-c5a6-42ba-a296-8cb7a776f718-metrics-certs podName:8952fc79-c5a6-42ba-a296-8cb7a776f718 nodeName:}" failed. No retries permitted until 2026-10-31 01:03:14.540291152 +0000 UTC m=+6.225770566 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8952fc79-c5a6-42ba-a296-8cb7a776f718-metrics-certs") pod "network-metrics-daemon-wdsmn" (UID: "8952fc79-c5a6-42ba-a296-8cb7a776f718") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 31 01:03:12.540493 ip-10-0-133-197 kubenswrapper[2578]: E1031 01:03:12.540417    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 31 01:03:12.540493 ip-10-0-133-197 kubenswrapper[2578]: E1031 01:03:12.540435    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 31 01:03:12.540493 ip-10-0-133-197 kubenswrapper[2578]: E1031 01:03:12.540448    2578 projected.go:196] Error preparing data for projected volume kube-api-access-r6zx9 for pod openshift-network-diagnostics/network-check-target-8mqbd: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 31 01:03:12.540493 ip-10-0-133-197 kubenswrapper[2578]: E1031 01:03:12.540487    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/565fefaf-82aa-42a8-9cc7-6517c4bddf1e-kube-api-access-r6zx9 podName:565fefaf-82aa-42a8-9cc7-6517c4bddf1e nodeName:}" failed. No retries permitted until 2026-10-31 01:03:14.54047553 +0000 UTC m=+6.225954933 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-r6zx9" (UniqueName: "kubernetes.io/projected/565fefaf-82aa-42a8-9cc7-6517c4bddf1e-kube-api-access-r6zx9") pod "network-check-target-8mqbd" (UID: "565fefaf-82aa-42a8-9cc7-6517c4bddf1e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 31 01:03:12.788589 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:12.785623    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8mqbd"
Oct 31 01:03:12.788589 ip-10-0-133-197 kubenswrapper[2578]: E1031 01:03:12.785766    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8mqbd" podUID="565fefaf-82aa-42a8-9cc7-6517c4bddf1e"
Oct 31 01:03:12.788589 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:12.785857    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wdsmn"
Oct 31 01:03:12.788589 ip-10-0-133-197 kubenswrapper[2578]: E1031 01:03:12.785944    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wdsmn" podUID="8952fc79-c5a6-42ba-a296-8cb7a776f718"
Oct 31 01:03:12.788589 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:12.786002    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zs5g7"
Oct 31 01:03:12.788589 ip-10-0-133-197 kubenswrapper[2578]: E1031 01:03:12.786071    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zs5g7" podUID="aa4836a5-c71a-485d-82fc-f8b53dae7717"
Oct 31 01:03:12.966603 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:12.966566    2578 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-30 00:58:11 +0000 UTC" deadline="2028-07-22 07:43:59.09326063 +0000 UTC"
Oct 31 01:03:12.966603 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:12.966594    2578 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15126h40m46.126669882s"
Oct 31 01:03:14.457748 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:14.457171    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/aa4836a5-c71a-485d-82fc-f8b53dae7717-original-pull-secret\") pod \"global-pull-secret-syncer-zs5g7\" (UID: \"aa4836a5-c71a-485d-82fc-f8b53dae7717\") " pod="kube-system/global-pull-secret-syncer-zs5g7"
Oct 31 01:03:14.457748 ip-10-0-133-197 kubenswrapper[2578]: E1031 01:03:14.457593    2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 31 01:03:14.457748 ip-10-0-133-197 kubenswrapper[2578]: E1031 01:03:14.457661    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/aa4836a5-c71a-485d-82fc-f8b53dae7717-original-pull-secret podName:aa4836a5-c71a-485d-82fc-f8b53dae7717 nodeName:}" failed. No retries permitted until 2026-10-31 01:03:18.457639077 +0000 UTC m=+10.143118474 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/aa4836a5-c71a-485d-82fc-f8b53dae7717-original-pull-secret") pod "global-pull-secret-syncer-zs5g7" (UID: "aa4836a5-c71a-485d-82fc-f8b53dae7717") : object "kube-system"/"original-pull-secret" not registered
Oct 31 01:03:14.558374 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:14.558262    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r6zx9\" (UniqueName: \"kubernetes.io/projected/565fefaf-82aa-42a8-9cc7-6517c4bddf1e-kube-api-access-r6zx9\") pod \"network-check-target-8mqbd\" (UID: \"565fefaf-82aa-42a8-9cc7-6517c4bddf1e\") " pod="openshift-network-diagnostics/network-check-target-8mqbd"
Oct 31 01:03:14.558374 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:14.558324    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8952fc79-c5a6-42ba-a296-8cb7a776f718-metrics-certs\") pod \"network-metrics-daemon-wdsmn\" (UID: \"8952fc79-c5a6-42ba-a296-8cb7a776f718\") " pod="openshift-multus/network-metrics-daemon-wdsmn"
Oct 31 01:03:14.558616 ip-10-0-133-197 kubenswrapper[2578]: E1031 01:03:14.558475    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 31 01:03:14.558616 ip-10-0-133-197 kubenswrapper[2578]: E1031 01:03:14.558541    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8952fc79-c5a6-42ba-a296-8cb7a776f718-metrics-certs podName:8952fc79-c5a6-42ba-a296-8cb7a776f718 nodeName:}" failed. No retries permitted until 2026-10-31 01:03:18.558522026 +0000 UTC m=+10.244001421 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8952fc79-c5a6-42ba-a296-8cb7a776f718-metrics-certs") pod "network-metrics-daemon-wdsmn" (UID: "8952fc79-c5a6-42ba-a296-8cb7a776f718") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 31 01:03:14.558735 ip-10-0-133-197 kubenswrapper[2578]: E1031 01:03:14.558635    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 31 01:03:14.558735 ip-10-0-133-197 kubenswrapper[2578]: E1031 01:03:14.558650    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 31 01:03:14.558735 ip-10-0-133-197 kubenswrapper[2578]: E1031 01:03:14.558662    2578 projected.go:196] Error preparing data for projected volume kube-api-access-r6zx9 for pod openshift-network-diagnostics/network-check-target-8mqbd: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 31 01:03:14.558735 ip-10-0-133-197 kubenswrapper[2578]: E1031 01:03:14.558701    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/565fefaf-82aa-42a8-9cc7-6517c4bddf1e-kube-api-access-r6zx9 podName:565fefaf-82aa-42a8-9cc7-6517c4bddf1e nodeName:}" failed. No retries permitted until 2026-10-31 01:03:18.558688242 +0000 UTC m=+10.244167642 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-r6zx9" (UniqueName: "kubernetes.io/projected/565fefaf-82aa-42a8-9cc7-6517c4bddf1e-kube-api-access-r6zx9") pod "network-check-target-8mqbd" (UID: "565fefaf-82aa-42a8-9cc7-6517c4bddf1e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 31 01:03:14.783537 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:14.783508    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wdsmn"
Oct 31 01:03:14.783674 ip-10-0-133-197 kubenswrapper[2578]: E1031 01:03:14.783648    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wdsmn" podUID="8952fc79-c5a6-42ba-a296-8cb7a776f718"
Oct 31 01:03:14.783752 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:14.783728    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zs5g7"
Oct 31 01:03:14.783833 ip-10-0-133-197 kubenswrapper[2578]: E1031 01:03:14.783807    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zs5g7" podUID="aa4836a5-c71a-485d-82fc-f8b53dae7717"
Oct 31 01:03:14.785371 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:14.785349    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8mqbd"
Oct 31 01:03:14.785493 ip-10-0-133-197 kubenswrapper[2578]: E1031 01:03:14.785459    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8mqbd" podUID="565fefaf-82aa-42a8-9cc7-6517c4bddf1e"
Oct 31 01:03:14.848103 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:14.848019    2578 generic.go:356] "Generic (PLEG): container finished" podID="107fc5ddbeb783edeed66dfa339956e5" containerID="14ef005331ab078378b218b4578003c218440dedf09c4fc214b9e46021669c2e" exitCode=0
Oct 31 01:03:14.848263 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:14.848121    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-197.us-west-1.compute.internal" event={"ID":"107fc5ddbeb783edeed66dfa339956e5","Type":"ContainerDied","Data":"14ef005331ab078378b218b4578003c218440dedf09c4fc214b9e46021669c2e"}
Oct 31 01:03:14.889465 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:14.889433    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-197.us-west-1.compute.internal" event={"ID":"8dea7006d41de627c8e06bf18f518df6","Type":"ContainerStarted","Data":"e8f14a64a85206c9692c0ca2e99f02bf0f4093c4a2200a98780d31afc67d05d5"}
Oct 31 01:03:15.104426 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:15.104058    2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-197.us-west-1.compute.internal" podStartSLOduration=5.104042332 podStartE2EDuration="5.104042332s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 01:03:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:03:15.103185851 +0000 UTC m=+6.788665272" watchObservedRunningTime="2026-10-31 01:03:15.104042332 +0000 UTC m=+6.789521749"
Oct 31 01:03:15.895245 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:15.895211    2578 generic.go:356] "Generic (PLEG): container finished" podID="fa085dd8-ca1e-4ec5-acb2-9b9dc878bf8a" containerID="b8a7d02a9cbe67bba3e73c26cc7fa100da57b1359175a1e04d6d4f9c5b2763de" exitCode=0
Oct 31 01:03:15.895730 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:15.895298    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jwthj" event={"ID":"fa085dd8-ca1e-4ec5-acb2-9b9dc878bf8a","Type":"ContainerDied","Data":"b8a7d02a9cbe67bba3e73c26cc7fa100da57b1359175a1e04d6d4f9c5b2763de"}
Oct 31 01:03:15.901365 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:15.901278    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-197.us-west-1.compute.internal" event={"ID":"107fc5ddbeb783edeed66dfa339956e5","Type":"ContainerStarted","Data":"eff8ed39a2c076a333e31fd286c4eab21b65ab2334b7add78f1945a3f83b013b"}
Oct 31 01:03:16.785194 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:16.785136    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8mqbd"
Oct 31 01:03:16.785355 ip-10-0-133-197 kubenswrapper[2578]: E1031 01:03:16.785265    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8mqbd" podUID="565fefaf-82aa-42a8-9cc7-6517c4bddf1e"
Oct 31 01:03:16.785403 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:16.785375    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wdsmn"
Oct 31 01:03:16.785474 ip-10-0-133-197 kubenswrapper[2578]: E1031 01:03:16.785457    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wdsmn" podUID="8952fc79-c5a6-42ba-a296-8cb7a776f718"
Oct 31 01:03:16.785539 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:16.785526    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zs5g7"
Oct 31 01:03:16.785618 ip-10-0-133-197 kubenswrapper[2578]: E1031 01:03:16.785603    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zs5g7" podUID="aa4836a5-c71a-485d-82fc-f8b53dae7717"
Oct 31 01:03:16.908672 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:16.908640    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-78szg" event={"ID":"f8855c2c-87b4-4b9e-b28a-227981205b11","Type":"ContainerStarted","Data":"c9964e2b9606a7d7fa07b83b6bd04e87756cf3a9aca9f98286c3ee856ae69a71"}
Oct 31 01:03:17.027516 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:17.027461    2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-197.us-west-1.compute.internal" podStartSLOduration=7.02744151 podStartE2EDuration="7.02744151s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 01:03:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:03:16.19563892 +0000 UTC m=+7.881118333" watchObservedRunningTime="2026-10-31 01:03:17.02744151 +0000 UTC m=+8.712920925"
Oct 31 01:03:17.912178 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:17.912140    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-snxz4" event={"ID":"318b3bc6-6e59-4211-af63-b116526a325d","Type":"ContainerStarted","Data":"9bff0c4796310c9e927ad3a80bd5a0e3735ac02d07ecfff8823be3c3483905a7"}
Oct 31 01:03:17.915633 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:17.915604    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-rlxt2" event={"ID":"dd8134f9-b2a6-4542-82ec-c10edc3fcd72","Type":"ContainerStarted","Data":"adbe378b8f7ea9844f3711fccac40036ece50901baf8a206e114462f9245950b"}
Oct 31 01:03:18.035979 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:18.034647    2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-78szg" podStartSLOduration=4.120404059 podStartE2EDuration="9.034626265s" totalImagesPullingTime="4.914222206s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:03:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:03:17.028889329 +0000 UTC m=+8.714368746" watchObservedRunningTime="2026-10-31 01:03:18.034626265 +0000 UTC m=+9.720105682"
Oct 31 01:03:18.493578 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:18.493515    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/aa4836a5-c71a-485d-82fc-f8b53dae7717-original-pull-secret\") pod \"global-pull-secret-syncer-zs5g7\" (UID: \"aa4836a5-c71a-485d-82fc-f8b53dae7717\") " pod="kube-system/global-pull-secret-syncer-zs5g7"
Oct 31 01:03:18.493783 ip-10-0-133-197 kubenswrapper[2578]: E1031 01:03:18.493624    2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 31 01:03:18.493783 ip-10-0-133-197 kubenswrapper[2578]: E1031 01:03:18.493672    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/aa4836a5-c71a-485d-82fc-f8b53dae7717-original-pull-secret podName:aa4836a5-c71a-485d-82fc-f8b53dae7717 nodeName:}" failed. No retries permitted until 2026-10-31 01:03:26.493654814 +0000 UTC m=+18.179134206 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/aa4836a5-c71a-485d-82fc-f8b53dae7717-original-pull-secret") pod "global-pull-secret-syncer-zs5g7" (UID: "aa4836a5-c71a-485d-82fc-f8b53dae7717") : object "kube-system"/"original-pull-secret" not registered
Oct 31 01:03:18.594888 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:18.594846    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r6zx9\" (UniqueName: \"kubernetes.io/projected/565fefaf-82aa-42a8-9cc7-6517c4bddf1e-kube-api-access-r6zx9\") pod \"network-check-target-8mqbd\" (UID: \"565fefaf-82aa-42a8-9cc7-6517c4bddf1e\") " pod="openshift-network-diagnostics/network-check-target-8mqbd"
Oct 31 01:03:18.595066 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:18.594911    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8952fc79-c5a6-42ba-a296-8cb7a776f718-metrics-certs\") pod \"network-metrics-daemon-wdsmn\" (UID: \"8952fc79-c5a6-42ba-a296-8cb7a776f718\") " pod="openshift-multus/network-metrics-daemon-wdsmn"
Oct 31 01:03:18.595066 ip-10-0-133-197 kubenswrapper[2578]: E1031 01:03:18.595029    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 31 01:03:18.595194 ip-10-0-133-197 kubenswrapper[2578]: E1031 01:03:18.595088    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8952fc79-c5a6-42ba-a296-8cb7a776f718-metrics-certs podName:8952fc79-c5a6-42ba-a296-8cb7a776f718 nodeName:}" failed. No retries permitted until 2026-10-31 01:03:26.595070426 +0000 UTC m=+18.280549822 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8952fc79-c5a6-42ba-a296-8cb7a776f718-metrics-certs") pod "network-metrics-daemon-wdsmn" (UID: "8952fc79-c5a6-42ba-a296-8cb7a776f718") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 31 01:03:18.595194 ip-10-0-133-197 kubenswrapper[2578]: E1031 01:03:18.595171    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 31 01:03:18.595194 ip-10-0-133-197 kubenswrapper[2578]: E1031 01:03:18.595187    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 31 01:03:18.595387 ip-10-0-133-197 kubenswrapper[2578]: E1031 01:03:18.595201    2578 projected.go:196] Error preparing data for projected volume kube-api-access-r6zx9 for pod openshift-network-diagnostics/network-check-target-8mqbd: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 31 01:03:18.595387 ip-10-0-133-197 kubenswrapper[2578]: E1031 01:03:18.595238    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/565fefaf-82aa-42a8-9cc7-6517c4bddf1e-kube-api-access-r6zx9 podName:565fefaf-82aa-42a8-9cc7-6517c4bddf1e nodeName:}" failed. No retries permitted until 2026-10-31 01:03:26.595226762 +0000 UTC m=+18.280706158 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-r6zx9" (UniqueName: "kubernetes.io/projected/565fefaf-82aa-42a8-9cc7-6517c4bddf1e-kube-api-access-r6zx9") pod "network-check-target-8mqbd" (UID: "565fefaf-82aa-42a8-9cc7-6517c4bddf1e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 31 01:03:18.785460 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:18.785395    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8mqbd"
Oct 31 01:03:18.785613 ip-10-0-133-197 kubenswrapper[2578]: E1031 01:03:18.785495    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8mqbd" podUID="565fefaf-82aa-42a8-9cc7-6517c4bddf1e"
Oct 31 01:03:18.785613 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:18.785537    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wdsmn"
Oct 31 01:03:18.785613 ip-10-0-133-197 kubenswrapper[2578]: E1031 01:03:18.785600    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wdsmn" podUID="8952fc79-c5a6-42ba-a296-8cb7a776f718"
Oct 31 01:03:18.785708 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:18.785629    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zs5g7"
Oct 31 01:03:18.785708 ip-10-0-133-197 kubenswrapper[2578]: E1031 01:03:18.785669    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zs5g7" podUID="aa4836a5-c71a-485d-82fc-f8b53dae7717"
Oct 31 01:03:19.833408 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:19.833377    2578 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Oct 31 01:03:19.921110 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:19.920978    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-snxz4" event={"ID":"318b3bc6-6e59-4211-af63-b116526a325d","Type":"ContainerStarted","Data":"fb3a430c8c8e085264f9e7ac14f4ccb89137bd6827716a11e4b186e89c571dd9"}
Oct 31 01:03:19.922809 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:19.922768    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-hxh6j" event={"ID":"58b0b75a-1568-4703-ac55-fa87f29225ad","Type":"ContainerStarted","Data":"bed21d982581a14005cd1f31be113f5d780237550bc9ae88facc837b43da5560"}
Oct 31 01:03:19.924772 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:19.924743    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-j46hv" event={"ID":"57c2f196-0d8a-4bce-a5bc-3b6b5c0d808c","Type":"ContainerStarted","Data":"83655581b607a4e51e685088465ff91a9394e3b159cd87108865ae877f94672f"}
Oct 31 01:03:20.040770 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:20.040719    2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-rlxt2" podStartSLOduration=5.277375177 podStartE2EDuration="11.040703056s" totalImagesPullingTime="5.763327879s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 01:03:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:03:18.035914585 +0000 UTC m=+9.721394000" watchObservedRunningTime="2026-10-31 01:03:20.040703056 +0000 UTC m=+11.726182474"
Oct 31 01:03:20.041111 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:20.041088    2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-hxh6j" podStartSLOduration=2.500737249 podStartE2EDuration="11.041082505s" totalImagesPullingTime="8.540345256s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 01:03:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:03:20.040493174 +0000 UTC m=+11.725972589" watchObservedRunningTime="2026-10-31 01:03:20.041082505 +0000 UTC m=+11.726561920"
Oct 31 01:03:20.154394 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:20.154276    2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-j46hv" podStartSLOduration=2.683515147 podStartE2EDuration="11.154257903s" totalImagesPullingTime="8.470742756s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 01:03:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:03:20.153898745 +0000 UTC m=+11.839378187" watchObservedRunningTime="2026-10-31 01:03:20.154257903 +0000 UTC m=+11.839737318"
Oct 31 01:03:20.783335 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:20.783075    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8mqbd"
Oct 31 01:03:20.783489 ip-10-0-133-197 kubenswrapper[2578]: E1031 01:03:20.783454    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8mqbd" podUID="565fefaf-82aa-42a8-9cc7-6517c4bddf1e"
Oct 31 01:03:20.783556 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:20.783547    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wdsmn"
Oct 31 01:03:20.783657 ip-10-0-133-197 kubenswrapper[2578]: E1031 01:03:20.783634    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wdsmn" podUID="8952fc79-c5a6-42ba-a296-8cb7a776f718"
Oct 31 01:03:20.783846 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:20.783801    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zs5g7"
Oct 31 01:03:20.783943 ip-10-0-133-197 kubenswrapper[2578]: E1031 01:03:20.783907    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zs5g7" podUID="aa4836a5-c71a-485d-82fc-f8b53dae7717"
Oct 31 01:03:20.790253 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:20.790143    2578 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-31T01:03:19.833399539Z","UUID":"0c8bb379-9bdb-4781-96f3-690cf28dfe09","Handler":null,"Name":"","Endpoint":""}
Oct 31 01:03:20.792042 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:20.792022    2578 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Oct 31 01:03:20.792139 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:20.792051    2578 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Oct 31 01:03:20.930192 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:20.930158    2578 generic.go:356] "Generic (PLEG): container finished" podID="fa085dd8-ca1e-4ec5-acb2-9b9dc878bf8a" containerID="f7dfaca6eb74d3c92ab4a0a332937e5ade4d9a209a977efbc75c7f0312bac8f4" exitCode=0
Oct 31 01:03:20.931153 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:20.931132    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jwthj" event={"ID":"fa085dd8-ca1e-4ec5-acb2-9b9dc878bf8a","Type":"ContainerDied","Data":"f7dfaca6eb74d3c92ab4a0a332937e5ade4d9a209a977efbc75c7f0312bac8f4"}
Oct 31 01:03:21.875751 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:21.875679    2578 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-78szg"
Oct 31 01:03:21.876159 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:21.876139    2578 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-78szg"
Oct 31 01:03:21.936038 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:21.936012    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-snxz4" event={"ID":"318b3bc6-6e59-4211-af63-b116526a325d","Type":"ContainerStarted","Data":"222b087197bb237465ad6f0594d7b29fbedd4c734ba61919c806b7ac71d2f4cd"}
Oct 31 01:03:21.938220 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:21.938170    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-ssw8w" event={"ID":"9cd1266e-c630-4259-b1f7-1b88dbde7385","Type":"ContainerStarted","Data":"d3da8f1524c5301888da9352fe23b390737413a259f00bba7a8c35b7d337ba23"}
Oct 31 01:03:22.115271 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:22.115220    2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-ssw8w" podStartSLOduration=4.394573409 podStartE2EDuration="13.115206909s" totalImagesPullingTime="8.7206335s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 01:03:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:03:22.114924044 +0000 UTC m=+13.800403458" watchObservedRunningTime="2026-10-31 01:03:22.115206909 +0000 UTC m=+13.800686321"
Oct 31 01:03:22.234641 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:22.234603    2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-snxz4" podStartSLOduration=2.185909784 podStartE2EDuration="12.23458981s" totalImagesPullingTime="10.048680026s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 01:03:10 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:03:22.234348704 +0000 UTC m=+13.919828119" watchObservedRunningTime="2026-10-31 01:03:22.23458981 +0000 UTC m=+13.920069222"
Oct 31 01:03:22.784228 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:22.782549    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zs5g7"
Oct 31 01:03:22.784228 ip-10-0-133-197 kubenswrapper[2578]: E1031 01:03:22.782702    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zs5g7" podUID="aa4836a5-c71a-485d-82fc-f8b53dae7717"
Oct 31 01:03:22.784228 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:22.782745    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8mqbd"
Oct 31 01:03:22.784228 ip-10-0-133-197 kubenswrapper[2578]: E1031 01:03:22.782816    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8mqbd" podUID="565fefaf-82aa-42a8-9cc7-6517c4bddf1e"
Oct 31 01:03:22.784228 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:22.782869    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wdsmn"
Oct 31 01:03:22.784228 ip-10-0-133-197 kubenswrapper[2578]: E1031 01:03:22.782931    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wdsmn" podUID="8952fc79-c5a6-42ba-a296-8cb7a776f718"
Oct 31 01:03:22.942880 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:22.942852    2578 generic.go:356] "Generic (PLEG): container finished" podID="fa085dd8-ca1e-4ec5-acb2-9b9dc878bf8a" containerID="fad5072047ce37042e99f2f3cf2719e6c5dcc87e827fd9f0c3c0f1db6688b2ce" exitCode=0
Oct 31 01:03:22.943562 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:22.943117    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jwthj" event={"ID":"fa085dd8-ca1e-4ec5-acb2-9b9dc878bf8a","Type":"ContainerDied","Data":"fad5072047ce37042e99f2f3cf2719e6c5dcc87e827fd9f0c3c0f1db6688b2ce"}
Oct 31 01:03:22.945043 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:22.945019    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-45bvr" event={"ID":"08490837-b3b6-4c2b-a498-568af0d7086d","Type":"ContainerStarted","Data":"106e93cb02c272ff4954582ab93d1252868bbbea05f93439fa61bbee7a731ef6"}
Oct 31 01:03:24.783086 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:24.782766    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wdsmn"
Oct 31 01:03:24.783086 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:24.782837    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zs5g7"
Oct 31 01:03:24.783628 ip-10-0-133-197 kubenswrapper[2578]: E1031 01:03:24.783136    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wdsmn" podUID="8952fc79-c5a6-42ba-a296-8cb7a776f718"
Oct 31 01:03:24.783628 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:24.782847    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8mqbd"
Oct 31 01:03:24.783628 ip-10-0-133-197 kubenswrapper[2578]: E1031 01:03:24.783174    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zs5g7" podUID="aa4836a5-c71a-485d-82fc-f8b53dae7717"
Oct 31 01:03:24.783628 ip-10-0-133-197 kubenswrapper[2578]: E1031 01:03:24.783238    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8mqbd" podUID="565fefaf-82aa-42a8-9cc7-6517c4bddf1e"
Oct 31 01:03:24.952328 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:24.952283    2578 generic.go:356] "Generic (PLEG): container finished" podID="fa085dd8-ca1e-4ec5-acb2-9b9dc878bf8a" containerID="84d45cdffaa60e8b1f6749d33a39b43d58bf6beb65ed560f586126b5ba104f87" exitCode=0
Oct 31 01:03:24.952482 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:24.952342    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jwthj" event={"ID":"fa085dd8-ca1e-4ec5-acb2-9b9dc878bf8a","Type":"ContainerDied","Data":"84d45cdffaa60e8b1f6749d33a39b43d58bf6beb65ed560f586126b5ba104f87"}
Oct 31 01:03:25.171849 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:25.171747    2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-45bvr" podStartSLOduration=5.420114206 podStartE2EDuration="16.171728865s" totalImagesPullingTime="10.751614659s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 01:03:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:03:23.180560624 +0000 UTC m=+14.866040036" watchObservedRunningTime="2026-10-31 01:03:25.171728865 +0000 UTC m=+16.857208281"
Oct 31 01:03:26.558266 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:26.558231    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/aa4836a5-c71a-485d-82fc-f8b53dae7717-original-pull-secret\") pod \"global-pull-secret-syncer-zs5g7\" (UID: \"aa4836a5-c71a-485d-82fc-f8b53dae7717\") " pod="kube-system/global-pull-secret-syncer-zs5g7"
Oct 31 01:03:26.558779 ip-10-0-133-197 kubenswrapper[2578]: E1031 01:03:26.558387    2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 31 01:03:26.558779 ip-10-0-133-197 kubenswrapper[2578]: E1031 01:03:26.558457    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/aa4836a5-c71a-485d-82fc-f8b53dae7717-original-pull-secret podName:aa4836a5-c71a-485d-82fc-f8b53dae7717 nodeName:}" failed. No retries permitted until 2026-10-31 01:03:42.558440042 +0000 UTC m=+34.243919452 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/aa4836a5-c71a-485d-82fc-f8b53dae7717-original-pull-secret") pod "global-pull-secret-syncer-zs5g7" (UID: "aa4836a5-c71a-485d-82fc-f8b53dae7717") : object "kube-system"/"original-pull-secret" not registered
Oct 31 01:03:26.659549 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:26.659511    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r6zx9\" (UniqueName: \"kubernetes.io/projected/565fefaf-82aa-42a8-9cc7-6517c4bddf1e-kube-api-access-r6zx9\") pod \"network-check-target-8mqbd\" (UID: \"565fefaf-82aa-42a8-9cc7-6517c4bddf1e\") " pod="openshift-network-diagnostics/network-check-target-8mqbd"
Oct 31 01:03:26.659717 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:26.659559    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8952fc79-c5a6-42ba-a296-8cb7a776f718-metrics-certs\") pod \"network-metrics-daemon-wdsmn\" (UID: \"8952fc79-c5a6-42ba-a296-8cb7a776f718\") " pod="openshift-multus/network-metrics-daemon-wdsmn"
Oct 31 01:03:26.659717 ip-10-0-133-197 kubenswrapper[2578]: E1031 01:03:26.659650    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 31 01:03:26.659717 ip-10-0-133-197 kubenswrapper[2578]: E1031 01:03:26.659663    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 31 01:03:26.659717 ip-10-0-133-197 kubenswrapper[2578]: E1031 01:03:26.659672    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 31 01:03:26.659717 ip-10-0-133-197 kubenswrapper[2578]: E1031 01:03:26.659684    2578 projected.go:196] Error preparing data for projected volume kube-api-access-r6zx9 for pod openshift-network-diagnostics/network-check-target-8mqbd: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 31 01:03:26.659717 ip-10-0-133-197 kubenswrapper[2578]: E1031 01:03:26.659710    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8952fc79-c5a6-42ba-a296-8cb7a776f718-metrics-certs podName:8952fc79-c5a6-42ba-a296-8cb7a776f718 nodeName:}" failed. No retries permitted until 2026-10-31 01:03:42.659697088 +0000 UTC m=+34.345176480 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8952fc79-c5a6-42ba-a296-8cb7a776f718-metrics-certs") pod "network-metrics-daemon-wdsmn" (UID: "8952fc79-c5a6-42ba-a296-8cb7a776f718") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 31 01:03:26.659957 ip-10-0-133-197 kubenswrapper[2578]: E1031 01:03:26.659730    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/565fefaf-82aa-42a8-9cc7-6517c4bddf1e-kube-api-access-r6zx9 podName:565fefaf-82aa-42a8-9cc7-6517c4bddf1e nodeName:}" failed. No retries permitted until 2026-10-31 01:03:42.659715822 +0000 UTC m=+34.345195220 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-r6zx9" (UniqueName: "kubernetes.io/projected/565fefaf-82aa-42a8-9cc7-6517c4bddf1e-kube-api-access-r6zx9") pod "network-check-target-8mqbd" (UID: "565fefaf-82aa-42a8-9cc7-6517c4bddf1e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 31 01:03:26.782944 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:26.782910    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zs5g7"
Oct 31 01:03:26.782944 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:26.782933    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8mqbd"
Oct 31 01:03:26.783185 ip-10-0-133-197 kubenswrapper[2578]: E1031 01:03:26.783045    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zs5g7" podUID="aa4836a5-c71a-485d-82fc-f8b53dae7717"
Oct 31 01:03:26.783301 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:26.782911    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wdsmn"
Oct 31 01:03:26.783422 ip-10-0-133-197 kubenswrapper[2578]: E1031 01:03:26.783397    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wdsmn" podUID="8952fc79-c5a6-42ba-a296-8cb7a776f718"
Oct 31 01:03:26.783543 ip-10-0-133-197 kubenswrapper[2578]: E1031 01:03:26.783459    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8mqbd" podUID="565fefaf-82aa-42a8-9cc7-6517c4bddf1e"
Oct 31 01:03:26.968048 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:26.967875    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-pghw2" event={"ID":"d55eb70a-f1a8-4346-b8df-ccd2c598dd6b","Type":"ContainerStarted","Data":"e61344fc20a68cee0a708867450f8ad4067da341953a99f85ae405a5ebaf1ee3"}
Oct 31 01:03:27.035959 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:27.035933    2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-78szg"
Oct 31 01:03:27.036095 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:27.036073    2578 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 31 01:03:27.036989 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:27.036794    2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-78szg"
Oct 31 01:03:27.972784 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:27.972756    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-pghw2" event={"ID":"d55eb70a-f1a8-4346-b8df-ccd2c598dd6b","Type":"ContainerStarted","Data":"aa94a1b7bc9cc41bbd74e43f83686add26685776327b72798e4d1ce2e957c38f"}
Oct 31 01:03:27.973122 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:27.972823    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-pghw2" event={"ID":"d55eb70a-f1a8-4346-b8df-ccd2c598dd6b","Type":"ContainerStarted","Data":"3155ec8a0d08da37284ea980a85ef9ba1d3e94910ec926d8f31a4666408080e7"}
Oct 31 01:03:27.973122 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:27.972857    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-pghw2" event={"ID":"d55eb70a-f1a8-4346-b8df-ccd2c598dd6b","Type":"ContainerStarted","Data":"9a97686d627259e1f6a9320df3a6ed23cc0007c397149d4efb4d8ec362bd8857"}
Oct 31 01:03:27.973122 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:27.972894    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-pghw2" event={"ID":"d55eb70a-f1a8-4346-b8df-ccd2c598dd6b","Type":"ContainerStarted","Data":"d6b08615e68807295cca89ca1b3e809531d994aaa39e7207263a21522562e8b3"}
Oct 31 01:03:27.973122 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:27.972919    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-pghw2" event={"ID":"d55eb70a-f1a8-4346-b8df-ccd2c598dd6b","Type":"ContainerStarted","Data":"fb6294cfc2da12614d6d5d7b7d5ab08780f750537378543b42cd10db16c9c3a3"}
Oct 31 01:03:27.975481 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:27.975455    2578 generic.go:356] "Generic (PLEG): container finished" podID="fa085dd8-ca1e-4ec5-acb2-9b9dc878bf8a" containerID="4949c5e8a1cb38f44dc8723d4446f6fcd6389b7b18951b8ef2eddc740e55f22f" exitCode=0
Oct 31 01:03:27.975589 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:27.975510    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jwthj" event={"ID":"fa085dd8-ca1e-4ec5-acb2-9b9dc878bf8a","Type":"ContainerDied","Data":"4949c5e8a1cb38f44dc8723d4446f6fcd6389b7b18951b8ef2eddc740e55f22f"}
Oct 31 01:03:28.783445 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:28.783418    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8mqbd"
Oct 31 01:03:28.783572 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:28.783496    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wdsmn"
Oct 31 01:03:28.783572 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:28.783538    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zs5g7"
Oct 31 01:03:28.783572 ip-10-0-133-197 kubenswrapper[2578]: E1031 01:03:28.783533    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8mqbd" podUID="565fefaf-82aa-42a8-9cc7-6517c4bddf1e"
Oct 31 01:03:28.783675 ip-10-0-133-197 kubenswrapper[2578]: E1031 01:03:28.783612    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zs5g7" podUID="aa4836a5-c71a-485d-82fc-f8b53dae7717"
Oct 31 01:03:28.783717 ip-10-0-133-197 kubenswrapper[2578]: E1031 01:03:28.783699    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wdsmn" podUID="8952fc79-c5a6-42ba-a296-8cb7a776f718"
Oct 31 01:03:28.980388 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:28.980351    2578 generic.go:356] "Generic (PLEG): container finished" podID="fa085dd8-ca1e-4ec5-acb2-9b9dc878bf8a" containerID="3ea05b5bb5f802eca92bcb6694d410cc0e27110fcb501abbe2054d0a7463e944" exitCode=0
Oct 31 01:03:28.980837 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:28.980402    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jwthj" event={"ID":"fa085dd8-ca1e-4ec5-acb2-9b9dc878bf8a","Type":"ContainerDied","Data":"3ea05b5bb5f802eca92bcb6694d410cc0e27110fcb501abbe2054d0a7463e944"}
Oct 31 01:03:29.985080 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:29.985046    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-pghw2" event={"ID":"d55eb70a-f1a8-4346-b8df-ccd2c598dd6b","Type":"ContainerStarted","Data":"b6788d3d0abc42aecccd6aa98bd1b25d2013dac3d141428ebb46b7d24ed8a65d"}
Oct 31 01:03:29.988102 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:29.988079    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jwthj" event={"ID":"fa085dd8-ca1e-4ec5-acb2-9b9dc878bf8a","Type":"ContainerStarted","Data":"f8d7ddb3ee203ead54ad8ac785e04179b7e0c4f296200f68356d1c44a44c9133"}
Oct 31 01:03:30.108112 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:30.108072    2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-jwthj" podStartSLOduration=8.30639451 podStartE2EDuration="21.108059847s" totalImagesPullingTime="12.801665337s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 01:03:09 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:03:30.107597585 +0000 UTC m=+21.793076999" watchObservedRunningTime="2026-10-31 01:03:30.108059847 +0000 UTC m=+21.793539280"
Oct 31 01:03:30.784968 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:30.784944    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zs5g7"
Oct 31 01:03:30.785122 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:30.785043    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wdsmn"
Oct 31 01:03:30.785161 ip-10-0-133-197 kubenswrapper[2578]: E1031 01:03:30.785124    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wdsmn" podUID="8952fc79-c5a6-42ba-a296-8cb7a776f718"
Oct 31 01:03:30.785161 ip-10-0-133-197 kubenswrapper[2578]: E1031 01:03:30.785041    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zs5g7" podUID="aa4836a5-c71a-485d-82fc-f8b53dae7717"
Oct 31 01:03:30.785161 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:30.785043    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8mqbd"
Oct 31 01:03:30.785241 ip-10-0-133-197 kubenswrapper[2578]: E1031 01:03:30.785188    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8mqbd" podUID="565fefaf-82aa-42a8-9cc7-6517c4bddf1e"
Oct 31 01:03:31.995077 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:31.995044    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-pghw2" event={"ID":"d55eb70a-f1a8-4346-b8df-ccd2c598dd6b","Type":"ContainerStarted","Data":"7b859676fa2301309b31ea49501ec998fb1c01ed52c9e35fb35b969550633a53"}
Oct 31 01:03:31.995748 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:31.995367    2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-pghw2"
Oct 31 01:03:31.995748 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:31.995425    2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-pghw2"
Oct 31 01:03:32.011003 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:32.010981    2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-pghw2"
Oct 31 01:03:32.276997 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:32.276921    2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-pghw2" podStartSLOduration=7.590559158 podStartE2EDuration="23.276909743s" totalImagesPullingTime="15.686350585s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 01:03:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:03:32.24789062 +0000 UTC m=+23.933370033" watchObservedRunningTime="2026-10-31 01:03:32.276909743 +0000 UTC m=+23.962389155"
Oct 31 01:03:32.783332 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:32.783141    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wdsmn"
Oct 31 01:03:32.783457 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:32.783161    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8mqbd"
Oct 31 01:03:32.783457 ip-10-0-133-197 kubenswrapper[2578]: E1031 01:03:32.783422    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wdsmn" podUID="8952fc79-c5a6-42ba-a296-8cb7a776f718"
Oct 31 01:03:32.783533 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:32.783199    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zs5g7"
Oct 31 01:03:32.783570 ip-10-0-133-197 kubenswrapper[2578]: E1031 01:03:32.783522    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8mqbd" podUID="565fefaf-82aa-42a8-9cc7-6517c4bddf1e"
Oct 31 01:03:32.783608 ip-10-0-133-197 kubenswrapper[2578]: E1031 01:03:32.783590    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zs5g7" podUID="aa4836a5-c71a-485d-82fc-f8b53dae7717"
Oct 31 01:03:33.004484 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:33.004457    2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-pghw2"
Oct 31 01:03:33.019443 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:33.019423    2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-pghw2"
Oct 31 01:03:34.002768 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:34.002716    2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-8mqbd"]
Oct 31 01:03:34.002937 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:34.002914    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8mqbd"
Oct 31 01:03:34.003237 ip-10-0-133-197 kubenswrapper[2578]: E1031 01:03:34.003182    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8mqbd" podUID="565fefaf-82aa-42a8-9cc7-6517c4bddf1e"
Oct 31 01:03:34.004301 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:34.004271    2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-wdsmn"]
Oct 31 01:03:34.004492 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:34.004440    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wdsmn"
Oct 31 01:03:34.004913 ip-10-0-133-197 kubenswrapper[2578]: E1031 01:03:34.004887    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wdsmn" podUID="8952fc79-c5a6-42ba-a296-8cb7a776f718"
Oct 31 01:03:34.005534 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:34.005514    2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-zs5g7"]
Oct 31 01:03:34.005623 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:34.005609    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zs5g7"
Oct 31 01:03:34.005733 ip-10-0-133-197 kubenswrapper[2578]: E1031 01:03:34.005714    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zs5g7" podUID="aa4836a5-c71a-485d-82fc-f8b53dae7717"
Oct 31 01:03:35.782900 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:35.782827    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wdsmn"
Oct 31 01:03:35.783292 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:35.782943    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zs5g7"
Oct 31 01:03:35.783292 ip-10-0-133-197 kubenswrapper[2578]: E1031 01:03:35.782958    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wdsmn" podUID="8952fc79-c5a6-42ba-a296-8cb7a776f718"
Oct 31 01:03:35.783292 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:35.782969    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8mqbd"
Oct 31 01:03:35.783292 ip-10-0-133-197 kubenswrapper[2578]: E1031 01:03:35.783041    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zs5g7" podUID="aa4836a5-c71a-485d-82fc-f8b53dae7717"
Oct 31 01:03:35.783292 ip-10-0-133-197 kubenswrapper[2578]: E1031 01:03:35.783110    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8mqbd" podUID="565fefaf-82aa-42a8-9cc7-6517c4bddf1e"
Oct 31 01:03:37.782703 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:37.782626    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zs5g7"
Oct 31 01:03:37.782703 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:37.782657    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8mqbd"
Oct 31 01:03:37.782703 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:37.782656    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wdsmn"
Oct 31 01:03:37.783221 ip-10-0-133-197 kubenswrapper[2578]: E1031 01:03:37.782742    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zs5g7" podUID="aa4836a5-c71a-485d-82fc-f8b53dae7717"
Oct 31 01:03:37.783221 ip-10-0-133-197 kubenswrapper[2578]: E1031 01:03:37.782812    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8mqbd" podUID="565fefaf-82aa-42a8-9cc7-6517c4bddf1e"
Oct 31 01:03:37.783221 ip-10-0-133-197 kubenswrapper[2578]: E1031 01:03:37.782881    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wdsmn" podUID="8952fc79-c5a6-42ba-a296-8cb7a776f718"
Oct 31 01:03:39.622498 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:39.622472    2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-197.us-west-1.compute.internal" event="NodeReady"
Oct 31 01:03:39.622799 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:39.622568    2578 kubelet_node_status.go:477] "Fast updating node status as it just became ready"
Oct 31 01:03:39.735750 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:39.735691    2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-vbbq6"]
Oct 31 01:03:39.737565 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:39.737551    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-vbbq6"
Oct 31 01:03:39.775080 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:39.775053    2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-q4z5f"]
Oct 31 01:03:39.776562 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:39.776548    2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-jm6bm"]
Oct 31 01:03:39.776670 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:39.776655    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-q4z5f"
Oct 31 01:03:39.778129 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:39.778115    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-jm6bm"
Oct 31 01:03:39.782332 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:39.782303    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8mqbd"
Oct 31 01:03:39.782466 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:39.782441    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wdsmn"
Oct 31 01:03:39.782545 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:39.782476    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zs5g7"
Oct 31 01:03:39.786280 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:39.786262    2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-75575b5db8-wtq4d"]
Oct 31 01:03:39.787644 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:39.787629    2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-56d8464d7c-vpwjc"]
Oct 31 01:03:39.787755 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:39.787741    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-75575b5db8-wtq4d"
Oct 31 01:03:39.789018 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:39.789002    2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-jccjk"]
Oct 31 01:03:39.789636 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:39.789445    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-56d8464d7c-vpwjc"
Oct 31 01:03:39.792982 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:39.791878    2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-5964fff8f9-8k6pp"]
Oct 31 01:03:39.793230 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:39.793213    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-jccjk"
Oct 31 01:03:39.793809 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:39.793788    2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-7c4986d449-jkk5x"]
Oct 31 01:03:39.793963 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:39.793915    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-5964fff8f9-8k6pp"
Oct 31 01:03:39.795680 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:39.795659    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-7c4986d449-jkk5x"
Oct 31 01:03:39.825290 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:39.825268    2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-pdzj4"]
Oct 31 01:03:39.826714 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:39.826697    2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-v585z"]
Oct 31 01:03:39.826827 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:39.826810    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-pdzj4"
Oct 31 01:03:39.828482 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:39.828464    2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-n5q9v"]
Oct 31 01:03:39.828567 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:39.828556    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-v585z"
Oct 31 01:03:39.830127 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:39.830110    2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-wm4qt"]
Oct 31 01:03:39.830204 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:39.830193    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-n5q9v"
Oct 31 01:03:39.832017 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:39.832001    2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-9257n"]
Oct 31 01:03:39.832135 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:39.832121    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-wm4qt"
Oct 31 01:03:39.834456 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:39.834437    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7cd56b8dbc-9257n"
Oct 31 01:03:39.835900 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:39.835811    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"kube-root-ca.crt\""
Oct 31 01:03:39.836010 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:39.835991    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-qlxzn\""
Oct 31 01:03:39.836145 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:39.836012    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Oct 31 01:03:39.836145 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:39.836023    2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-srcwl"]
Oct 31 01:03:39.836145 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:39.836154    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-lc659\""
Oct 31 01:03:39.836337 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:39.836099    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Oct 31 01:03:39.836337 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:39.836076    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-tg5xh\""
Oct 31 01:03:39.837992 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:39.837972    2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-bpmjc"]
Oct 31 01:03:39.838480 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:39.838463    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-srcwl"
Oct 31 01:03:39.839547 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:39.839521    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Oct 31 01:03:39.839686 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:39.839669    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Oct 31 01:03:39.839745 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:39.839713    2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-t7r6c"]
Oct 31 01:03:39.839851 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:39.839834    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-bpmjc"
Oct 31 01:03:39.841992 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:39.841975    2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-7dpwv"]
Oct 31 01:03:39.842101 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:39.842085    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-t7r6c"
Oct 31 01:03:39.843574 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:39.843554    2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-q4z5f"]
Oct 31 01:03:39.843672 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:39.843588    2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-jm6bm"]
Oct 31 01:03:39.843672 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:39.843605    2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-vbbq6"]
Oct 31 01:03:39.843672 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:39.843624    2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-75575b5db8-wtq4d"]
Oct 31 01:03:39.843869 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:39.843682    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-7dpwv"
Oct 31 01:03:39.858992 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:39.858972    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/aa112079-b749-404c-9081-113220707fcd-registry-certificates\") pod \"image-registry-75575b5db8-wtq4d\" (UID: \"aa112079-b749-404c-9081-113220707fcd\") " pod="openshift-image-registry/image-registry-75575b5db8-wtq4d"
Oct 31 01:03:39.859079 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:39.859001    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/ff4dcba2-df8d-4324-9e0c-abb23a5e6982-installation-pull-secrets\") pod \"image-registry-56d8464d7c-vpwjc\" (UID: \"ff4dcba2-df8d-4324-9e0c-abb23a5e6982\") " pod="openshift-image-registry/image-registry-56d8464d7c-vpwjc"
Oct 31 01:03:39.859079 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:39.859019    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/ff4dcba2-df8d-4324-9e0c-abb23a5e6982-registry-certificates\") pod \"image-registry-56d8464d7c-vpwjc\" (UID: \"ff4dcba2-df8d-4324-9e0c-abb23a5e6982\") " pod="openshift-image-registry/image-registry-56d8464d7c-vpwjc"
Oct 31 01:03:39.859079 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:39.859038    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rpmxj\" (UniqueName: \"kubernetes.io/projected/aa112079-b749-404c-9081-113220707fcd-kube-api-access-rpmxj\") pod \"image-registry-75575b5db8-wtq4d\" (UID: \"aa112079-b749-404c-9081-113220707fcd\") " pod="openshift-image-registry/image-registry-75575b5db8-wtq4d"
Oct 31 01:03:39.859216 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:39.859087    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/ff4dcba2-df8d-4324-9e0c-abb23a5e6982-ca-trust-extracted\") pod \"image-registry-56d8464d7c-vpwjc\" (UID: \"ff4dcba2-df8d-4324-9e0c-abb23a5e6982\") " pod="openshift-image-registry/image-registry-56d8464d7c-vpwjc"
Oct 31 01:03:39.859216 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:39.859145    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v9dnm\" (UniqueName: \"kubernetes.io/projected/ff4dcba2-df8d-4324-9e0c-abb23a5e6982-kube-api-access-v9dnm\") pod \"image-registry-56d8464d7c-vpwjc\" (UID: \"ff4dcba2-df8d-4324-9e0c-abb23a5e6982\") " pod="openshift-image-registry/image-registry-56d8464d7c-vpwjc"
Oct 31 01:03:39.859216 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:39.859190    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/aa112079-b749-404c-9081-113220707fcd-image-registry-private-configuration\") pod \"image-registry-75575b5db8-wtq4d\" (UID: \"aa112079-b749-404c-9081-113220707fcd\") " pod="openshift-image-registry/image-registry-75575b5db8-wtq4d"
Oct 31 01:03:39.859383 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:39.859236    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ff4dcba2-df8d-4324-9e0c-abb23a5e6982-registry-tls\") pod \"image-registry-56d8464d7c-vpwjc\" (UID: \"ff4dcba2-df8d-4324-9e0c-abb23a5e6982\") " pod="openshift-image-registry/image-registry-56d8464d7c-vpwjc"
Oct 31 01:03:39.859383 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:39.859279    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/aa112079-b749-404c-9081-113220707fcd-ca-trust-extracted\") pod \"image-registry-75575b5db8-wtq4d\" (UID: \"aa112079-b749-404c-9081-113220707fcd\") " pod="openshift-image-registry/image-registry-75575b5db8-wtq4d"
Oct 31 01:03:39.859383 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:39.859353    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jkn67\" (UniqueName: \"kubernetes.io/projected/da8395ef-7464-410e-96ee-b27378a03fed-kube-api-access-jkn67\") pod \"kas-connection-checker-685895c7c9-vbbq6\" (UID: \"da8395ef-7464-410e-96ee-b27378a03fed\") " pod="kube-system/kas-connection-checker-685895c7c9-vbbq6"
Oct 31 01:03:39.859585 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:39.859502    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/ff4dcba2-df8d-4324-9e0c-abb23a5e6982-image-registry-private-configuration\") pod \"image-registry-56d8464d7c-vpwjc\" (UID: \"ff4dcba2-df8d-4324-9e0c-abb23a5e6982\") " pod="openshift-image-registry/image-registry-56d8464d7c-vpwjc"
Oct 31 01:03:39.859585 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:39.859568    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/aa112079-b749-404c-9081-113220707fcd-installation-pull-secrets\") pod \"image-registry-75575b5db8-wtq4d\" (UID: \"aa112079-b749-404c-9081-113220707fcd\") " pod="openshift-image-registry/image-registry-75575b5db8-wtq4d"
Oct 31 01:03:39.859680 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:39.859585    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/aa112079-b749-404c-9081-113220707fcd-bound-sa-token\") pod \"image-registry-75575b5db8-wtq4d\" (UID: \"aa112079-b749-404c-9081-113220707fcd\") " pod="openshift-image-registry/image-registry-75575b5db8-wtq4d"
Oct 31 01:03:39.859680 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:39.859604    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ff4dcba2-df8d-4324-9e0c-abb23a5e6982-trusted-ca\") pod \"image-registry-56d8464d7c-vpwjc\" (UID: \"ff4dcba2-df8d-4324-9e0c-abb23a5e6982\") " pod="openshift-image-registry/image-registry-56d8464d7c-vpwjc"
Oct 31 01:03:39.859680 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:39.859620    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/aa112079-b749-404c-9081-113220707fcd-registry-tls\") pod \"image-registry-75575b5db8-wtq4d\" (UID: \"aa112079-b749-404c-9081-113220707fcd\") " pod="openshift-image-registry/image-registry-75575b5db8-wtq4d"
Oct 31 01:03:39.859680 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:39.859637    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/aa112079-b749-404c-9081-113220707fcd-trusted-ca\") pod \"image-registry-75575b5db8-wtq4d\" (UID: \"aa112079-b749-404c-9081-113220707fcd\") " pod="openshift-image-registry/image-registry-75575b5db8-wtq4d"
Oct 31 01:03:39.859680 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:39.859664    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rz545\" (UniqueName: \"kubernetes.io/projected/d719eda9-bc9f-4307-952a-99d99aa0f62b-kube-api-access-rz545\") pod \"kas-connection-checker-685895c7c9-jm6bm\" (UID: \"d719eda9-bc9f-4307-952a-99d99aa0f62b\") " pod="kube-system/kas-connection-checker-685895c7c9-jm6bm"
Oct 31 01:03:39.859882 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:39.859736    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ff4dcba2-df8d-4324-9e0c-abb23a5e6982-bound-sa-token\") pod \"image-registry-56d8464d7c-vpwjc\" (UID: \"ff4dcba2-df8d-4324-9e0c-abb23a5e6982\") " pod="openshift-image-registry/image-registry-56d8464d7c-vpwjc"
Oct 31 01:03:39.859882 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:39.859799    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vms79\" (UniqueName: \"kubernetes.io/projected/0ab3954e-be64-4260-bca8-ff5d58bd6ea2-kube-api-access-vms79\") pod \"kas-connection-checker-685895c7c9-q4z5f\" (UID: \"0ab3954e-be64-4260-bca8-ff5d58bd6ea2\") " pod="kube-system/kas-connection-checker-685895c7c9-q4z5f"
Oct 31 01:03:39.875464 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:39.875435    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-n7jgv\""
Oct 31 01:03:39.875672 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:39.875639    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Oct 31 01:03:39.876248 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:39.876233    2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-v585z"]
Oct 31 01:03:39.876813 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:39.876795    2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-9257n"]
Oct 31 01:03:39.877501 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:39.877483    2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-7c4986d449-jkk5x"]
Oct 31 01:03:39.885080 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:39.885063    2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-wm4qt"]
Oct 31 01:03:39.889832 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:39.889814    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Oct 31 01:03:39.889928 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:39.889889    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Oct 31 01:03:39.889928 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:39.889923    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\""
Oct 31 01:03:39.890040 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:39.889964    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"openshift-service-ca.crt\""
Oct 31 01:03:39.890040 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:39.889973    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\""
Oct 31 01:03:39.890139 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:39.890055    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\""
Oct 31 01:03:39.890139 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:39.890060    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Oct 31 01:03:39.890139 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:39.890055    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Oct 31 01:03:39.890139 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:39.890077    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Oct 31 01:03:39.890139 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:39.890096    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\""
Oct 31 01:03:39.890139 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:39.890110    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\""
Oct 31 01:03:39.890139 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:39.890114    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\""
Oct 31 01:03:39.890139 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:39.890134    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Oct 31 01:03:39.890139 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:39.890100    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Oct 31 01:03:39.890579 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:39.890524    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\""
Oct 31 01:03:39.890579 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:39.890570    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\""
Oct 31 01:03:39.890679 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:39.890665    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Oct 31 01:03:39.890721 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:39.890686    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\""
Oct 31 01:03:39.890894 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:39.890882    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\""
Oct 31 01:03:39.890951 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:39.890930    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\""
Oct 31 01:03:39.890951 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:39.890933    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Oct 31 01:03:39.891051 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:39.890949    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\""
Oct 31 01:03:39.891100 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:39.890988    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\""
Oct 31 01:03:39.925614 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:39.925588    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\""
Oct 31 01:03:39.925614 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:39.925589    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-config\""
Oct 31 01:03:39.925765 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:39.925627    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\""
Oct 31 01:03:39.939991 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:39.939972    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\""
Oct 31 01:03:39.960143 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:39.960123    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ff4dcba2-df8d-4324-9e0c-abb23a5e6982-bound-sa-token\") pod \"image-registry-56d8464d7c-vpwjc\" (UID: \"ff4dcba2-df8d-4324-9e0c-abb23a5e6982\") " pod="openshift-image-registry/image-registry-56d8464d7c-vpwjc"
Oct 31 01:03:39.960257 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:39.960152    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/f273b9e5-0419-4587-a9c7-a0c8bec02acf-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-srcwl\" (UID: \"f273b9e5-0419-4587-a9c7-a0c8bec02acf\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-srcwl"
Oct 31 01:03:39.960257 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:39.960170    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vms79\" (UniqueName: \"kubernetes.io/projected/0ab3954e-be64-4260-bca8-ff5d58bd6ea2-kube-api-access-vms79\") pod \"kas-connection-checker-685895c7c9-q4z5f\" (UID: \"0ab3954e-be64-4260-bca8-ff5d58bd6ea2\") " pod="kube-system/kas-connection-checker-685895c7c9-q4z5f"
Oct 31 01:03:39.960257 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:39.960191    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/f0c0ada7-7688-4dc2-97fc-813e4d0c4ef3-serving-cert\") pod \"console-operator-7cd56b8dbc-9257n\" (UID: \"f0c0ada7-7688-4dc2-97fc-813e4d0c4ef3\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-9257n"
Oct 31 01:03:39.960257 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:39.960218    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/c8e98df5-b2f7-49dd-a565-ed60534da496-snapshots\") pod \"insights-operator-7c4986d449-jkk5x\" (UID: \"c8e98df5-b2f7-49dd-a565-ed60534da496\") " pod="openshift-insights/insights-operator-7c4986d449-jkk5x"
Oct 31 01:03:39.960488 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:39.960340    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e49453a8-c7d9-4248-a8ef-4a819b76806c-metrics-certs\") pod \"router-default-5964fff8f9-8k6pp\" (UID: \"e49453a8-c7d9-4248-a8ef-4a819b76806c\") " pod="openshift-ingress/router-default-5964fff8f9-8k6pp"
Oct 31 01:03:39.960488 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:39.960378    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c8e98df5-b2f7-49dd-a565-ed60534da496-service-ca-bundle\") pod \"insights-operator-7c4986d449-jkk5x\" (UID: \"c8e98df5-b2f7-49dd-a565-ed60534da496\") " pod="openshift-insights/insights-operator-7c4986d449-jkk5x"
Oct 31 01:03:39.960488 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:39.960411    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/aa112079-b749-404c-9081-113220707fcd-registry-certificates\") pod \"image-registry-75575b5db8-wtq4d\" (UID: \"aa112079-b749-404c-9081-113220707fcd\") " pod="openshift-image-registry/image-registry-75575b5db8-wtq4d"
Oct 31 01:03:39.960629 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:39.960553    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g9wls\" (UniqueName: \"kubernetes.io/projected/147bb9af-dfc7-4785-bb5e-906f25d65dd1-kube-api-access-g9wls\") pod \"ingress-canary-7dpwv\" (UID: \"147bb9af-dfc7-4785-bb5e-906f25d65dd1\") " pod="openshift-ingress-canary/ingress-canary-7dpwv"
Oct 31 01:03:39.960681 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:39.960650    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rpmxj\" (UniqueName: \"kubernetes.io/projected/aa112079-b749-404c-9081-113220707fcd-kube-api-access-rpmxj\") pod \"image-registry-75575b5db8-wtq4d\" (UID: \"aa112079-b749-404c-9081-113220707fcd\") " pod="openshift-image-registry/image-registry-75575b5db8-wtq4d"
Oct 31 01:03:39.960726 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:39.960685    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/9dfae6d6-b9a6-4d85-b594-9d947a9e6383-nginx-conf\") pod \"networking-console-plugin-86c478bb88-wm4qt\" (UID: \"9dfae6d6-b9a6-4d85-b594-9d947a9e6383\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-wm4qt"
Oct 31 01:03:39.960726 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:39.960717    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v9dnm\" (UniqueName: \"kubernetes.io/projected/ff4dcba2-df8d-4324-9e0c-abb23a5e6982-kube-api-access-v9dnm\") pod \"image-registry-56d8464d7c-vpwjc\" (UID: \"ff4dcba2-df8d-4324-9e0c-abb23a5e6982\") " pod="openshift-image-registry/image-registry-56d8464d7c-vpwjc"
Oct 31 01:03:39.960809 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:39.960747    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jkn67\" (UniqueName: \"kubernetes.io/projected/da8395ef-7464-410e-96ee-b27378a03fed-kube-api-access-jkn67\") pod \"kas-connection-checker-685895c7c9-vbbq6\" (UID: \"da8395ef-7464-410e-96ee-b27378a03fed\") " pod="kube-system/kas-connection-checker-685895c7c9-vbbq6"
Oct 31 01:03:39.960809 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:39.960772    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lldxn\" (UniqueName: \"kubernetes.io/projected/f273b9e5-0419-4587-a9c7-a0c8bec02acf-kube-api-access-lldxn\") pod \"cluster-samples-operator-6ddb6c8b94-srcwl\" (UID: \"f273b9e5-0419-4587-a9c7-a0c8bec02acf\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-srcwl"
Oct 31 01:03:39.960809 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:39.960801    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/f0c0ada7-7688-4dc2-97fc-813e4d0c4ef3-config\") pod \"console-operator-7cd56b8dbc-9257n\" (UID: \"f0c0ada7-7688-4dc2-97fc-813e4d0c4ef3\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-9257n"
Oct 31 01:03:39.960941 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:39.960906    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/aa112079-b749-404c-9081-113220707fcd-bound-sa-token\") pod \"image-registry-75575b5db8-wtq4d\" (UID: \"aa112079-b749-404c-9081-113220707fcd\") " pod="openshift-image-registry/image-registry-75575b5db8-wtq4d"
Oct 31 01:03:39.960994 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:39.960940    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/aa112079-b749-404c-9081-113220707fcd-installation-pull-secrets\") pod \"image-registry-75575b5db8-wtq4d\" (UID: \"aa112079-b749-404c-9081-113220707fcd\") " pod="openshift-image-registry/image-registry-75575b5db8-wtq4d"
Oct 31 01:03:39.960994 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:39.960968    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/e49453a8-c7d9-4248-a8ef-4a819b76806c-default-certificate\") pod \"router-default-5964fff8f9-8k6pp\" (UID: \"e49453a8-c7d9-4248-a8ef-4a819b76806c\") " pod="openshift-ingress/router-default-5964fff8f9-8k6pp"
Oct 31 01:03:39.961091 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:39.961057    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/c2be658b-ec92-4782-a78b-d2dccf2249cb-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-n5q9v\" (UID: \"c2be658b-ec92-4782-a78b-d2dccf2249cb\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-n5q9v"
Oct 31 01:03:39.961091 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:39.961069    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/aa112079-b749-404c-9081-113220707fcd-registry-certificates\") pod \"image-registry-75575b5db8-wtq4d\" (UID: \"aa112079-b749-404c-9081-113220707fcd\") " pod="openshift-image-registry/image-registry-75575b5db8-wtq4d"
Oct 31 01:03:39.961174 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:39.961152    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ff4dcba2-df8d-4324-9e0c-abb23a5e6982-trusted-ca\") pod \"image-registry-56d8464d7c-vpwjc\" (UID: \"ff4dcba2-df8d-4324-9e0c-abb23a5e6982\") " pod="openshift-image-registry/image-registry-56d8464d7c-vpwjc"
Oct 31 01:03:39.961208 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:39.961185    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/b66619db-fd81-4435-be2a-a5ea6111e204-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-jccjk\" (UID: \"b66619db-fd81-4435-be2a-a5ea6111e204\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-jccjk"
Oct 31 01:03:39.961273 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:39.961245    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f0c0ada7-7688-4dc2-97fc-813e4d0c4ef3-trusted-ca\") pod \"console-operator-7cd56b8dbc-9257n\" (UID: \"f0c0ada7-7688-4dc2-97fc-813e4d0c4ef3\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-9257n"
Oct 31 01:03:39.961440 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:39.961298    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6qc9x\" (UniqueName: \"kubernetes.io/projected/69ff2b62-4588-4cb0-bdad-747c3692138c-kube-api-access-6qc9x\") pod \"network-check-source-6f78d59f99-bpmjc\" (UID: \"69ff2b62-4588-4cb0-bdad-747c3692138c\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-bpmjc"
Oct 31 01:03:39.961440 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:39.961358    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zlmlp\" (UniqueName: \"kubernetes.io/projected/f0c0ada7-7688-4dc2-97fc-813e4d0c4ef3-kube-api-access-zlmlp\") pod \"console-operator-7cd56b8dbc-9257n\" (UID: \"f0c0ada7-7688-4dc2-97fc-813e4d0c4ef3\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-9257n"
Oct 31 01:03:39.961440 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:39.961382    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/147bb9af-dfc7-4785-bb5e-906f25d65dd1-cert\") pod \"ingress-canary-7dpwv\" (UID: \"147bb9af-dfc7-4785-bb5e-906f25d65dd1\") " pod="openshift-ingress-canary/ingress-canary-7dpwv"
Oct 31 01:03:39.961440 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:39.961424    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-59kdl\" (UniqueName: \"kubernetes.io/projected/b66619db-fd81-4435-be2a-a5ea6111e204-kube-api-access-59kdl\") pod \"cluster-monitoring-operator-884576c57-jccjk\" (UID: \"b66619db-fd81-4435-be2a-a5ea6111e204\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-jccjk"
Oct 31 01:03:39.961618 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:39.961452    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/85bb69db-c69b-467e-adeb-950bf2947c94-metrics-tls\") pod \"dns-default-t7r6c\" (UID: \"85bb69db-c69b-467e-adeb-950bf2947c94\") " pod="openshift-dns/dns-default-t7r6c"
Oct 31 01:03:39.961618 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:39.961479    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tlnlx\" (UniqueName: \"kubernetes.io/projected/c8e98df5-b2f7-49dd-a565-ed60534da496-kube-api-access-tlnlx\") pod \"insights-operator-7c4986d449-jkk5x\" (UID: \"c8e98df5-b2f7-49dd-a565-ed60534da496\") " pod="openshift-insights/insights-operator-7c4986d449-jkk5x"
Oct 31 01:03:39.961618 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:39.961534    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/f273b9e5-0419-4587-a9c7-a0c8bec02acf-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-srcwl\" (UID: \"f273b9e5-0419-4587-a9c7-a0c8bec02acf\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-srcwl"
Oct 31 01:03:39.961618 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:39.961581    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/85bb69db-c69b-467e-adeb-950bf2947c94-config-volume\") pod \"dns-default-t7r6c\" (UID: \"85bb69db-c69b-467e-adeb-950bf2947c94\") " pod="openshift-dns/dns-default-t7r6c"
Oct 31 01:03:39.961764 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:39.961670    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/9dfae6d6-b9a6-4d85-b594-9d947a9e6383-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-wm4qt\" (UID: \"9dfae6d6-b9a6-4d85-b594-9d947a9e6383\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-wm4qt"
Oct 31 01:03:39.961764 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:39.961696    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/c8e98df5-b2f7-49dd-a565-ed60534da496-serving-cert\") pod \"insights-operator-7c4986d449-jkk5x\" (UID: \"c8e98df5-b2f7-49dd-a565-ed60534da496\") " pod="openshift-insights/insights-operator-7c4986d449-jkk5x"
Oct 31 01:03:39.961764 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:39.961723    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/ff4dcba2-df8d-4324-9e0c-abb23a5e6982-installation-pull-secrets\") pod \"image-registry-56d8464d7c-vpwjc\" (UID: \"ff4dcba2-df8d-4324-9e0c-abb23a5e6982\") " pod="openshift-image-registry/image-registry-56d8464d7c-vpwjc"
Oct 31 01:03:39.961889 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:39.961779    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ks6fn\" (UniqueName: \"kubernetes.io/projected/e54a9dc6-36e6-4f1b-9c3a-623a520e11da-kube-api-access-ks6fn\") pod \"service-ca-operator-544d749c8c-pdzj4\" (UID: \"e54a9dc6-36e6-4f1b-9c3a-623a520e11da\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-pdzj4"
Oct 31 01:03:39.961975 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:39.961952    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/ff4dcba2-df8d-4324-9e0c-abb23a5e6982-registry-certificates\") pod \"image-registry-56d8464d7c-vpwjc\" (UID: \"ff4dcba2-df8d-4324-9e0c-abb23a5e6982\") " pod="openshift-image-registry/image-registry-56d8464d7c-vpwjc"
Oct 31 01:03:39.962024 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:39.961992    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c8e98df5-b2f7-49dd-a565-ed60534da496-tmp\") pod \"insights-operator-7c4986d449-jkk5x\" (UID: \"c8e98df5-b2f7-49dd-a565-ed60534da496\") " pod="openshift-insights/insights-operator-7c4986d449-jkk5x"
Oct 31 01:03:39.962024 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:39.962015    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/ff4dcba2-df8d-4324-9e0c-abb23a5e6982-ca-trust-extracted\") pod \"image-registry-56d8464d7c-vpwjc\" (UID: \"ff4dcba2-df8d-4324-9e0c-abb23a5e6982\") " pod="openshift-image-registry/image-registry-56d8464d7c-vpwjc"
Oct 31 01:03:39.962117 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:39.962036    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/aa112079-b749-404c-9081-113220707fcd-image-registry-private-configuration\") pod \"image-registry-75575b5db8-wtq4d\" (UID: \"aa112079-b749-404c-9081-113220707fcd\") " pod="openshift-image-registry/image-registry-75575b5db8-wtq4d"
Oct 31 01:03:39.962117 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:39.962062    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/aa112079-b749-404c-9081-113220707fcd-ca-trust-extracted\") pod \"image-registry-75575b5db8-wtq4d\" (UID: \"aa112079-b749-404c-9081-113220707fcd\") " pod="openshift-image-registry/image-registry-75575b5db8-wtq4d"
Oct 31 01:03:39.962117 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:39.962080    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-98n75\" (UniqueName: \"kubernetes.io/projected/e49453a8-c7d9-4248-a8ef-4a819b76806c-kube-api-access-98n75\") pod \"router-default-5964fff8f9-8k6pp\" (UID: \"e49453a8-c7d9-4248-a8ef-4a819b76806c\") " pod="openshift-ingress/router-default-5964fff8f9-8k6pp"
Oct 31 01:03:39.962117 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:39.962106    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ff4dcba2-df8d-4324-9e0c-abb23a5e6982-registry-tls\") pod \"image-registry-56d8464d7c-vpwjc\" (UID: \"ff4dcba2-df8d-4324-9e0c-abb23a5e6982\") " pod="openshift-image-registry/image-registry-56d8464d7c-vpwjc"
Oct 31 01:03:39.962369 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:39.962138    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e49453a8-c7d9-4248-a8ef-4a819b76806c-service-ca-bundle\") pod \"router-default-5964fff8f9-8k6pp\" (UID: \"e49453a8-c7d9-4248-a8ef-4a819b76806c\") " pod="openshift-ingress/router-default-5964fff8f9-8k6pp"
Oct 31 01:03:39.962369 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:39.962162    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c8e98df5-b2f7-49dd-a565-ed60534da496-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-jkk5x\" (UID: \"c8e98df5-b2f7-49dd-a565-ed60534da496\") " pod="openshift-insights/insights-operator-7c4986d449-jkk5x"
Oct 31 01:03:39.962369 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:39.962185    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/b66619db-fd81-4435-be2a-a5ea6111e204-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-jccjk\" (UID: \"b66619db-fd81-4435-be2a-a5ea6111e204\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-jccjk"
Oct 31 01:03:39.962369 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:39.962233    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/ff4dcba2-df8d-4324-9e0c-abb23a5e6982-image-registry-private-configuration\") pod \"image-registry-56d8464d7c-vpwjc\" (UID: \"ff4dcba2-df8d-4324-9e0c-abb23a5e6982\") " pod="openshift-image-registry/image-registry-56d8464d7c-vpwjc"
Oct 31 01:03:39.962369 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:39.962254    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/aa112079-b749-404c-9081-113220707fcd-registry-tls\") pod \"image-registry-75575b5db8-wtq4d\" (UID: \"aa112079-b749-404c-9081-113220707fcd\") " pod="openshift-image-registry/image-registry-75575b5db8-wtq4d"
Oct 31 01:03:39.962369 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:39.962274    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/e49453a8-c7d9-4248-a8ef-4a819b76806c-stats-auth\") pod \"router-default-5964fff8f9-8k6pp\" (UID: \"e49453a8-c7d9-4248-a8ef-4a819b76806c\") " pod="openshift-ingress/router-default-5964fff8f9-8k6pp"
Oct 31 01:03:39.962369 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:39.962297    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8n8tt\" (UniqueName: \"kubernetes.io/projected/85bb69db-c69b-467e-adeb-950bf2947c94-kube-api-access-8n8tt\") pod \"dns-default-t7r6c\" (UID: \"85bb69db-c69b-467e-adeb-950bf2947c94\") " pod="openshift-dns/dns-default-t7r6c"
Oct 31 01:03:39.962369 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:39.962343    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/e54a9dc6-36e6-4f1b-9c3a-623a520e11da-serving-cert\") pod \"service-ca-operator-544d749c8c-pdzj4\" (UID: \"e54a9dc6-36e6-4f1b-9c3a-623a520e11da\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-pdzj4"
Oct 31 01:03:39.962369 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:39.962369    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bjgwm\" (UniqueName: \"kubernetes.io/projected/c2be658b-ec92-4782-a78b-d2dccf2249cb-kube-api-access-bjgwm\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-n5q9v\" (UID: \"c2be658b-ec92-4782-a78b-d2dccf2249cb\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-n5q9v"
Oct 31 01:03:39.962799 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:39.962390    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/e54a9dc6-36e6-4f1b-9c3a-623a520e11da-config\") pod \"service-ca-operator-544d749c8c-pdzj4\" (UID: \"e54a9dc6-36e6-4f1b-9c3a-623a520e11da\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-pdzj4"
Oct 31 01:03:39.962799 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:39.962414    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/aa112079-b749-404c-9081-113220707fcd-trusted-ca\") pod \"image-registry-75575b5db8-wtq4d\" (UID: \"aa112079-b749-404c-9081-113220707fcd\") " pod="openshift-image-registry/image-registry-75575b5db8-wtq4d"
Oct 31 01:03:39.962799 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:39.962446    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/c2be658b-ec92-4782-a78b-d2dccf2249cb-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-n5q9v\" (UID: \"c2be658b-ec92-4782-a78b-d2dccf2249cb\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-n5q9v"
Oct 31 01:03:39.962799 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:39.962447    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/ff4dcba2-df8d-4324-9e0c-abb23a5e6982-ca-trust-extracted\") pod \"image-registry-56d8464d7c-vpwjc\" (UID: \"ff4dcba2-df8d-4324-9e0c-abb23a5e6982\") " pod="openshift-image-registry/image-registry-56d8464d7c-vpwjc"
Oct 31 01:03:39.962799 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:39.962467    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j78lj\" (UniqueName: \"kubernetes.io/projected/eafc69c5-15f1-4e4c-97ca-be586e55fbae-kube-api-access-j78lj\") pod \"volume-data-source-validator-58c9f5f46d-v585z\" (UID: \"eafc69c5-15f1-4e4c-97ca-be586e55fbae\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-v585z"
Oct 31 01:03:39.962799 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:39.962494    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rz545\" (UniqueName: \"kubernetes.io/projected/d719eda9-bc9f-4307-952a-99d99aa0f62b-kube-api-access-rz545\") pod \"kas-connection-checker-685895c7c9-jm6bm\" (UID: \"d719eda9-bc9f-4307-952a-99d99aa0f62b\") " pod="kube-system/kas-connection-checker-685895c7c9-jm6bm"
Oct 31 01:03:39.962799 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:39.962517    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/85bb69db-c69b-467e-adeb-950bf2947c94-tmp-dir\") pod \"dns-default-t7r6c\" (UID: \"85bb69db-c69b-467e-adeb-950bf2947c94\") " pod="openshift-dns/dns-default-t7r6c"
Oct 31 01:03:39.962799 ip-10-0-133-197 kubenswrapper[2578]: E1031 01:03:39.962540    2578 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 31 01:03:39.962799 ip-10-0-133-197 kubenswrapper[2578]: E1031 01:03:39.962555    2578 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-75575b5db8-wtq4d: secret "image-registry-tls" not found
Oct 31 01:03:39.962799 ip-10-0-133-197 kubenswrapper[2578]: E1031 01:03:39.962606    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/aa112079-b749-404c-9081-113220707fcd-registry-tls podName:aa112079-b749-404c-9081-113220707fcd nodeName:}" failed. No retries permitted until 2026-10-31 01:03:40.462587413 +0000 UTC m=+32.148066812 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/aa112079-b749-404c-9081-113220707fcd-registry-tls") pod "image-registry-75575b5db8-wtq4d" (UID: "aa112079-b749-404c-9081-113220707fcd") : secret "image-registry-tls" not found
Oct 31 01:03:39.962799 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:39.962617    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/aa112079-b749-404c-9081-113220707fcd-ca-trust-extracted\") pod \"image-registry-75575b5db8-wtq4d\" (UID: \"aa112079-b749-404c-9081-113220707fcd\") " pod="openshift-image-registry/image-registry-75575b5db8-wtq4d"
Oct 31 01:03:39.962799 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:39.962624    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/ff4dcba2-df8d-4324-9e0c-abb23a5e6982-registry-certificates\") pod \"image-registry-56d8464d7c-vpwjc\" (UID: \"ff4dcba2-df8d-4324-9e0c-abb23a5e6982\") " pod="openshift-image-registry/image-registry-56d8464d7c-vpwjc"
Oct 31 01:03:39.963866 ip-10-0-133-197 kubenswrapper[2578]: E1031 01:03:39.963445    2578 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 31 01:03:39.963866 ip-10-0-133-197 kubenswrapper[2578]: E1031 01:03:39.963461    2578 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-56d8464d7c-vpwjc: secret "image-registry-tls" not found
Oct 31 01:03:39.963866 ip-10-0-133-197 kubenswrapper[2578]: E1031 01:03:39.963507    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ff4dcba2-df8d-4324-9e0c-abb23a5e6982-registry-tls podName:ff4dcba2-df8d-4324-9e0c-abb23a5e6982 nodeName:}" failed. No retries permitted until 2026-10-31 01:03:40.46349105 +0000 UTC m=+32.148970455 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/ff4dcba2-df8d-4324-9e0c-abb23a5e6982-registry-tls") pod "image-registry-56d8464d7c-vpwjc" (UID: "ff4dcba2-df8d-4324-9e0c-abb23a5e6982") : secret "image-registry-tls" not found
Oct 31 01:03:39.966074 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:39.966047    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/ff4dcba2-df8d-4324-9e0c-abb23a5e6982-image-registry-private-configuration\") pod \"image-registry-56d8464d7c-vpwjc\" (UID: \"ff4dcba2-df8d-4324-9e0c-abb23a5e6982\") " pod="openshift-image-registry/image-registry-56d8464d7c-vpwjc"
Oct 31 01:03:39.966172 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:39.966158    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/aa112079-b749-404c-9081-113220707fcd-image-registry-private-configuration\") pod \"image-registry-75575b5db8-wtq4d\" (UID: \"aa112079-b749-404c-9081-113220707fcd\") " pod="openshift-image-registry/image-registry-75575b5db8-wtq4d"
Oct 31 01:03:39.966261 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:39.966241    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/aa112079-b749-404c-9081-113220707fcd-installation-pull-secrets\") pod \"image-registry-75575b5db8-wtq4d\" (UID: \"aa112079-b749-404c-9081-113220707fcd\") " pod="openshift-image-registry/image-registry-75575b5db8-wtq4d"
Oct 31 01:03:39.966297 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:39.966241    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/ff4dcba2-df8d-4324-9e0c-abb23a5e6982-installation-pull-secrets\") pod \"image-registry-56d8464d7c-vpwjc\" (UID: \"ff4dcba2-df8d-4324-9e0c-abb23a5e6982\") " pod="openshift-image-registry/image-registry-56d8464d7c-vpwjc"
Oct 31 01:03:39.975679 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:39.975660    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Oct 31 01:03:39.975766 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:39.975678    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Oct 31 01:03:39.975815 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:39.975768    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\""
Oct 31 01:03:39.976307 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:39.976189    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\""
Oct 31 01:03:39.976307 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:39.976207    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\""
Oct 31 01:03:39.976307 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:39.976213    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\""
Oct 31 01:03:39.976307 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:39.976222    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\""
Oct 31 01:03:39.976307 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:39.976254    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\""
Oct 31 01:03:39.976307 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:39.976258    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\""
Oct 31 01:03:39.976307 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:39.976263    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\""
Oct 31 01:03:39.976307 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:39.976297    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\""
Oct 31 01:03:39.976307 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:39.976308    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Oct 31 01:03:39.976307 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:39.976342    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\""
Oct 31 01:03:39.976307 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:39.976369    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-dockercfg-jsq9r\""
Oct 31 01:03:39.976307 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:39.976385    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\""
Oct 31 01:03:39.976788 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:39.976421    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Oct 31 01:03:39.976788 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:39.976462    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-xgzbp\""
Oct 31 01:03:39.976788 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:39.976606    2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-n5q9v"]
Oct 31 01:03:39.977306 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:39.977288    2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-t7r6c"]
Oct 31 01:03:39.977931 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:39.977919    2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-56d8464d7c-vpwjc"]
Oct 31 01:03:39.978611 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:39.978595    2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-5964fff8f9-8k6pp"]
Oct 31 01:03:39.979803 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:39.979788    2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-jccjk"]
Oct 31 01:03:39.980372 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:39.980358    2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-bpmjc"]
Oct 31 01:03:39.980941 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:39.980925    2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-7dpwv"]
Oct 31 01:03:39.983362 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:39.983282    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ff4dcba2-df8d-4324-9e0c-abb23a5e6982-trusted-ca\") pod \"image-registry-56d8464d7c-vpwjc\" (UID: \"ff4dcba2-df8d-4324-9e0c-abb23a5e6982\") " pod="openshift-image-registry/image-registry-56d8464d7c-vpwjc"
Oct 31 01:03:39.986066 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:39.984435    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-dvb7t\""
Oct 31 01:03:39.986066 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:39.985167    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/aa112079-b749-404c-9081-113220707fcd-trusted-ca\") pod \"image-registry-75575b5db8-wtq4d\" (UID: \"aa112079-b749-404c-9081-113220707fcd\") " pod="openshift-image-registry/image-registry-75575b5db8-wtq4d"
Oct 31 01:03:39.986174 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:39.986093    2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-srcwl"]
Oct 31 01:03:39.987080 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:39.987064    2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-pdzj4"]
Oct 31 01:03:39.990505 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:39.990484    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Oct 31 01:03:39.990588 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:39.990540    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Oct 31 01:03:39.990588 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:39.990578    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Oct 31 01:03:40.018391 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:40.018349    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vms79\" (UniqueName: \"kubernetes.io/projected/0ab3954e-be64-4260-bca8-ff5d58bd6ea2-kube-api-access-vms79\") pod \"kas-connection-checker-685895c7c9-q4z5f\" (UID: \"0ab3954e-be64-4260-bca8-ff5d58bd6ea2\") " pod="kube-system/kas-connection-checker-685895c7c9-q4z5f"
Oct 31 01:03:40.018497 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:40.018480    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rpmxj\" (UniqueName: \"kubernetes.io/projected/aa112079-b749-404c-9081-113220707fcd-kube-api-access-rpmxj\") pod \"image-registry-75575b5db8-wtq4d\" (UID: \"aa112079-b749-404c-9081-113220707fcd\") " pod="openshift-image-registry/image-registry-75575b5db8-wtq4d"
Oct 31 01:03:40.018801 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:40.018779    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jkn67\" (UniqueName: \"kubernetes.io/projected/da8395ef-7464-410e-96ee-b27378a03fed-kube-api-access-jkn67\") pod \"kas-connection-checker-685895c7c9-vbbq6\" (UID: \"da8395ef-7464-410e-96ee-b27378a03fed\") " pod="kube-system/kas-connection-checker-685895c7c9-vbbq6"
Oct 31 01:03:40.019541 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:40.019523    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ff4dcba2-df8d-4324-9e0c-abb23a5e6982-bound-sa-token\") pod \"image-registry-56d8464d7c-vpwjc\" (UID: \"ff4dcba2-df8d-4324-9e0c-abb23a5e6982\") " pod="openshift-image-registry/image-registry-56d8464d7c-vpwjc"
Oct 31 01:03:40.020100 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:40.020083    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v9dnm\" (UniqueName: \"kubernetes.io/projected/ff4dcba2-df8d-4324-9e0c-abb23a5e6982-kube-api-access-v9dnm\") pod \"image-registry-56d8464d7c-vpwjc\" (UID: \"ff4dcba2-df8d-4324-9e0c-abb23a5e6982\") " pod="openshift-image-registry/image-registry-56d8464d7c-vpwjc"
Oct 31 01:03:40.020309 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:40.020293    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/aa112079-b749-404c-9081-113220707fcd-bound-sa-token\") pod \"image-registry-75575b5db8-wtq4d\" (UID: \"aa112079-b749-404c-9081-113220707fcd\") " pod="openshift-image-registry/image-registry-75575b5db8-wtq4d"
Oct 31 01:03:40.023134 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:40.023116    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rz545\" (UniqueName: \"kubernetes.io/projected/d719eda9-bc9f-4307-952a-99d99aa0f62b-kube-api-access-rz545\") pod \"kas-connection-checker-685895c7c9-jm6bm\" (UID: \"d719eda9-bc9f-4307-952a-99d99aa0f62b\") " pod="kube-system/kas-connection-checker-685895c7c9-jm6bm"
Oct 31 01:03:40.047185 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:40.047169    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-vbbq6"
Oct 31 01:03:40.063722 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:40.063588    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/e54a9dc6-36e6-4f1b-9c3a-623a520e11da-serving-cert\") pod \"service-ca-operator-544d749c8c-pdzj4\" (UID: \"e54a9dc6-36e6-4f1b-9c3a-623a520e11da\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-pdzj4"
Oct 31 01:03:40.063819 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:40.063738    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bjgwm\" (UniqueName: \"kubernetes.io/projected/c2be658b-ec92-4782-a78b-d2dccf2249cb-kube-api-access-bjgwm\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-n5q9v\" (UID: \"c2be658b-ec92-4782-a78b-d2dccf2249cb\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-n5q9v"
Oct 31 01:03:40.063819 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:40.063755    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/e54a9dc6-36e6-4f1b-9c3a-623a520e11da-config\") pod \"service-ca-operator-544d749c8c-pdzj4\" (UID: \"e54a9dc6-36e6-4f1b-9c3a-623a520e11da\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-pdzj4"
Oct 31 01:03:40.063819 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:40.063778    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/c2be658b-ec92-4782-a78b-d2dccf2249cb-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-n5q9v\" (UID: \"c2be658b-ec92-4782-a78b-d2dccf2249cb\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-n5q9v"
Oct 31 01:03:40.063819 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:40.063793    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j78lj\" (UniqueName: \"kubernetes.io/projected/eafc69c5-15f1-4e4c-97ca-be586e55fbae-kube-api-access-j78lj\") pod \"volume-data-source-validator-58c9f5f46d-v585z\" (UID: \"eafc69c5-15f1-4e4c-97ca-be586e55fbae\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-v585z"
Oct 31 01:03:40.063819 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:40.063810    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/85bb69db-c69b-467e-adeb-950bf2947c94-tmp-dir\") pod \"dns-default-t7r6c\" (UID: \"85bb69db-c69b-467e-adeb-950bf2947c94\") " pod="openshift-dns/dns-default-t7r6c"
Oct 31 01:03:40.064046 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:40.063828    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/f273b9e5-0419-4587-a9c7-a0c8bec02acf-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-srcwl\" (UID: \"f273b9e5-0419-4587-a9c7-a0c8bec02acf\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-srcwl"
Oct 31 01:03:40.064046 ip-10-0-133-197 kubenswrapper[2578]: E1031 01:03:40.063912    2578 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Oct 31 01:03:40.064046 ip-10-0-133-197 kubenswrapper[2578]: E1031 01:03:40.063973    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f273b9e5-0419-4587-a9c7-a0c8bec02acf-samples-operator-tls podName:f273b9e5-0419-4587-a9c7-a0c8bec02acf nodeName:}" failed. No retries permitted until 2026-10-31 01:03:40.563957059 +0000 UTC m=+32.249436463 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/f273b9e5-0419-4587-a9c7-a0c8bec02acf-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-srcwl" (UID: "f273b9e5-0419-4587-a9c7-a0c8bec02acf") : secret "samples-operator-tls" not found
Oct 31 01:03:40.064046 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:40.064000    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/f0c0ada7-7688-4dc2-97fc-813e4d0c4ef3-serving-cert\") pod \"console-operator-7cd56b8dbc-9257n\" (UID: \"f0c0ada7-7688-4dc2-97fc-813e4d0c4ef3\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-9257n"
Oct 31 01:03:40.064046 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:40.064027    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/c8e98df5-b2f7-49dd-a565-ed60534da496-snapshots\") pod \"insights-operator-7c4986d449-jkk5x\" (UID: \"c8e98df5-b2f7-49dd-a565-ed60534da496\") " pod="openshift-insights/insights-operator-7c4986d449-jkk5x"
Oct 31 01:03:40.064345 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:40.064059    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e49453a8-c7d9-4248-a8ef-4a819b76806c-metrics-certs\") pod \"router-default-5964fff8f9-8k6pp\" (UID: \"e49453a8-c7d9-4248-a8ef-4a819b76806c\") " pod="openshift-ingress/router-default-5964fff8f9-8k6pp"
Oct 31 01:03:40.064345 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:40.064083    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c8e98df5-b2f7-49dd-a565-ed60534da496-service-ca-bundle\") pod \"insights-operator-7c4986d449-jkk5x\" (UID: \"c8e98df5-b2f7-49dd-a565-ed60534da496\") " pod="openshift-insights/insights-operator-7c4986d449-jkk5x"
Oct 31 01:03:40.064345 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:40.064117    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g9wls\" (UniqueName: \"kubernetes.io/projected/147bb9af-dfc7-4785-bb5e-906f25d65dd1-kube-api-access-g9wls\") pod \"ingress-canary-7dpwv\" (UID: \"147bb9af-dfc7-4785-bb5e-906f25d65dd1\") " pod="openshift-ingress-canary/ingress-canary-7dpwv"
Oct 31 01:03:40.064345 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:40.064146    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/9dfae6d6-b9a6-4d85-b594-9d947a9e6383-nginx-conf\") pod \"networking-console-plugin-86c478bb88-wm4qt\" (UID: \"9dfae6d6-b9a6-4d85-b594-9d947a9e6383\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-wm4qt"
Oct 31 01:03:40.064345 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:40.064178    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lldxn\" (UniqueName: \"kubernetes.io/projected/f273b9e5-0419-4587-a9c7-a0c8bec02acf-kube-api-access-lldxn\") pod \"cluster-samples-operator-6ddb6c8b94-srcwl\" (UID: \"f273b9e5-0419-4587-a9c7-a0c8bec02acf\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-srcwl"
Oct 31 01:03:40.064345 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:40.064202    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/f0c0ada7-7688-4dc2-97fc-813e4d0c4ef3-config\") pod \"console-operator-7cd56b8dbc-9257n\" (UID: \"f0c0ada7-7688-4dc2-97fc-813e4d0c4ef3\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-9257n"
Oct 31 01:03:40.064345 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:40.064239    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/e49453a8-c7d9-4248-a8ef-4a819b76806c-default-certificate\") pod \"router-default-5964fff8f9-8k6pp\" (UID: \"e49453a8-c7d9-4248-a8ef-4a819b76806c\") " pod="openshift-ingress/router-default-5964fff8f9-8k6pp"
Oct 31 01:03:40.064345 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:40.064257    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/85bb69db-c69b-467e-adeb-950bf2947c94-tmp-dir\") pod \"dns-default-t7r6c\" (UID: \"85bb69db-c69b-467e-adeb-950bf2947c94\") " pod="openshift-dns/dns-default-t7r6c"
Oct 31 01:03:40.064345 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:40.064263    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/c2be658b-ec92-4782-a78b-d2dccf2249cb-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-n5q9v\" (UID: \"c2be658b-ec92-4782-a78b-d2dccf2249cb\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-n5q9v"
Oct 31 01:03:40.064345 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:40.064302    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/b66619db-fd81-4435-be2a-a5ea6111e204-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-jccjk\" (UID: \"b66619db-fd81-4435-be2a-a5ea6111e204\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-jccjk"
Oct 31 01:03:40.064792 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:40.064369    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f0c0ada7-7688-4dc2-97fc-813e4d0c4ef3-trusted-ca\") pod \"console-operator-7cd56b8dbc-9257n\" (UID: \"f0c0ada7-7688-4dc2-97fc-813e4d0c4ef3\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-9257n"
Oct 31 01:03:40.064792 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:40.064400    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6qc9x\" (UniqueName: \"kubernetes.io/projected/69ff2b62-4588-4cb0-bdad-747c3692138c-kube-api-access-6qc9x\") pod \"network-check-source-6f78d59f99-bpmjc\" (UID: \"69ff2b62-4588-4cb0-bdad-747c3692138c\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-bpmjc"
Oct 31 01:03:40.064792 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:40.064422    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zlmlp\" (UniqueName: \"kubernetes.io/projected/f0c0ada7-7688-4dc2-97fc-813e4d0c4ef3-kube-api-access-zlmlp\") pod \"console-operator-7cd56b8dbc-9257n\" (UID: \"f0c0ada7-7688-4dc2-97fc-813e4d0c4ef3\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-9257n"
Oct 31 01:03:40.064792 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:40.064446    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/147bb9af-dfc7-4785-bb5e-906f25d65dd1-cert\") pod \"ingress-canary-7dpwv\" (UID: \"147bb9af-dfc7-4785-bb5e-906f25d65dd1\") " pod="openshift-ingress-canary/ingress-canary-7dpwv"
Oct 31 01:03:40.064792 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:40.064490    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-59kdl\" (UniqueName: \"kubernetes.io/projected/b66619db-fd81-4435-be2a-a5ea6111e204-kube-api-access-59kdl\") pod \"cluster-monitoring-operator-884576c57-jccjk\" (UID: \"b66619db-fd81-4435-be2a-a5ea6111e204\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-jccjk"
Oct 31 01:03:40.064792 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:40.064518    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/85bb69db-c69b-467e-adeb-950bf2947c94-metrics-tls\") pod \"dns-default-t7r6c\" (UID: \"85bb69db-c69b-467e-adeb-950bf2947c94\") " pod="openshift-dns/dns-default-t7r6c"
Oct 31 01:03:40.064792 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:40.064544    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tlnlx\" (UniqueName: \"kubernetes.io/projected/c8e98df5-b2f7-49dd-a565-ed60534da496-kube-api-access-tlnlx\") pod \"insights-operator-7c4986d449-jkk5x\" (UID: \"c8e98df5-b2f7-49dd-a565-ed60534da496\") " pod="openshift-insights/insights-operator-7c4986d449-jkk5x"
Oct 31 01:03:40.064792 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:40.064555    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/c2be658b-ec92-4782-a78b-d2dccf2249cb-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-n5q9v\" (UID: \"c2be658b-ec92-4782-a78b-d2dccf2249cb\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-n5q9v"
Oct 31 01:03:40.064792 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:40.064563    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/c8e98df5-b2f7-49dd-a565-ed60534da496-snapshots\") pod \"insights-operator-7c4986d449-jkk5x\" (UID: \"c8e98df5-b2f7-49dd-a565-ed60534da496\") " pod="openshift-insights/insights-operator-7c4986d449-jkk5x"
Oct 31 01:03:40.064792 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:40.064574    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/f273b9e5-0419-4587-a9c7-a0c8bec02acf-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-srcwl\" (UID: \"f273b9e5-0419-4587-a9c7-a0c8bec02acf\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-srcwl"
Oct 31 01:03:40.064792 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:40.064609    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/85bb69db-c69b-467e-adeb-950bf2947c94-config-volume\") pod \"dns-default-t7r6c\" (UID: \"85bb69db-c69b-467e-adeb-950bf2947c94\") " pod="openshift-dns/dns-default-t7r6c"
Oct 31 01:03:40.064792 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:40.064620    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/e54a9dc6-36e6-4f1b-9c3a-623a520e11da-config\") pod \"service-ca-operator-544d749c8c-pdzj4\" (UID: \"e54a9dc6-36e6-4f1b-9c3a-623a520e11da\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-pdzj4"
Oct 31 01:03:40.064792 ip-10-0-133-197 kubenswrapper[2578]: E1031 01:03:40.064659    2578 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Oct 31 01:03:40.064792 ip-10-0-133-197 kubenswrapper[2578]: E1031 01:03:40.064712    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b66619db-fd81-4435-be2a-a5ea6111e204-cluster-monitoring-operator-tls podName:b66619db-fd81-4435-be2a-a5ea6111e204 nodeName:}" failed. No retries permitted until 2026-10-31 01:03:40.564699571 +0000 UTC m=+32.250178963 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/b66619db-fd81-4435-be2a-a5ea6111e204-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-jccjk" (UID: "b66619db-fd81-4435-be2a-a5ea6111e204") : secret "cluster-monitoring-operator-tls" not found
Oct 31 01:03:40.064792 ip-10-0-133-197 kubenswrapper[2578]: E1031 01:03:40.064716    2578 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Oct 31 01:03:40.065532 ip-10-0-133-197 kubenswrapper[2578]: E1031 01:03:40.064775    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9dfae6d6-b9a6-4d85-b594-9d947a9e6383-networking-console-plugin-cert podName:9dfae6d6-b9a6-4d85-b594-9d947a9e6383 nodeName:}" failed. No retries permitted until 2026-10-31 01:03:40.564760175 +0000 UTC m=+32.250239593 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/9dfae6d6-b9a6-4d85-b594-9d947a9e6383-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-wm4qt" (UID: "9dfae6d6-b9a6-4d85-b594-9d947a9e6383") : secret "networking-console-plugin-cert" not found
Oct 31 01:03:40.065532 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:40.065153    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/9dfae6d6-b9a6-4d85-b594-9d947a9e6383-nginx-conf\") pod \"networking-console-plugin-86c478bb88-wm4qt\" (UID: \"9dfae6d6-b9a6-4d85-b594-9d947a9e6383\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-wm4qt"
Oct 31 01:03:40.065532 ip-10-0-133-197 kubenswrapper[2578]: E1031 01:03:40.065165    2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 31 01:03:40.065532 ip-10-0-133-197 kubenswrapper[2578]: E1031 01:03:40.065225    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/85bb69db-c69b-467e-adeb-950bf2947c94-metrics-tls podName:85bb69db-c69b-467e-adeb-950bf2947c94 nodeName:}" failed. No retries permitted until 2026-10-31 01:03:40.565210994 +0000 UTC m=+32.250690399 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/85bb69db-c69b-467e-adeb-950bf2947c94-metrics-tls") pod "dns-default-t7r6c" (UID: "85bb69db-c69b-467e-adeb-950bf2947c94") : secret "dns-default-metrics-tls" not found
Oct 31 01:03:40.065532 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:40.065275    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/85bb69db-c69b-467e-adeb-950bf2947c94-config-volume\") pod \"dns-default-t7r6c\" (UID: \"85bb69db-c69b-467e-adeb-950bf2947c94\") " pod="openshift-dns/dns-default-t7r6c"
Oct 31 01:03:40.065532 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:40.064656    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/9dfae6d6-b9a6-4d85-b594-9d947a9e6383-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-wm4qt\" (UID: \"9dfae6d6-b9a6-4d85-b594-9d947a9e6383\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-wm4qt"
Oct 31 01:03:40.065532 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:40.065479    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/c8e98df5-b2f7-49dd-a565-ed60534da496-serving-cert\") pod \"insights-operator-7c4986d449-jkk5x\" (UID: \"c8e98df5-b2f7-49dd-a565-ed60534da496\") " pod="openshift-insights/insights-operator-7c4986d449-jkk5x"
Oct 31 01:03:40.065532 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:40.065512    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ks6fn\" (UniqueName: \"kubernetes.io/projected/e54a9dc6-36e6-4f1b-9c3a-623a520e11da-kube-api-access-ks6fn\") pod \"service-ca-operator-544d749c8c-pdzj4\" (UID: \"e54a9dc6-36e6-4f1b-9c3a-623a520e11da\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-pdzj4"
Oct 31 01:03:40.065914 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:40.065538    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c8e98df5-b2f7-49dd-a565-ed60534da496-tmp\") pod \"insights-operator-7c4986d449-jkk5x\" (UID: \"c8e98df5-b2f7-49dd-a565-ed60534da496\") " pod="openshift-insights/insights-operator-7c4986d449-jkk5x"
Oct 31 01:03:40.065914 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:40.065550    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c8e98df5-b2f7-49dd-a565-ed60534da496-service-ca-bundle\") pod \"insights-operator-7c4986d449-jkk5x\" (UID: \"c8e98df5-b2f7-49dd-a565-ed60534da496\") " pod="openshift-insights/insights-operator-7c4986d449-jkk5x"
Oct 31 01:03:40.065914 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:40.065570    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-98n75\" (UniqueName: \"kubernetes.io/projected/e49453a8-c7d9-4248-a8ef-4a819b76806c-kube-api-access-98n75\") pod \"router-default-5964fff8f9-8k6pp\" (UID: \"e49453a8-c7d9-4248-a8ef-4a819b76806c\") " pod="openshift-ingress/router-default-5964fff8f9-8k6pp"
Oct 31 01:03:40.065914 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:40.065612    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e49453a8-c7d9-4248-a8ef-4a819b76806c-service-ca-bundle\") pod \"router-default-5964fff8f9-8k6pp\" (UID: \"e49453a8-c7d9-4248-a8ef-4a819b76806c\") " pod="openshift-ingress/router-default-5964fff8f9-8k6pp"
Oct 31 01:03:40.065914 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:40.065637    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c8e98df5-b2f7-49dd-a565-ed60534da496-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-jkk5x\" (UID: \"c8e98df5-b2f7-49dd-a565-ed60534da496\") " pod="openshift-insights/insights-operator-7c4986d449-jkk5x"
Oct 31 01:03:40.065914 ip-10-0-133-197 kubenswrapper[2578]: E1031 01:03:40.065642    2578 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Oct 31 01:03:40.065914 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:40.065662    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/b66619db-fd81-4435-be2a-a5ea6111e204-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-jccjk\" (UID: \"b66619db-fd81-4435-be2a-a5ea6111e204\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-jccjk"
Oct 31 01:03:40.065914 ip-10-0-133-197 kubenswrapper[2578]: E1031 01:03:40.065698    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e49453a8-c7d9-4248-a8ef-4a819b76806c-metrics-certs podName:e49453a8-c7d9-4248-a8ef-4a819b76806c nodeName:}" failed. No retries permitted until 2026-10-31 01:03:40.565683646 +0000 UTC m=+32.251163050 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e49453a8-c7d9-4248-a8ef-4a819b76806c-metrics-certs") pod "router-default-5964fff8f9-8k6pp" (UID: "e49453a8-c7d9-4248-a8ef-4a819b76806c") : secret "router-metrics-certs-default" not found
Oct 31 01:03:40.065914 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:40.065743    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/e49453a8-c7d9-4248-a8ef-4a819b76806c-stats-auth\") pod \"router-default-5964fff8f9-8k6pp\" (UID: \"e49453a8-c7d9-4248-a8ef-4a819b76806c\") " pod="openshift-ingress/router-default-5964fff8f9-8k6pp"
Oct 31 01:03:40.065914 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:40.065774    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8n8tt\" (UniqueName: \"kubernetes.io/projected/85bb69db-c69b-467e-adeb-950bf2947c94-kube-api-access-8n8tt\") pod \"dns-default-t7r6c\" (UID: \"85bb69db-c69b-467e-adeb-950bf2947c94\") " pod="openshift-dns/dns-default-t7r6c"
Oct 31 01:03:40.066435 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:40.066008    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/f273b9e5-0419-4587-a9c7-a0c8bec02acf-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-srcwl\" (UID: \"f273b9e5-0419-4587-a9c7-a0c8bec02acf\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-srcwl"
Oct 31 01:03:40.066489 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:40.066436    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/b66619db-fd81-4435-be2a-a5ea6111e204-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-jccjk\" (UID: \"b66619db-fd81-4435-be2a-a5ea6111e204\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-jccjk"
Oct 31 01:03:40.066489 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:40.066459    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/f0c0ada7-7688-4dc2-97fc-813e4d0c4ef3-config\") pod \"console-operator-7cd56b8dbc-9257n\" (UID: \"f0c0ada7-7688-4dc2-97fc-813e4d0c4ef3\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-9257n"
Oct 31 01:03:40.067496 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:40.067157    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f0c0ada7-7688-4dc2-97fc-813e4d0c4ef3-trusted-ca\") pod \"console-operator-7cd56b8dbc-9257n\" (UID: \"f0c0ada7-7688-4dc2-97fc-813e4d0c4ef3\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-9257n"
Oct 31 01:03:40.067496 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:40.067284    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/c2be658b-ec92-4782-a78b-d2dccf2249cb-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-n5q9v\" (UID: \"c2be658b-ec92-4782-a78b-d2dccf2249cb\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-n5q9v"
Oct 31 01:03:40.067496 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:40.067399    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c8e98df5-b2f7-49dd-a565-ed60534da496-tmp\") pod \"insights-operator-7c4986d449-jkk5x\" (UID: \"c8e98df5-b2f7-49dd-a565-ed60534da496\") " pod="openshift-insights/insights-operator-7c4986d449-jkk5x"
Oct 31 01:03:40.067496 ip-10-0-133-197 kubenswrapper[2578]: E1031 01:03:40.067445    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e49453a8-c7d9-4248-a8ef-4a819b76806c-service-ca-bundle podName:e49453a8-c7d9-4248-a8ef-4a819b76806c nodeName:}" failed. No retries permitted until 2026-10-31 01:03:40.567429481 +0000 UTC m=+32.252908894 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/e49453a8-c7d9-4248-a8ef-4a819b76806c-service-ca-bundle") pod "router-default-5964fff8f9-8k6pp" (UID: "e49453a8-c7d9-4248-a8ef-4a819b76806c") : configmap references non-existent config key: service-ca.crt
Oct 31 01:03:40.067743 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:40.067587    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/f0c0ada7-7688-4dc2-97fc-813e4d0c4ef3-serving-cert\") pod \"console-operator-7cd56b8dbc-9257n\" (UID: \"f0c0ada7-7688-4dc2-97fc-813e4d0c4ef3\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-9257n"
Oct 31 01:03:40.068200 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:40.068174    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c8e98df5-b2f7-49dd-a565-ed60534da496-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-jkk5x\" (UID: \"c8e98df5-b2f7-49dd-a565-ed60534da496\") " pod="openshift-insights/insights-operator-7c4986d449-jkk5x"
Oct 31 01:03:40.068720 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:40.068685    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/e49453a8-c7d9-4248-a8ef-4a819b76806c-default-certificate\") pod \"router-default-5964fff8f9-8k6pp\" (UID: \"e49453a8-c7d9-4248-a8ef-4a819b76806c\") " pod="openshift-ingress/router-default-5964fff8f9-8k6pp"
Oct 31 01:03:40.068835 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:40.068820    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/e54a9dc6-36e6-4f1b-9c3a-623a520e11da-serving-cert\") pod \"service-ca-operator-544d749c8c-pdzj4\" (UID: \"e54a9dc6-36e6-4f1b-9c3a-623a520e11da\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-pdzj4"
Oct 31 01:03:40.069131 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:40.069109    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/c8e98df5-b2f7-49dd-a565-ed60534da496-serving-cert\") pod \"insights-operator-7c4986d449-jkk5x\" (UID: \"c8e98df5-b2f7-49dd-a565-ed60534da496\") " pod="openshift-insights/insights-operator-7c4986d449-jkk5x"
Oct 31 01:03:40.069602 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:40.069579    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/e49453a8-c7d9-4248-a8ef-4a819b76806c-stats-auth\") pod \"router-default-5964fff8f9-8k6pp\" (UID: \"e49453a8-c7d9-4248-a8ef-4a819b76806c\") " pod="openshift-ingress/router-default-5964fff8f9-8k6pp"
Oct 31 01:03:40.069846 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:40.069830    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/147bb9af-dfc7-4785-bb5e-906f25d65dd1-cert\") pod \"ingress-canary-7dpwv\" (UID: \"147bb9af-dfc7-4785-bb5e-906f25d65dd1\") " pod="openshift-ingress-canary/ingress-canary-7dpwv"
Oct 31 01:03:40.087273 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:40.087232    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-q4z5f"
Oct 31 01:03:40.095333 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:40.095297    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-jm6bm"
Oct 31 01:03:40.124866 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:40.124393    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j78lj\" (UniqueName: \"kubernetes.io/projected/eafc69c5-15f1-4e4c-97ca-be586e55fbae-kube-api-access-j78lj\") pod \"volume-data-source-validator-58c9f5f46d-v585z\" (UID: \"eafc69c5-15f1-4e4c-97ca-be586e55fbae\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-v585z"
Oct 31 01:03:40.127350 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:40.127117    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bjgwm\" (UniqueName: \"kubernetes.io/projected/c2be658b-ec92-4782-a78b-d2dccf2249cb-kube-api-access-bjgwm\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-n5q9v\" (UID: \"c2be658b-ec92-4782-a78b-d2dccf2249cb\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-n5q9v"
Oct 31 01:03:40.130388 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:40.130360    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zlmlp\" (UniqueName: \"kubernetes.io/projected/f0c0ada7-7688-4dc2-97fc-813e4d0c4ef3-kube-api-access-zlmlp\") pod \"console-operator-7cd56b8dbc-9257n\" (UID: \"f0c0ada7-7688-4dc2-97fc-813e4d0c4ef3\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-9257n"
Oct 31 01:03:40.130954 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:40.130899    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8n8tt\" (UniqueName: \"kubernetes.io/projected/85bb69db-c69b-467e-adeb-950bf2947c94-kube-api-access-8n8tt\") pod \"dns-default-t7r6c\" (UID: \"85bb69db-c69b-467e-adeb-950bf2947c94\") " pod="openshift-dns/dns-default-t7r6c"
Oct 31 01:03:40.132065 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:40.131630    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lldxn\" (UniqueName: \"kubernetes.io/projected/f273b9e5-0419-4587-a9c7-a0c8bec02acf-kube-api-access-lldxn\") pod \"cluster-samples-operator-6ddb6c8b94-srcwl\" (UID: \"f273b9e5-0419-4587-a9c7-a0c8bec02acf\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-srcwl"
Oct 31 01:03:40.132065 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:40.131867    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-59kdl\" (UniqueName: \"kubernetes.io/projected/b66619db-fd81-4435-be2a-a5ea6111e204-kube-api-access-59kdl\") pod \"cluster-monitoring-operator-884576c57-jccjk\" (UID: \"b66619db-fd81-4435-be2a-a5ea6111e204\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-jccjk"
Oct 31 01:03:40.132065 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:40.132031    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ks6fn\" (UniqueName: \"kubernetes.io/projected/e54a9dc6-36e6-4f1b-9c3a-623a520e11da-kube-api-access-ks6fn\") pod \"service-ca-operator-544d749c8c-pdzj4\" (UID: \"e54a9dc6-36e6-4f1b-9c3a-623a520e11da\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-pdzj4"
Oct 31 01:03:40.133400 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:40.132968    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tlnlx\" (UniqueName: \"kubernetes.io/projected/c8e98df5-b2f7-49dd-a565-ed60534da496-kube-api-access-tlnlx\") pod \"insights-operator-7c4986d449-jkk5x\" (UID: \"c8e98df5-b2f7-49dd-a565-ed60534da496\") " pod="openshift-insights/insights-operator-7c4986d449-jkk5x"
Oct 31 01:03:40.133400 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:40.132982    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6qc9x\" (UniqueName: \"kubernetes.io/projected/69ff2b62-4588-4cb0-bdad-747c3692138c-kube-api-access-6qc9x\") pod \"network-check-source-6f78d59f99-bpmjc\" (UID: \"69ff2b62-4588-4cb0-bdad-747c3692138c\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-bpmjc"
Oct 31 01:03:40.133783 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:40.133740    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-98n75\" (UniqueName: \"kubernetes.io/projected/e49453a8-c7d9-4248-a8ef-4a819b76806c-kube-api-access-98n75\") pod \"router-default-5964fff8f9-8k6pp\" (UID: \"e49453a8-c7d9-4248-a8ef-4a819b76806c\") " pod="openshift-ingress/router-default-5964fff8f9-8k6pp"
Oct 31 01:03:40.134753 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:40.134722    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g9wls\" (UniqueName: \"kubernetes.io/projected/147bb9af-dfc7-4785-bb5e-906f25d65dd1-kube-api-access-g9wls\") pod \"ingress-canary-7dpwv\" (UID: \"147bb9af-dfc7-4785-bb5e-906f25d65dd1\") " pod="openshift-ingress-canary/ingress-canary-7dpwv"
Oct 31 01:03:40.151139 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:40.148954    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-7c4986d449-jkk5x"
Oct 31 01:03:40.157729 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:40.157677    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-pdzj4"
Oct 31 01:03:40.164072 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:40.163938    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-v585z"
Oct 31 01:03:40.171689 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:40.171547    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-n5q9v"
Oct 31 01:03:40.184269 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:40.184033    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7cd56b8dbc-9257n"
Oct 31 01:03:40.198510 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:40.198484    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-bpmjc"
Oct 31 01:03:40.214540 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:40.214510    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-7dpwv"
Oct 31 01:03:40.431460 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:40.431434    2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-vbbq6"]
Oct 31 01:03:40.434830 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:40.434803    2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda8395ef_7464_410e_96ee_b27378a03fed.slice/crio-357678e8fb63a8473b0f2c085a256547f8cd0e54439948bcf9e6493824d0ff36 WatchSource:0}: Error finding container 357678e8fb63a8473b0f2c085a256547f8cd0e54439948bcf9e6493824d0ff36: Status 404 returned error can't find the container with id 357678e8fb63a8473b0f2c085a256547f8cd0e54439948bcf9e6493824d0ff36
Oct 31 01:03:40.467015 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:40.466990    2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-q4z5f"]
Oct 31 01:03:40.470112 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:40.470086    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ff4dcba2-df8d-4324-9e0c-abb23a5e6982-registry-tls\") pod \"image-registry-56d8464d7c-vpwjc\" (UID: \"ff4dcba2-df8d-4324-9e0c-abb23a5e6982\") " pod="openshift-image-registry/image-registry-56d8464d7c-vpwjc"
Oct 31 01:03:40.470198 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:40.470127    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/aa112079-b749-404c-9081-113220707fcd-registry-tls\") pod \"image-registry-75575b5db8-wtq4d\" (UID: \"aa112079-b749-404c-9081-113220707fcd\") " pod="openshift-image-registry/image-registry-75575b5db8-wtq4d"
Oct 31 01:03:40.470262 ip-10-0-133-197 kubenswrapper[2578]: E1031 01:03:40.470239    2578 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 31 01:03:40.470262 ip-10-0-133-197 kubenswrapper[2578]: E1031 01:03:40.470249    2578 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-75575b5db8-wtq4d: secret "image-registry-tls" not found
Oct 31 01:03:40.470446 ip-10-0-133-197 kubenswrapper[2578]: E1031 01:03:40.470285    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/aa112079-b749-404c-9081-113220707fcd-registry-tls podName:aa112079-b749-404c-9081-113220707fcd nodeName:}" failed. No retries permitted until 2026-10-31 01:03:41.470274405 +0000 UTC m=+33.155753798 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/aa112079-b749-404c-9081-113220707fcd-registry-tls") pod "image-registry-75575b5db8-wtq4d" (UID: "aa112079-b749-404c-9081-113220707fcd") : secret "image-registry-tls" not found
Oct 31 01:03:40.470446 ip-10-0-133-197 kubenswrapper[2578]: E1031 01:03:40.470344    2578 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 31 01:03:40.470446 ip-10-0-133-197 kubenswrapper[2578]: E1031 01:03:40.470351    2578 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-56d8464d7c-vpwjc: secret "image-registry-tls" not found
Oct 31 01:03:40.470446 ip-10-0-133-197 kubenswrapper[2578]: E1031 01:03:40.470371    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ff4dcba2-df8d-4324-9e0c-abb23a5e6982-registry-tls podName:ff4dcba2-df8d-4324-9e0c-abb23a5e6982 nodeName:}" failed. No retries permitted until 2026-10-31 01:03:41.470364751 +0000 UTC m=+33.155844143 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/ff4dcba2-df8d-4324-9e0c-abb23a5e6982-registry-tls") pod "image-registry-56d8464d7c-vpwjc" (UID: "ff4dcba2-df8d-4324-9e0c-abb23a5e6982") : secret "image-registry-tls" not found
Oct 31 01:03:40.471248 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:40.471221    2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ab3954e_be64_4260_bca8_ff5d58bd6ea2.slice/crio-075a77de8990101732600f618149824f4c9470f0e266030d9b437c8c27451e87 WatchSource:0}: Error finding container 075a77de8990101732600f618149824f4c9470f0e266030d9b437c8c27451e87: Status 404 returned error can't find the container with id 075a77de8990101732600f618149824f4c9470f0e266030d9b437c8c27451e87
Oct 31 01:03:40.485057 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:40.484987    2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-jm6bm"]
Oct 31 01:03:40.489574 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:40.489511    2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd719eda9_bc9f_4307_952a_99d99aa0f62b.slice/crio-9b0ccc77fd551612f3962882d98c4ffc275de65a04563263a46cb4830ae83020 WatchSource:0}: Error finding container 9b0ccc77fd551612f3962882d98c4ffc275de65a04563263a46cb4830ae83020: Status 404 returned error can't find the container with id 9b0ccc77fd551612f3962882d98c4ffc275de65a04563263a46cb4830ae83020
Oct 31 01:03:40.556528 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:40.555216    2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-v585z"]
Oct 31 01:03:40.558487 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:40.558399    2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-7c4986d449-jkk5x"]
Oct 31 01:03:40.559746 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:40.559722    2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-pdzj4"]
Oct 31 01:03:40.568212 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:40.568034    2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeafc69c5_15f1_4e4c_97ca_be586e55fbae.slice/crio-a146455d1f30eb2f7f4ad58212035a0c1f733cb997565d267edda0121121fa40 WatchSource:0}: Error finding container a146455d1f30eb2f7f4ad58212035a0c1f733cb997565d267edda0121121fa40: Status 404 returned error can't find the container with id a146455d1f30eb2f7f4ad58212035a0c1f733cb997565d267edda0121121fa40
Oct 31 01:03:40.569588 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:40.569562    2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc8e98df5_b2f7_49dd_a565_ed60534da496.slice/crio-7e90fa928e74f963359d215e04ec3bd27f51c1496663feae053c18839e060ba3 WatchSource:0}: Error finding container 7e90fa928e74f963359d215e04ec3bd27f51c1496663feae053c18839e060ba3: Status 404 returned error can't find the container with id 7e90fa928e74f963359d215e04ec3bd27f51c1496663feae053c18839e060ba3
Oct 31 01:03:40.570145 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:40.570123    2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode54a9dc6_36e6_4f1b_9c3a_623a520e11da.slice/crio-5073165fba6902e2ceb840e7ba07c727561dac833681c45f203b6cf37a9c42bd WatchSource:0}: Error finding container 5073165fba6902e2ceb840e7ba07c727561dac833681c45f203b6cf37a9c42bd: Status 404 returned error can't find the container with id 5073165fba6902e2ceb840e7ba07c727561dac833681c45f203b6cf37a9c42bd
Oct 31 01:03:40.570545 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:40.570476    2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-n5q9v"]
Oct 31 01:03:40.571384 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:40.570703    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e49453a8-c7d9-4248-a8ef-4a819b76806c-metrics-certs\") pod \"router-default-5964fff8f9-8k6pp\" (UID: \"e49453a8-c7d9-4248-a8ef-4a819b76806c\") " pod="openshift-ingress/router-default-5964fff8f9-8k6pp"
Oct 31 01:03:40.571384 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:40.570787    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/b66619db-fd81-4435-be2a-a5ea6111e204-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-jccjk\" (UID: \"b66619db-fd81-4435-be2a-a5ea6111e204\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-jccjk"
Oct 31 01:03:40.571384 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:40.570840    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/85bb69db-c69b-467e-adeb-950bf2947c94-metrics-tls\") pod \"dns-default-t7r6c\" (UID: \"85bb69db-c69b-467e-adeb-950bf2947c94\") " pod="openshift-dns/dns-default-t7r6c"
Oct 31 01:03:40.571384 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:40.570888    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/9dfae6d6-b9a6-4d85-b594-9d947a9e6383-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-wm4qt\" (UID: \"9dfae6d6-b9a6-4d85-b594-9d947a9e6383\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-wm4qt"
Oct 31 01:03:40.571384 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:40.570931    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e49453a8-c7d9-4248-a8ef-4a819b76806c-service-ca-bundle\") pod \"router-default-5964fff8f9-8k6pp\" (UID: \"e49453a8-c7d9-4248-a8ef-4a819b76806c\") " pod="openshift-ingress/router-default-5964fff8f9-8k6pp"
Oct 31 01:03:40.571384 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:40.570997    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/f273b9e5-0419-4587-a9c7-a0c8bec02acf-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-srcwl\" (UID: \"f273b9e5-0419-4587-a9c7-a0c8bec02acf\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-srcwl"
Oct 31 01:03:40.571384 ip-10-0-133-197 kubenswrapper[2578]: E1031 01:03:40.571148    2578 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Oct 31 01:03:40.571384 ip-10-0-133-197 kubenswrapper[2578]: E1031 01:03:40.571197    2578 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Oct 31 01:03:40.571384 ip-10-0-133-197 kubenswrapper[2578]: E1031 01:03:40.571261    2578 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Oct 31 01:03:40.571384 ip-10-0-133-197 kubenswrapper[2578]: E1031 01:03:40.571275    2578 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Oct 31 01:03:40.571384 ip-10-0-133-197 kubenswrapper[2578]: E1031 01:03:40.571203    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f273b9e5-0419-4587-a9c7-a0c8bec02acf-samples-operator-tls podName:f273b9e5-0419-4587-a9c7-a0c8bec02acf nodeName:}" failed. No retries permitted until 2026-10-31 01:03:41.571184571 +0000 UTC m=+33.256663979 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/f273b9e5-0419-4587-a9c7-a0c8bec02acf-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-srcwl" (UID: "f273b9e5-0419-4587-a9c7-a0c8bec02acf") : secret "samples-operator-tls" not found
Oct 31 01:03:40.571384 ip-10-0-133-197 kubenswrapper[2578]: E1031 01:03:40.571350    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e49453a8-c7d9-4248-a8ef-4a819b76806c-metrics-certs podName:e49453a8-c7d9-4248-a8ef-4a819b76806c nodeName:}" failed. No retries permitted until 2026-10-31 01:03:41.571306609 +0000 UTC m=+33.256786006 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e49453a8-c7d9-4248-a8ef-4a819b76806c-metrics-certs") pod "router-default-5964fff8f9-8k6pp" (UID: "e49453a8-c7d9-4248-a8ef-4a819b76806c") : secret "router-metrics-certs-default" not found
Oct 31 01:03:40.571384 ip-10-0-133-197 kubenswrapper[2578]: E1031 01:03:40.571152    2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 31 01:03:40.571384 ip-10-0-133-197 kubenswrapper[2578]: E1031 01:03:40.571366    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9dfae6d6-b9a6-4d85-b594-9d947a9e6383-networking-console-plugin-cert podName:9dfae6d6-b9a6-4d85-b594-9d947a9e6383 nodeName:}" failed. No retries permitted until 2026-10-31 01:03:41.571357514 +0000 UTC m=+33.256836911 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/9dfae6d6-b9a6-4d85-b594-9d947a9e6383-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-wm4qt" (UID: "9dfae6d6-b9a6-4d85-b594-9d947a9e6383") : secret "networking-console-plugin-cert" not found
Oct 31 01:03:40.571384 ip-10-0-133-197 kubenswrapper[2578]: E1031 01:03:40.571385    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/85bb69db-c69b-467e-adeb-950bf2947c94-metrics-tls podName:85bb69db-c69b-467e-adeb-950bf2947c94 nodeName:}" failed. No retries permitted until 2026-10-31 01:03:41.571374401 +0000 UTC m=+33.256853834 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/85bb69db-c69b-467e-adeb-950bf2947c94-metrics-tls") pod "dns-default-t7r6c" (UID: "85bb69db-c69b-467e-adeb-950bf2947c94") : secret "dns-default-metrics-tls" not found
Oct 31 01:03:40.572108 ip-10-0-133-197 kubenswrapper[2578]: E1031 01:03:40.571401    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b66619db-fd81-4435-be2a-a5ea6111e204-cluster-monitoring-operator-tls podName:b66619db-fd81-4435-be2a-a5ea6111e204 nodeName:}" failed. No retries permitted until 2026-10-31 01:03:41.571392564 +0000 UTC m=+33.256871955 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/b66619db-fd81-4435-be2a-a5ea6111e204-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-jccjk" (UID: "b66619db-fd81-4435-be2a-a5ea6111e204") : secret "cluster-monitoring-operator-tls" not found
Oct 31 01:03:40.572108 ip-10-0-133-197 kubenswrapper[2578]: E1031 01:03:40.571417    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e49453a8-c7d9-4248-a8ef-4a819b76806c-service-ca-bundle podName:e49453a8-c7d9-4248-a8ef-4a819b76806c nodeName:}" failed. No retries permitted until 2026-10-31 01:03:41.571408593 +0000 UTC m=+33.256887984 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/e49453a8-c7d9-4248-a8ef-4a819b76806c-service-ca-bundle") pod "router-default-5964fff8f9-8k6pp" (UID: "e49453a8-c7d9-4248-a8ef-4a819b76806c") : configmap references non-existent config key: service-ca.crt
Oct 31 01:03:40.576708 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:40.576407    2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc2be658b_ec92_4782_a78b_d2dccf2249cb.slice/crio-073fc7fbb6f5887bd5ad9f67b0ea840f764095bda44fa96f2e5959f1d52f73d5 WatchSource:0}: Error finding container 073fc7fbb6f5887bd5ad9f67b0ea840f764095bda44fa96f2e5959f1d52f73d5: Status 404 returned error can't find the container with id 073fc7fbb6f5887bd5ad9f67b0ea840f764095bda44fa96f2e5959f1d52f73d5
Oct 31 01:03:40.582393 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:40.582369    2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-9257n"]
Oct 31 01:03:40.587257 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:40.587238    2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf0c0ada7_7688_4dc2_97fc_813e4d0c4ef3.slice/crio-e459951077da3523c5cc454f50947afe46c379d6e8c40321e83519e9f3c10d61 WatchSource:0}: Error finding container e459951077da3523c5cc454f50947afe46c379d6e8c40321e83519e9f3c10d61: Status 404 returned error can't find the container with id e459951077da3523c5cc454f50947afe46c379d6e8c40321e83519e9f3c10d61
Oct 31 01:03:40.618975 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:40.618949    2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-bpmjc"]
Oct 31 01:03:40.622264 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:40.622241    2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod69ff2b62_4588_4cb0_bdad_747c3692138c.slice/crio-3497d23ca426dcc552e2336dd5013e9a8e2a9195710bceeb655f082d4aef724d WatchSource:0}: Error finding container 3497d23ca426dcc552e2336dd5013e9a8e2a9195710bceeb655f082d4aef724d: Status 404 returned error can't find the container with id 3497d23ca426dcc552e2336dd5013e9a8e2a9195710bceeb655f082d4aef724d
Oct 31 01:03:40.622475 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:40.622454    2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-7dpwv"]
Oct 31 01:03:40.624977 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:40.624961    2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod147bb9af_dfc7_4785_bb5e_906f25d65dd1.slice/crio-12bf89ab2c442ecd84c7a7607d703dd213b5513e6a93c92ef2dde5bf06ad6f99 WatchSource:0}: Error finding container 12bf89ab2c442ecd84c7a7607d703dd213b5513e6a93c92ef2dde5bf06ad6f99: Status 404 returned error can't find the container with id 12bf89ab2c442ecd84c7a7607d703dd213b5513e6a93c92ef2dde5bf06ad6f99
Oct 31 01:03:41.021100 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:41.021062    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-v585z" event={"ID":"eafc69c5-15f1-4e4c-97ca-be586e55fbae","Type":"ContainerStarted","Data":"a146455d1f30eb2f7f4ad58212035a0c1f733cb997565d267edda0121121fa40"}
Oct 31 01:03:41.022512 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:41.022480    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-q4z5f" event={"ID":"0ab3954e-be64-4260-bca8-ff5d58bd6ea2","Type":"ContainerStarted","Data":"1e0ba8a225bf88cbb6609f0eb908def00ac8131cf30afe12c221f736292be0ea"}
Oct 31 01:03:41.022651 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:41.022533    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-q4z5f" event={"ID":"0ab3954e-be64-4260-bca8-ff5d58bd6ea2","Type":"ContainerStarted","Data":"075a77de8990101732600f618149824f4c9470f0e266030d9b437c8c27451e87"}
Oct 31 01:03:41.023535 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:41.023513    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-9257n" event={"ID":"f0c0ada7-7688-4dc2-97fc-813e4d0c4ef3","Type":"ContainerStarted","Data":"e459951077da3523c5cc454f50947afe46c379d6e8c40321e83519e9f3c10d61"}
Oct 31 01:03:41.024573 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:41.024550    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-pdzj4" event={"ID":"e54a9dc6-36e6-4f1b-9c3a-623a520e11da","Type":"ContainerStarted","Data":"5073165fba6902e2ceb840e7ba07c727561dac833681c45f203b6cf37a9c42bd"}
Oct 31 01:03:41.025557 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:41.025539    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-jkk5x" event={"ID":"c8e98df5-b2f7-49dd-a565-ed60534da496","Type":"ContainerStarted","Data":"7e90fa928e74f963359d215e04ec3bd27f51c1496663feae053c18839e060ba3"}
Oct 31 01:03:41.029083 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:41.029064    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-n5q9v" event={"ID":"c2be658b-ec92-4782-a78b-d2dccf2249cb","Type":"ContainerStarted","Data":"073fc7fbb6f5887bd5ad9f67b0ea840f764095bda44fa96f2e5959f1d52f73d5"}
Oct 31 01:03:41.030627 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:41.030605    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-vbbq6" event={"ID":"da8395ef-7464-410e-96ee-b27378a03fed","Type":"ContainerStarted","Data":"8a499c24ec397f0116dfb2cf6ffbe9234715ffa5dbbbf7778f5c92b3a72cc715"}
Oct 31 01:03:41.030714 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:41.030640    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-vbbq6" event={"ID":"da8395ef-7464-410e-96ee-b27378a03fed","Type":"ContainerStarted","Data":"357678e8fb63a8473b0f2c085a256547f8cd0e54439948bcf9e6493824d0ff36"}
Oct 31 01:03:41.031580 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:41.031560    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-bpmjc" event={"ID":"69ff2b62-4588-4cb0-bdad-747c3692138c","Type":"ContainerStarted","Data":"3497d23ca426dcc552e2336dd5013e9a8e2a9195710bceeb655f082d4aef724d"}
Oct 31 01:03:41.032807 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:41.032785    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-jm6bm" event={"ID":"d719eda9-bc9f-4307-952a-99d99aa0f62b","Type":"ContainerStarted","Data":"2b4e04eae0d22eef99aeda2025b8247d9ec0536d125ea6ec58d9a61aa1c40c83"}
Oct 31 01:03:41.032895 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:41.032820    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-jm6bm" event={"ID":"d719eda9-bc9f-4307-952a-99d99aa0f62b","Type":"ContainerStarted","Data":"9b0ccc77fd551612f3962882d98c4ffc275de65a04563263a46cb4830ae83020"}
Oct 31 01:03:41.033847 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:41.033830    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-7dpwv" event={"ID":"147bb9af-dfc7-4785-bb5e-906f25d65dd1","Type":"ContainerStarted","Data":"12bf89ab2c442ecd84c7a7607d703dd213b5513e6a93c92ef2dde5bf06ad6f99"}
Oct 31 01:03:41.479464 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:41.479394    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ff4dcba2-df8d-4324-9e0c-abb23a5e6982-registry-tls\") pod \"image-registry-56d8464d7c-vpwjc\" (UID: \"ff4dcba2-df8d-4324-9e0c-abb23a5e6982\") " pod="openshift-image-registry/image-registry-56d8464d7c-vpwjc"
Oct 31 01:03:41.479464 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:41.479437    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/aa112079-b749-404c-9081-113220707fcd-registry-tls\") pod \"image-registry-75575b5db8-wtq4d\" (UID: \"aa112079-b749-404c-9081-113220707fcd\") " pod="openshift-image-registry/image-registry-75575b5db8-wtq4d"
Oct 31 01:03:41.479613 ip-10-0-133-197 kubenswrapper[2578]: E1031 01:03:41.479541    2578 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 31 01:03:41.479613 ip-10-0-133-197 kubenswrapper[2578]: E1031 01:03:41.479560    2578 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-56d8464d7c-vpwjc: secret "image-registry-tls" not found
Oct 31 01:03:41.479613 ip-10-0-133-197 kubenswrapper[2578]: E1031 01:03:41.479597    2578 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 31 01:03:41.479613 ip-10-0-133-197 kubenswrapper[2578]: E1031 01:03:41.479608    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ff4dcba2-df8d-4324-9e0c-abb23a5e6982-registry-tls podName:ff4dcba2-df8d-4324-9e0c-abb23a5e6982 nodeName:}" failed. No retries permitted until 2026-10-31 01:03:43.479594622 +0000 UTC m=+35.165074013 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/ff4dcba2-df8d-4324-9e0c-abb23a5e6982-registry-tls") pod "image-registry-56d8464d7c-vpwjc" (UID: "ff4dcba2-df8d-4324-9e0c-abb23a5e6982") : secret "image-registry-tls" not found
Oct 31 01:03:41.479613 ip-10-0-133-197 kubenswrapper[2578]: E1031 01:03:41.479612    2578 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-75575b5db8-wtq4d: secret "image-registry-tls" not found
Oct 31 01:03:41.479758 ip-10-0-133-197 kubenswrapper[2578]: E1031 01:03:41.479647    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/aa112079-b749-404c-9081-113220707fcd-registry-tls podName:aa112079-b749-404c-9081-113220707fcd nodeName:}" failed. No retries permitted until 2026-10-31 01:03:43.479637626 +0000 UTC m=+35.165117017 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/aa112079-b749-404c-9081-113220707fcd-registry-tls") pod "image-registry-75575b5db8-wtq4d" (UID: "aa112079-b749-404c-9081-113220707fcd") : secret "image-registry-tls" not found
Oct 31 01:03:41.580032 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:41.579997    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e49453a8-c7d9-4248-a8ef-4a819b76806c-metrics-certs\") pod \"router-default-5964fff8f9-8k6pp\" (UID: \"e49453a8-c7d9-4248-a8ef-4a819b76806c\") " pod="openshift-ingress/router-default-5964fff8f9-8k6pp"
Oct 31 01:03:41.580200 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:41.580083    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/b66619db-fd81-4435-be2a-a5ea6111e204-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-jccjk\" (UID: \"b66619db-fd81-4435-be2a-a5ea6111e204\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-jccjk"
Oct 31 01:03:41.580200 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:41.580130    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/85bb69db-c69b-467e-adeb-950bf2947c94-metrics-tls\") pod \"dns-default-t7r6c\" (UID: \"85bb69db-c69b-467e-adeb-950bf2947c94\") " pod="openshift-dns/dns-default-t7r6c"
Oct 31 01:03:41.580200 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:41.580172    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/9dfae6d6-b9a6-4d85-b594-9d947a9e6383-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-wm4qt\" (UID: \"9dfae6d6-b9a6-4d85-b594-9d947a9e6383\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-wm4qt"
Oct 31 01:03:41.580381 ip-10-0-133-197 kubenswrapper[2578]: E1031 01:03:41.580366    2578 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Oct 31 01:03:41.580436 ip-10-0-133-197 kubenswrapper[2578]: E1031 01:03:41.580417    2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 31 01:03:41.580436 ip-10-0-133-197 kubenswrapper[2578]: E1031 01:03:41.580425    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b66619db-fd81-4435-be2a-a5ea6111e204-cluster-monitoring-operator-tls podName:b66619db-fd81-4435-be2a-a5ea6111e204 nodeName:}" failed. No retries permitted until 2026-10-31 01:03:43.580410633 +0000 UTC m=+35.265890036 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/b66619db-fd81-4435-be2a-a5ea6111e204-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-jccjk" (UID: "b66619db-fd81-4435-be2a-a5ea6111e204") : secret "cluster-monitoring-operator-tls" not found
Oct 31 01:03:41.580533 ip-10-0-133-197 kubenswrapper[2578]: E1031 01:03:41.580473    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/85bb69db-c69b-467e-adeb-950bf2947c94-metrics-tls podName:85bb69db-c69b-467e-adeb-950bf2947c94 nodeName:}" failed. No retries permitted until 2026-10-31 01:03:43.580455784 +0000 UTC m=+35.265935189 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/85bb69db-c69b-467e-adeb-950bf2947c94-metrics-tls") pod "dns-default-t7r6c" (UID: "85bb69db-c69b-467e-adeb-950bf2947c94") : secret "dns-default-metrics-tls" not found
Oct 31 01:03:41.580533 ip-10-0-133-197 kubenswrapper[2578]: E1031 01:03:41.580478    2578 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Oct 31 01:03:41.580533 ip-10-0-133-197 kubenswrapper[2578]: E1031 01:03:41.580502    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e49453a8-c7d9-4248-a8ef-4a819b76806c-metrics-certs podName:e49453a8-c7d9-4248-a8ef-4a819b76806c nodeName:}" failed. No retries permitted until 2026-10-31 01:03:43.580495135 +0000 UTC m=+35.265974558 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e49453a8-c7d9-4248-a8ef-4a819b76806c-metrics-certs") pod "router-default-5964fff8f9-8k6pp" (UID: "e49453a8-c7d9-4248-a8ef-4a819b76806c") : secret "router-metrics-certs-default" not found
Oct 31 01:03:41.580685 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:41.580544    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e49453a8-c7d9-4248-a8ef-4a819b76806c-service-ca-bundle\") pod \"router-default-5964fff8f9-8k6pp\" (UID: \"e49453a8-c7d9-4248-a8ef-4a819b76806c\") " pod="openshift-ingress/router-default-5964fff8f9-8k6pp"
Oct 31 01:03:41.580685 ip-10-0-133-197 kubenswrapper[2578]: E1031 01:03:41.580599    2578 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Oct 31 01:03:41.580685 ip-10-0-133-197 kubenswrapper[2578]: E1031 01:03:41.580603    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e49453a8-c7d9-4248-a8ef-4a819b76806c-service-ca-bundle podName:e49453a8-c7d9-4248-a8ef-4a819b76806c nodeName:}" failed. No retries permitted until 2026-10-31 01:03:43.580597225 +0000 UTC m=+35.266076647 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/e49453a8-c7d9-4248-a8ef-4a819b76806c-service-ca-bundle") pod "router-default-5964fff8f9-8k6pp" (UID: "e49453a8-c7d9-4248-a8ef-4a819b76806c") : configmap references non-existent config key: service-ca.crt
Oct 31 01:03:41.580829 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:41.580693    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/f273b9e5-0419-4587-a9c7-a0c8bec02acf-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-srcwl\" (UID: \"f273b9e5-0419-4587-a9c7-a0c8bec02acf\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-srcwl"
Oct 31 01:03:41.580829 ip-10-0-133-197 kubenswrapper[2578]: E1031 01:03:41.580758    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9dfae6d6-b9a6-4d85-b594-9d947a9e6383-networking-console-plugin-cert podName:9dfae6d6-b9a6-4d85-b594-9d947a9e6383 nodeName:}" failed. No retries permitted until 2026-10-31 01:03:43.580743909 +0000 UTC m=+35.266223304 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/9dfae6d6-b9a6-4d85-b594-9d947a9e6383-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-wm4qt" (UID: "9dfae6d6-b9a6-4d85-b594-9d947a9e6383") : secret "networking-console-plugin-cert" not found
Oct 31 01:03:41.580829 ip-10-0-133-197 kubenswrapper[2578]: E1031 01:03:41.580808    2578 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Oct 31 01:03:41.580967 ip-10-0-133-197 kubenswrapper[2578]: E1031 01:03:41.580846    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f273b9e5-0419-4587-a9c7-a0c8bec02acf-samples-operator-tls podName:f273b9e5-0419-4587-a9c7-a0c8bec02acf nodeName:}" failed. No retries permitted until 2026-10-31 01:03:43.580833481 +0000 UTC m=+35.266312893 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/f273b9e5-0419-4587-a9c7-a0c8bec02acf-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-srcwl" (UID: "f273b9e5-0419-4587-a9c7-a0c8bec02acf") : secret "samples-operator-tls" not found
Oct 31 01:03:41.931616 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:41.931510    2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-jm6bm" podStartSLOduration=415.93149128 podStartE2EDuration="6m55.93149128s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:56:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:03:41.930346589 +0000 UTC m=+33.615826008" watchObservedRunningTime="2026-10-31 01:03:41.93149128 +0000 UTC m=+33.616970690"
Oct 31 01:03:42.047165 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:42.047122    2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-vbbq6" podStartSLOduration=416.047108184 podStartE2EDuration="6m56.047108184s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:56:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:03:42.045592335 +0000 UTC m=+33.731071752" watchObservedRunningTime="2026-10-31 01:03:42.047108184 +0000 UTC m=+33.732587597"
Oct 31 01:03:42.162664 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:42.162607    2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-q4z5f" podStartSLOduration=416.162587602 podStartE2EDuration="6m56.162587602s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:56:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:03:42.161523129 +0000 UTC m=+33.847002543" watchObservedRunningTime="2026-10-31 01:03:42.162587602 +0000 UTC m=+33.848067034"
Oct 31 01:03:42.592444 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:42.592407    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/aa4836a5-c71a-485d-82fc-f8b53dae7717-original-pull-secret\") pod \"global-pull-secret-syncer-zs5g7\" (UID: \"aa4836a5-c71a-485d-82fc-f8b53dae7717\") " pod="kube-system/global-pull-secret-syncer-zs5g7"
Oct 31 01:03:42.609038 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:42.608960    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/aa4836a5-c71a-485d-82fc-f8b53dae7717-original-pull-secret\") pod \"global-pull-secret-syncer-zs5g7\" (UID: \"aa4836a5-c71a-485d-82fc-f8b53dae7717\") " pod="kube-system/global-pull-secret-syncer-zs5g7"
Oct 31 01:03:42.693784 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:42.693750    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r6zx9\" (UniqueName: \"kubernetes.io/projected/565fefaf-82aa-42a8-9cc7-6517c4bddf1e-kube-api-access-r6zx9\") pod \"network-check-target-8mqbd\" (UID: \"565fefaf-82aa-42a8-9cc7-6517c4bddf1e\") " pod="openshift-network-diagnostics/network-check-target-8mqbd"
Oct 31 01:03:42.693937 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:42.693818    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8952fc79-c5a6-42ba-a296-8cb7a776f718-metrics-certs\") pod \"network-metrics-daemon-wdsmn\" (UID: \"8952fc79-c5a6-42ba-a296-8cb7a776f718\") " pod="openshift-multus/network-metrics-daemon-wdsmn"
Oct 31 01:03:42.694019 ip-10-0-133-197 kubenswrapper[2578]: E1031 01:03:42.694005    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Oct 31 01:03:42.694075 ip-10-0-133-197 kubenswrapper[2578]: E1031 01:03:42.694067    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8952fc79-c5a6-42ba-a296-8cb7a776f718-metrics-certs podName:8952fc79-c5a6-42ba-a296-8cb7a776f718 nodeName:}" failed. No retries permitted until 2026-10-31 01:04:14.694049853 +0000 UTC m=+66.379529262 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8952fc79-c5a6-42ba-a296-8cb7a776f718-metrics-certs") pod "network-metrics-daemon-wdsmn" (UID: "8952fc79-c5a6-42ba-a296-8cb7a776f718") : secret "metrics-daemon-secret" not found
Oct 31 01:03:42.700776 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:42.700749    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r6zx9\" (UniqueName: \"kubernetes.io/projected/565fefaf-82aa-42a8-9cc7-6517c4bddf1e-kube-api-access-r6zx9\") pod \"network-check-target-8mqbd\" (UID: \"565fefaf-82aa-42a8-9cc7-6517c4bddf1e\") " pod="openshift-network-diagnostics/network-check-target-8mqbd"
Oct 31 01:03:42.803456 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:42.803429    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8mqbd"
Oct 31 01:03:42.814390 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:42.814349    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zs5g7"
Oct 31 01:03:43.218870 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:43.218506    2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-8mqbd"]
Oct 31 01:03:43.226571 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:43.226546    2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod565fefaf_82aa_42a8_9cc7_6517c4bddf1e.slice/crio-e7be247cfa0d4a3703d5d9670caf05d05e034c7c4142a470ca2f537b939bc2b9 WatchSource:0}: Error finding container e7be247cfa0d4a3703d5d9670caf05d05e034c7c4142a470ca2f537b939bc2b9: Status 404 returned error can't find the container with id e7be247cfa0d4a3703d5d9670caf05d05e034c7c4142a470ca2f537b939bc2b9
Oct 31 01:03:43.267675 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:43.267649    2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-zs5g7"]
Oct 31 01:03:43.273668 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:43.273631    2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa4836a5_c71a_485d_82fc_f8b53dae7717.slice/crio-bba607643ebf9639b0c8044c3e3694561fdacdd3731bcf15feae4311301c9f2b WatchSource:0}: Error finding container bba607643ebf9639b0c8044c3e3694561fdacdd3731bcf15feae4311301c9f2b: Status 404 returned error can't find the container with id bba607643ebf9639b0c8044c3e3694561fdacdd3731bcf15feae4311301c9f2b
Oct 31 01:03:43.509824 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:43.508946    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ff4dcba2-df8d-4324-9e0c-abb23a5e6982-registry-tls\") pod \"image-registry-56d8464d7c-vpwjc\" (UID: \"ff4dcba2-df8d-4324-9e0c-abb23a5e6982\") " pod="openshift-image-registry/image-registry-56d8464d7c-vpwjc"
Oct 31 01:03:43.509824 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:43.509008    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/aa112079-b749-404c-9081-113220707fcd-registry-tls\") pod \"image-registry-75575b5db8-wtq4d\" (UID: \"aa112079-b749-404c-9081-113220707fcd\") " pod="openshift-image-registry/image-registry-75575b5db8-wtq4d"
Oct 31 01:03:43.509824 ip-10-0-133-197 kubenswrapper[2578]: E1031 01:03:43.509204    2578 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 31 01:03:43.509824 ip-10-0-133-197 kubenswrapper[2578]: E1031 01:03:43.509221    2578 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-75575b5db8-wtq4d: secret "image-registry-tls" not found
Oct 31 01:03:43.509824 ip-10-0-133-197 kubenswrapper[2578]: E1031 01:03:43.509287    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/aa112079-b749-404c-9081-113220707fcd-registry-tls podName:aa112079-b749-404c-9081-113220707fcd nodeName:}" failed. No retries permitted until 2026-10-31 01:03:47.509263044 +0000 UTC m=+39.194742439 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/aa112079-b749-404c-9081-113220707fcd-registry-tls") pod "image-registry-75575b5db8-wtq4d" (UID: "aa112079-b749-404c-9081-113220707fcd") : secret "image-registry-tls" not found
Oct 31 01:03:43.509824 ip-10-0-133-197 kubenswrapper[2578]: E1031 01:03:43.509715    2578 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 31 01:03:43.509824 ip-10-0-133-197 kubenswrapper[2578]: E1031 01:03:43.509730    2578 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-56d8464d7c-vpwjc: secret "image-registry-tls" not found
Oct 31 01:03:43.509824 ip-10-0-133-197 kubenswrapper[2578]: E1031 01:03:43.509773    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ff4dcba2-df8d-4324-9e0c-abb23a5e6982-registry-tls podName:ff4dcba2-df8d-4324-9e0c-abb23a5e6982 nodeName:}" failed. No retries permitted until 2026-10-31 01:03:47.509759529 +0000 UTC m=+39.195238923 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/ff4dcba2-df8d-4324-9e0c-abb23a5e6982-registry-tls") pod "image-registry-56d8464d7c-vpwjc" (UID: "ff4dcba2-df8d-4324-9e0c-abb23a5e6982") : secret "image-registry-tls" not found
Oct 31 01:03:43.610729 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:43.609747    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e49453a8-c7d9-4248-a8ef-4a819b76806c-metrics-certs\") pod \"router-default-5964fff8f9-8k6pp\" (UID: \"e49453a8-c7d9-4248-a8ef-4a819b76806c\") " pod="openshift-ingress/router-default-5964fff8f9-8k6pp"
Oct 31 01:03:43.610729 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:43.609819    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/b66619db-fd81-4435-be2a-a5ea6111e204-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-jccjk\" (UID: \"b66619db-fd81-4435-be2a-a5ea6111e204\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-jccjk"
Oct 31 01:03:43.610729 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:43.609873    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/85bb69db-c69b-467e-adeb-950bf2947c94-metrics-tls\") pod \"dns-default-t7r6c\" (UID: \"85bb69db-c69b-467e-adeb-950bf2947c94\") " pod="openshift-dns/dns-default-t7r6c"
Oct 31 01:03:43.610729 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:43.609946    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/9dfae6d6-b9a6-4d85-b594-9d947a9e6383-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-wm4qt\" (UID: \"9dfae6d6-b9a6-4d85-b594-9d947a9e6383\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-wm4qt"
Oct 31 01:03:43.610729 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:43.609988    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e49453a8-c7d9-4248-a8ef-4a819b76806c-service-ca-bundle\") pod \"router-default-5964fff8f9-8k6pp\" (UID: \"e49453a8-c7d9-4248-a8ef-4a819b76806c\") " pod="openshift-ingress/router-default-5964fff8f9-8k6pp"
Oct 31 01:03:43.610729 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:43.610049    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/f273b9e5-0419-4587-a9c7-a0c8bec02acf-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-srcwl\" (UID: \"f273b9e5-0419-4587-a9c7-a0c8bec02acf\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-srcwl"
Oct 31 01:03:43.610729 ip-10-0-133-197 kubenswrapper[2578]: E1031 01:03:43.610175    2578 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Oct 31 01:03:43.610729 ip-10-0-133-197 kubenswrapper[2578]: E1031 01:03:43.610232    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f273b9e5-0419-4587-a9c7-a0c8bec02acf-samples-operator-tls podName:f273b9e5-0419-4587-a9c7-a0c8bec02acf nodeName:}" failed. No retries permitted until 2026-10-31 01:03:47.610215323 +0000 UTC m=+39.295694729 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/f273b9e5-0419-4587-a9c7-a0c8bec02acf-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-srcwl" (UID: "f273b9e5-0419-4587-a9c7-a0c8bec02acf") : secret "samples-operator-tls" not found
Oct 31 01:03:43.610729 ip-10-0-133-197 kubenswrapper[2578]: E1031 01:03:43.610292    2578 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Oct 31 01:03:43.610729 ip-10-0-133-197 kubenswrapper[2578]: E1031 01:03:43.610343    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e49453a8-c7d9-4248-a8ef-4a819b76806c-metrics-certs podName:e49453a8-c7d9-4248-a8ef-4a819b76806c nodeName:}" failed. No retries permitted until 2026-10-31 01:03:47.610331686 +0000 UTC m=+39.295811091 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e49453a8-c7d9-4248-a8ef-4a819b76806c-metrics-certs") pod "router-default-5964fff8f9-8k6pp" (UID: "e49453a8-c7d9-4248-a8ef-4a819b76806c") : secret "router-metrics-certs-default" not found
Oct 31 01:03:43.610729 ip-10-0-133-197 kubenswrapper[2578]: E1031 01:03:43.610411    2578 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Oct 31 01:03:43.610729 ip-10-0-133-197 kubenswrapper[2578]: E1031 01:03:43.610453    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b66619db-fd81-4435-be2a-a5ea6111e204-cluster-monitoring-operator-tls podName:b66619db-fd81-4435-be2a-a5ea6111e204 nodeName:}" failed. No retries permitted until 2026-10-31 01:03:47.610443512 +0000 UTC m=+39.295922918 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/b66619db-fd81-4435-be2a-a5ea6111e204-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-jccjk" (UID: "b66619db-fd81-4435-be2a-a5ea6111e204") : secret "cluster-monitoring-operator-tls" not found
Oct 31 01:03:43.610729 ip-10-0-133-197 kubenswrapper[2578]: E1031 01:03:43.610502    2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 31 01:03:43.610729 ip-10-0-133-197 kubenswrapper[2578]: E1031 01:03:43.610529    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/85bb69db-c69b-467e-adeb-950bf2947c94-metrics-tls podName:85bb69db-c69b-467e-adeb-950bf2947c94 nodeName:}" failed. No retries permitted until 2026-10-31 01:03:47.610520462 +0000 UTC m=+39.295999868 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/85bb69db-c69b-467e-adeb-950bf2947c94-metrics-tls") pod "dns-default-t7r6c" (UID: "85bb69db-c69b-467e-adeb-950bf2947c94") : secret "dns-default-metrics-tls" not found
Oct 31 01:03:43.610729 ip-10-0-133-197 kubenswrapper[2578]: E1031 01:03:43.610581    2578 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Oct 31 01:03:43.611612 ip-10-0-133-197 kubenswrapper[2578]: E1031 01:03:43.610609    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9dfae6d6-b9a6-4d85-b594-9d947a9e6383-networking-console-plugin-cert podName:9dfae6d6-b9a6-4d85-b594-9d947a9e6383 nodeName:}" failed. No retries permitted until 2026-10-31 01:03:47.610599899 +0000 UTC m=+39.296079303 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/9dfae6d6-b9a6-4d85-b594-9d947a9e6383-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-wm4qt" (UID: "9dfae6d6-b9a6-4d85-b594-9d947a9e6383") : secret "networking-console-plugin-cert" not found
Oct 31 01:03:43.617122 ip-10-0-133-197 kubenswrapper[2578]: E1031 01:03:43.617063    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e49453a8-c7d9-4248-a8ef-4a819b76806c-service-ca-bundle podName:e49453a8-c7d9-4248-a8ef-4a819b76806c nodeName:}" failed. No retries permitted until 2026-10-31 01:03:47.610662566 +0000 UTC m=+39.296141970 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/e49453a8-c7d9-4248-a8ef-4a819b76806c-service-ca-bundle") pod "router-default-5964fff8f9-8k6pp" (UID: "e49453a8-c7d9-4248-a8ef-4a819b76806c") : configmap references non-existent config key: service-ca.crt
Oct 31 01:03:44.055418 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:44.055381    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-v585z" event={"ID":"eafc69c5-15f1-4e4c-97ca-be586e55fbae","Type":"ContainerStarted","Data":"cb0ff96bd1514f24f5fc4be06b5d2e3d40ce3b4ed5c6d47b1b11807fa1e31061"}
Oct 31 01:03:44.061817 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:44.061756    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-8mqbd" event={"ID":"565fefaf-82aa-42a8-9cc7-6517c4bddf1e","Type":"ContainerStarted","Data":"e7be247cfa0d4a3703d5d9670caf05d05e034c7c4142a470ca2f537b939bc2b9"}
Oct 31 01:03:44.065185 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:44.065083    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-zs5g7" event={"ID":"aa4836a5-c71a-485d-82fc-f8b53dae7717","Type":"ContainerStarted","Data":"bba607643ebf9639b0c8044c3e3694561fdacdd3731bcf15feae4311301c9f2b"}
Oct 31 01:03:44.171479 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:44.170226    2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-v585z" podStartSLOduration=386.802294864 podStartE2EDuration="6m29.170206683s" totalImagesPullingTime="2.367911819s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:57:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:03:44.168368549 +0000 UTC m=+35.853847959" watchObservedRunningTime="2026-10-31 01:03:44.170206683 +0000 UTC m=+35.855686098"
Oct 31 01:03:44.478120 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:44.478065    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:03:44.712031 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:44.711993    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:03:45.071090 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:45.071064    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-9257n_f0c0ada7-7688-4dc2-97fc-813e4d0c4ef3/console-operator/0.log"
Oct 31 01:03:45.071209 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:45.071110    2578 generic.go:356] "Generic (PLEG): container finished" podID="f0c0ada7-7688-4dc2-97fc-813e4d0c4ef3" containerID="5543f6096be3ac253192e8a55cfeeef9bb591da46243b768781cfee0eace6aa1" exitCode=255
Oct 31 01:03:45.071453 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:45.071424    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-9257n" event={"ID":"f0c0ada7-7688-4dc2-97fc-813e4d0c4ef3","Type":"ContainerDied","Data":"5543f6096be3ac253192e8a55cfeeef9bb591da46243b768781cfee0eace6aa1"}
Oct 31 01:03:45.071996 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:45.071977    2578 scope.go:122] "RemoveContainer" containerID="5543f6096be3ac253192e8a55cfeeef9bb591da46243b768781cfee0eace6aa1"
Oct 31 01:03:46.079973 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:46.079875    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-9257n_f0c0ada7-7688-4dc2-97fc-813e4d0c4ef3/console-operator/1.log"
Oct 31 01:03:46.081013 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:46.080985    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-9257n_f0c0ada7-7688-4dc2-97fc-813e4d0c4ef3/console-operator/0.log"
Oct 31 01:03:46.081111 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:46.081034    2578 generic.go:356] "Generic (PLEG): container finished" podID="f0c0ada7-7688-4dc2-97fc-813e4d0c4ef3" containerID="46df6877e0e8bb148e755797ac3b08c39be418ccb7326153e8ea50c8505fe54b" exitCode=255
Oct 31 01:03:46.081159 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:46.081113    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-9257n" event={"ID":"f0c0ada7-7688-4dc2-97fc-813e4d0c4ef3","Type":"ContainerDied","Data":"46df6877e0e8bb148e755797ac3b08c39be418ccb7326153e8ea50c8505fe54b"}
Oct 31 01:03:46.081213 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:46.081169    2578 scope.go:122] "RemoveContainer" containerID="5543f6096be3ac253192e8a55cfeeef9bb591da46243b768781cfee0eace6aa1"
Oct 31 01:03:46.081465 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:46.081442    2578 scope.go:122] "RemoveContainer" containerID="46df6877e0e8bb148e755797ac3b08c39be418ccb7326153e8ea50c8505fe54b"
Oct 31 01:03:46.081756 ip-10-0-133-197 kubenswrapper[2578]: E1031 01:03:46.081730    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-9257n_openshift-console-operator(f0c0ada7-7688-4dc2-97fc-813e4d0c4ef3)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-9257n" podUID="f0c0ada7-7688-4dc2-97fc-813e4d0c4ef3"
Oct 31 01:03:46.085629 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:46.085601    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-jkk5x" event={"ID":"c8e98df5-b2f7-49dd-a565-ed60534da496","Type":"ContainerStarted","Data":"84ec5dc0d3ef6b0dc04ef56239f35a37a96800c7a0abc2dd95835fd6cd612524"}
Oct 31 01:03:46.321125 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:46.321068    2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/insights-operator-7c4986d449-jkk5x" podStartSLOduration=386.400301544 podStartE2EDuration="6m31.321049168s" totalImagesPullingTime="4.920747624s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:57:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:03:46.320984837 +0000 UTC m=+38.006464261" watchObservedRunningTime="2026-10-31 01:03:46.321049168 +0000 UTC m=+38.006528582"
Oct 31 01:03:47.092550 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:47.091773    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-pdzj4" event={"ID":"e54a9dc6-36e6-4f1b-9c3a-623a520e11da","Type":"ContainerStarted","Data":"2901f432a0e6351847808a4a8fe2e646d08659e910d2b351767f477cbde70b75"}
Oct 31 01:03:47.095024 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:47.094465    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-n5q9v" event={"ID":"c2be658b-ec92-4782-a78b-d2dccf2249cb","Type":"ContainerStarted","Data":"2fa284236f6fb4b670ac25d5b32b5e19dea2607e333f6376be9cd9cffac81ef5"}
Oct 31 01:03:47.097144 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:47.097121    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-7dpwv" event={"ID":"147bb9af-dfc7-4785-bb5e-906f25d65dd1","Type":"ContainerStarted","Data":"186571f6cf85496c87b9118fbb6644489ea7e1e11307d033a6964ef23ef622e0"}
Oct 31 01:03:47.099214 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:47.099195    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-9257n_f0c0ada7-7688-4dc2-97fc-813e4d0c4ef3/console-operator/1.log"
Oct 31 01:03:47.099758 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:47.099729    2578 scope.go:122] "RemoveContainer" containerID="46df6877e0e8bb148e755797ac3b08c39be418ccb7326153e8ea50c8505fe54b"
Oct 31 01:03:47.099918 ip-10-0-133-197 kubenswrapper[2578]: E1031 01:03:47.099893    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-9257n_openshift-console-operator(f0c0ada7-7688-4dc2-97fc-813e4d0c4ef3)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-9257n" podUID="f0c0ada7-7688-4dc2-97fc-813e4d0c4ef3"
Oct 31 01:03:47.137374 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:47.137293    2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-q56x5"]
Oct 31 01:03:47.170741 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:47.170711    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-q56x5"
Oct 31 01:03:47.199570 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:47.199544    2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-q56x5"]
Oct 31 01:03:47.209394 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:47.209351    2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-pdzj4" podStartSLOduration=386.374559617 podStartE2EDuration="6m32.209338708s" totalImagesPullingTime="5.834779091s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:57:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:03:47.208376218 +0000 UTC m=+38.893855630" watchObservedRunningTime="2026-10-31 01:03:47.209338708 +0000 UTC m=+38.894818114"
Oct 31 01:03:47.227824 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:47.226010    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Oct 31 01:03:47.227824 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:47.226073    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Oct 31 01:03:47.227824 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:47.226208    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-5sjdn\""
Oct 31 01:03:47.260384 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:47.260347    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7dd98c21-e0b1-4289-8fe8-036bf413d444-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-q56x5\" (UID: \"7dd98c21-e0b1-4289-8fe8-036bf413d444\") " pod="openshift-insights/insights-runtime-extractor-q56x5"
Oct 31 01:03:47.260557 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:47.260416    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7fhrx\" (UniqueName: \"kubernetes.io/projected/7dd98c21-e0b1-4289-8fe8-036bf413d444-kube-api-access-7fhrx\") pod \"insights-runtime-extractor-q56x5\" (UID: \"7dd98c21-e0b1-4289-8fe8-036bf413d444\") " pod="openshift-insights/insights-runtime-extractor-q56x5"
Oct 31 01:03:47.260557 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:47.260468    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7dd98c21-e0b1-4289-8fe8-036bf413d444-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-q56x5\" (UID: \"7dd98c21-e0b1-4289-8fe8-036bf413d444\") " pod="openshift-insights/insights-runtime-extractor-q56x5"
Oct 31 01:03:47.260557 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:47.260510    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/7dd98c21-e0b1-4289-8fe8-036bf413d444-data-volume\") pod \"insights-runtime-extractor-q56x5\" (UID: \"7dd98c21-e0b1-4289-8fe8-036bf413d444\") " pod="openshift-insights/insights-runtime-extractor-q56x5"
Oct 31 01:03:47.260781 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:47.260607    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/7dd98c21-e0b1-4289-8fe8-036bf413d444-crio-socket\") pod \"insights-runtime-extractor-q56x5\" (UID: \"7dd98c21-e0b1-4289-8fe8-036bf413d444\") " pod="openshift-insights/insights-runtime-extractor-q56x5"
Oct 31 01:03:47.348635 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:47.348575    2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-n5q9v" podStartSLOduration=386.518025595 podStartE2EDuration="6m32.34855802s" totalImagesPullingTime="5.830532425s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:57:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:03:47.346818815 +0000 UTC m=+39.032298229" watchObservedRunningTime="2026-10-31 01:03:47.34855802 +0000 UTC m=+39.034037434"
Oct 31 01:03:47.362471 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:47.362255    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7fhrx\" (UniqueName: \"kubernetes.io/projected/7dd98c21-e0b1-4289-8fe8-036bf413d444-kube-api-access-7fhrx\") pod \"insights-runtime-extractor-q56x5\" (UID: \"7dd98c21-e0b1-4289-8fe8-036bf413d444\") " pod="openshift-insights/insights-runtime-extractor-q56x5"
Oct 31 01:03:47.362471 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:47.362337    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7dd98c21-e0b1-4289-8fe8-036bf413d444-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-q56x5\" (UID: \"7dd98c21-e0b1-4289-8fe8-036bf413d444\") " pod="openshift-insights/insights-runtime-extractor-q56x5"
Oct 31 01:03:47.362471 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:47.362378    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/7dd98c21-e0b1-4289-8fe8-036bf413d444-data-volume\") pod \"insights-runtime-extractor-q56x5\" (UID: \"7dd98c21-e0b1-4289-8fe8-036bf413d444\") " pod="openshift-insights/insights-runtime-extractor-q56x5"
Oct 31 01:03:47.362700 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:47.362478    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/7dd98c21-e0b1-4289-8fe8-036bf413d444-crio-socket\") pod \"insights-runtime-extractor-q56x5\" (UID: \"7dd98c21-e0b1-4289-8fe8-036bf413d444\") " pod="openshift-insights/insights-runtime-extractor-q56x5"
Oct 31 01:03:47.362700 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:47.362557    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7dd98c21-e0b1-4289-8fe8-036bf413d444-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-q56x5\" (UID: \"7dd98c21-e0b1-4289-8fe8-036bf413d444\") " pod="openshift-insights/insights-runtime-extractor-q56x5"
Oct 31 01:03:47.362700 ip-10-0-133-197 kubenswrapper[2578]: E1031 01:03:47.362644    2578 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Oct 31 01:03:47.362822 ip-10-0-133-197 kubenswrapper[2578]: E1031 01:03:47.362703    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/7dd98c21-e0b1-4289-8fe8-036bf413d444-kube-rbac-proxy-cm podName:7dd98c21-e0b1-4289-8fe8-036bf413d444 nodeName:}" failed. No retries permitted until 2026-10-31 01:03:47.86268657 +0000 UTC m=+39.548165966 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/7dd98c21-e0b1-4289-8fe8-036bf413d444-kube-rbac-proxy-cm") pod "insights-runtime-extractor-q56x5" (UID: "7dd98c21-e0b1-4289-8fe8-036bf413d444") : configmap "kube-rbac-proxy" not found
Oct 31 01:03:47.363050 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:47.363017    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/7dd98c21-e0b1-4289-8fe8-036bf413d444-data-volume\") pod \"insights-runtime-extractor-q56x5\" (UID: \"7dd98c21-e0b1-4289-8fe8-036bf413d444\") " pod="openshift-insights/insights-runtime-extractor-q56x5"
Oct 31 01:03:47.363153 ip-10-0-133-197 kubenswrapper[2578]: E1031 01:03:47.363124    2578 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 31 01:03:47.363153 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:47.363137    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/7dd98c21-e0b1-4289-8fe8-036bf413d444-crio-socket\") pod \"insights-runtime-extractor-q56x5\" (UID: \"7dd98c21-e0b1-4289-8fe8-036bf413d444\") " pod="openshift-insights/insights-runtime-extractor-q56x5"
Oct 31 01:03:47.363249 ip-10-0-133-197 kubenswrapper[2578]: E1031 01:03:47.363176    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7dd98c21-e0b1-4289-8fe8-036bf413d444-insights-runtime-extractor-tls podName:7dd98c21-e0b1-4289-8fe8-036bf413d444 nodeName:}" failed. No retries permitted until 2026-10-31 01:03:47.86315905 +0000 UTC m=+39.548638447 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/7dd98c21-e0b1-4289-8fe8-036bf413d444-insights-runtime-extractor-tls") pod "insights-runtime-extractor-q56x5" (UID: "7dd98c21-e0b1-4289-8fe8-036bf413d444") : secret "insights-runtime-extractor-tls" not found
Oct 31 01:03:47.430110 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:47.430043    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7fhrx\" (UniqueName: \"kubernetes.io/projected/7dd98c21-e0b1-4289-8fe8-036bf413d444-kube-api-access-7fhrx\") pod \"insights-runtime-extractor-q56x5\" (UID: \"7dd98c21-e0b1-4289-8fe8-036bf413d444\") " pod="openshift-insights/insights-runtime-extractor-q56x5"
Oct 31 01:03:47.453664 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:47.453616    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:03:47.455127 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:47.455106    2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/periodic-gathering-b2z6s-vxz9h"]
Oct 31 01:03:47.460216 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:47.460192    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-b2z6s-vxz9h"
Oct 31 01:03:47.514197 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:47.513009    2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-b2z6s-vxz9h"]
Oct 31 01:03:47.514197 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:47.513310    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-fbmr7\""
Oct 31 01:03:47.564682 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:47.564659    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9122c2fd-f29f-4e74-9615-15078517a3c0-service-ca-bundle\") pod \"periodic-gathering-b2z6s-vxz9h\" (UID: \"9122c2fd-f29f-4e74-9615-15078517a3c0\") " pod="openshift-insights/periodic-gathering-b2z6s-vxz9h"
Oct 31 01:03:47.564813 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:47.564727    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ff4dcba2-df8d-4324-9e0c-abb23a5e6982-registry-tls\") pod \"image-registry-56d8464d7c-vpwjc\" (UID: \"ff4dcba2-df8d-4324-9e0c-abb23a5e6982\") " pod="openshift-image-registry/image-registry-56d8464d7c-vpwjc"
Oct 31 01:03:47.564857 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:47.564826    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/aa112079-b749-404c-9081-113220707fcd-registry-tls\") pod \"image-registry-75575b5db8-wtq4d\" (UID: \"aa112079-b749-404c-9081-113220707fcd\") " pod="openshift-image-registry/image-registry-75575b5db8-wtq4d"
Oct 31 01:03:47.564937 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:47.564920    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r4l7n\" (UniqueName: \"kubernetes.io/projected/9122c2fd-f29f-4e74-9615-15078517a3c0-kube-api-access-r4l7n\") pod \"periodic-gathering-b2z6s-vxz9h\" (UID: \"9122c2fd-f29f-4e74-9615-15078517a3c0\") " pod="openshift-insights/periodic-gathering-b2z6s-vxz9h"
Oct 31 01:03:47.564982 ip-10-0-133-197 kubenswrapper[2578]: E1031 01:03:47.564946    2578 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 31 01:03:47.564982 ip-10-0-133-197 kubenswrapper[2578]: E1031 01:03:47.564962    2578 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-56d8464d7c-vpwjc: secret "image-registry-tls" not found
Oct 31 01:03:47.565077 ip-10-0-133-197 kubenswrapper[2578]: E1031 01:03:47.565020    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ff4dcba2-df8d-4324-9e0c-abb23a5e6982-registry-tls podName:ff4dcba2-df8d-4324-9e0c-abb23a5e6982 nodeName:}" failed. No retries permitted until 2026-10-31 01:03:55.565003555 +0000 UTC m=+47.250482964 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/ff4dcba2-df8d-4324-9e0c-abb23a5e6982-registry-tls") pod "image-registry-56d8464d7c-vpwjc" (UID: "ff4dcba2-df8d-4324-9e0c-abb23a5e6982") : secret "image-registry-tls" not found
Oct 31 01:03:47.565077 ip-10-0-133-197 kubenswrapper[2578]: E1031 01:03:47.565039    2578 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 31 01:03:47.565077 ip-10-0-133-197 kubenswrapper[2578]: E1031 01:03:47.565055    2578 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-75575b5db8-wtq4d: secret "image-registry-tls" not found
Oct 31 01:03:47.565219 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:47.565078    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9122c2fd-f29f-4e74-9615-15078517a3c0-trusted-ca-bundle\") pod \"periodic-gathering-b2z6s-vxz9h\" (UID: \"9122c2fd-f29f-4e74-9615-15078517a3c0\") " pod="openshift-insights/periodic-gathering-b2z6s-vxz9h"
Oct 31 01:03:47.565219 ip-10-0-133-197 kubenswrapper[2578]: E1031 01:03:47.565102    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/aa112079-b749-404c-9081-113220707fcd-registry-tls podName:aa112079-b749-404c-9081-113220707fcd nodeName:}" failed. No retries permitted until 2026-10-31 01:03:55.5650869 +0000 UTC m=+47.250566322 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/aa112079-b749-404c-9081-113220707fcd-registry-tls") pod "image-registry-75575b5db8-wtq4d" (UID: "aa112079-b749-404c-9081-113220707fcd") : secret "image-registry-tls" not found
Oct 31 01:03:47.565219 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:47.565191    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/9122c2fd-f29f-4e74-9615-15078517a3c0-archives-path\") pod \"periodic-gathering-b2z6s-vxz9h\" (UID: \"9122c2fd-f29f-4e74-9615-15078517a3c0\") " pod="openshift-insights/periodic-gathering-b2z6s-vxz9h"
Oct 31 01:03:47.643894 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:47.643868    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:03:47.665915 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:47.665888    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/b66619db-fd81-4435-be2a-a5ea6111e204-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-jccjk\" (UID: \"b66619db-fd81-4435-be2a-a5ea6111e204\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-jccjk"
Oct 31 01:03:47.666003 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:47.665948    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/9122c2fd-f29f-4e74-9615-15078517a3c0-archives-path\") pod \"periodic-gathering-b2z6s-vxz9h\" (UID: \"9122c2fd-f29f-4e74-9615-15078517a3c0\") " pod="openshift-insights/periodic-gathering-b2z6s-vxz9h"
Oct 31 01:03:47.666003 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:47.665980    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/85bb69db-c69b-467e-adeb-950bf2947c94-metrics-tls\") pod \"dns-default-t7r6c\" (UID: \"85bb69db-c69b-467e-adeb-950bf2947c94\") " pod="openshift-dns/dns-default-t7r6c"
Oct 31 01:03:47.666118 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:47.666008    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9122c2fd-f29f-4e74-9615-15078517a3c0-service-ca-bundle\") pod \"periodic-gathering-b2z6s-vxz9h\" (UID: \"9122c2fd-f29f-4e74-9615-15078517a3c0\") " pod="openshift-insights/periodic-gathering-b2z6s-vxz9h"
Oct 31 01:03:47.666118 ip-10-0-133-197 kubenswrapper[2578]: E1031 01:03:47.666015    2578 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Oct 31 01:03:47.666118 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:47.666046    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/9dfae6d6-b9a6-4d85-b594-9d947a9e6383-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-wm4qt\" (UID: \"9dfae6d6-b9a6-4d85-b594-9d947a9e6383\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-wm4qt"
Oct 31 01:03:47.666118 ip-10-0-133-197 kubenswrapper[2578]: E1031 01:03:47.666073    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b66619db-fd81-4435-be2a-a5ea6111e204-cluster-monitoring-operator-tls podName:b66619db-fd81-4435-be2a-a5ea6111e204 nodeName:}" failed. No retries permitted until 2026-10-31 01:03:55.666056301 +0000 UTC m=+47.351535699 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/b66619db-fd81-4435-be2a-a5ea6111e204-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-jccjk" (UID: "b66619db-fd81-4435-be2a-a5ea6111e204") : secret "cluster-monitoring-operator-tls" not found
Oct 31 01:03:47.666118 ip-10-0-133-197 kubenswrapper[2578]: E1031 01:03:47.666110    2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 31 01:03:47.666390 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:47.666120    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e49453a8-c7d9-4248-a8ef-4a819b76806c-service-ca-bundle\") pod \"router-default-5964fff8f9-8k6pp\" (UID: \"e49453a8-c7d9-4248-a8ef-4a819b76806c\") " pod="openshift-ingress/router-default-5964fff8f9-8k6pp"
Oct 31 01:03:47.666390 ip-10-0-133-197 kubenswrapper[2578]: E1031 01:03:47.666170    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/85bb69db-c69b-467e-adeb-950bf2947c94-metrics-tls podName:85bb69db-c69b-467e-adeb-950bf2947c94 nodeName:}" failed. No retries permitted until 2026-10-31 01:03:55.666150209 +0000 UTC m=+47.351629617 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/85bb69db-c69b-467e-adeb-950bf2947c94-metrics-tls") pod "dns-default-t7r6c" (UID: "85bb69db-c69b-467e-adeb-950bf2947c94") : secret "dns-default-metrics-tls" not found
Oct 31 01:03:47.666390 ip-10-0-133-197 kubenswrapper[2578]: E1031 01:03:47.666115    2578 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Oct 31 01:03:47.666390 ip-10-0-133-197 kubenswrapper[2578]: E1031 01:03:47.666230    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e49453a8-c7d9-4248-a8ef-4a819b76806c-service-ca-bundle podName:e49453a8-c7d9-4248-a8ef-4a819b76806c nodeName:}" failed. No retries permitted until 2026-10-31 01:03:55.666219924 +0000 UTC m=+47.351699316 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/e49453a8-c7d9-4248-a8ef-4a819b76806c-service-ca-bundle") pod "router-default-5964fff8f9-8k6pp" (UID: "e49453a8-c7d9-4248-a8ef-4a819b76806c") : configmap references non-existent config key: service-ca.crt
Oct 31 01:03:47.666390 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:47.666252    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/f273b9e5-0419-4587-a9c7-a0c8bec02acf-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-srcwl\" (UID: \"f273b9e5-0419-4587-a9c7-a0c8bec02acf\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-srcwl"
Oct 31 01:03:47.666390 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:47.666290    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r4l7n\" (UniqueName: \"kubernetes.io/projected/9122c2fd-f29f-4e74-9615-15078517a3c0-kube-api-access-r4l7n\") pod \"periodic-gathering-b2z6s-vxz9h\" (UID: \"9122c2fd-f29f-4e74-9615-15078517a3c0\") " pod="openshift-insights/periodic-gathering-b2z6s-vxz9h"
Oct 31 01:03:47.666390 ip-10-0-133-197 kubenswrapper[2578]: E1031 01:03:47.666350    2578 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Oct 31 01:03:47.666390 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:47.666371    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e49453a8-c7d9-4248-a8ef-4a819b76806c-metrics-certs\") pod \"router-default-5964fff8f9-8k6pp\" (UID: \"e49453a8-c7d9-4248-a8ef-4a819b76806c\") " pod="openshift-ingress/router-default-5964fff8f9-8k6pp"
Oct 31 01:03:47.666390 ip-10-0-133-197 kubenswrapper[2578]: E1031 01:03:47.666388    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f273b9e5-0419-4587-a9c7-a0c8bec02acf-samples-operator-tls podName:f273b9e5-0419-4587-a9c7-a0c8bec02acf nodeName:}" failed. No retries permitted until 2026-10-31 01:03:55.666378968 +0000 UTC m=+47.351858373 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/f273b9e5-0419-4587-a9c7-a0c8bec02acf-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-srcwl" (UID: "f273b9e5-0419-4587-a9c7-a0c8bec02acf") : secret "samples-operator-tls" not found
Oct 31 01:03:47.666728 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:47.666389    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/9122c2fd-f29f-4e74-9615-15078517a3c0-archives-path\") pod \"periodic-gathering-b2z6s-vxz9h\" (UID: \"9122c2fd-f29f-4e74-9615-15078517a3c0\") " pod="openshift-insights/periodic-gathering-b2z6s-vxz9h"
Oct 31 01:03:47.666728 ip-10-0-133-197 kubenswrapper[2578]: E1031 01:03:47.666416    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9dfae6d6-b9a6-4d85-b594-9d947a9e6383-networking-console-plugin-cert podName:9dfae6d6-b9a6-4d85-b594-9d947a9e6383 nodeName:}" failed. No retries permitted until 2026-10-31 01:03:55.66640438 +0000 UTC m=+47.351883776 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/9dfae6d6-b9a6-4d85-b594-9d947a9e6383-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-wm4qt" (UID: "9dfae6d6-b9a6-4d85-b594-9d947a9e6383") : secret "networking-console-plugin-cert" not found
Oct 31 01:03:47.666728 ip-10-0-133-197 kubenswrapper[2578]: E1031 01:03:47.666434    2578 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Oct 31 01:03:47.666728 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:47.666460    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9122c2fd-f29f-4e74-9615-15078517a3c0-trusted-ca-bundle\") pod \"periodic-gathering-b2z6s-vxz9h\" (UID: \"9122c2fd-f29f-4e74-9615-15078517a3c0\") " pod="openshift-insights/periodic-gathering-b2z6s-vxz9h"
Oct 31 01:03:47.666728 ip-10-0-133-197 kubenswrapper[2578]: E1031 01:03:47.666519    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e49453a8-c7d9-4248-a8ef-4a819b76806c-metrics-certs podName:e49453a8-c7d9-4248-a8ef-4a819b76806c nodeName:}" failed. No retries permitted until 2026-10-31 01:03:55.666505324 +0000 UTC m=+47.351984727 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e49453a8-c7d9-4248-a8ef-4a819b76806c-metrics-certs") pod "router-default-5964fff8f9-8k6pp" (UID: "e49453a8-c7d9-4248-a8ef-4a819b76806c") : secret "router-metrics-certs-default" not found
Oct 31 01:03:47.666728 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:47.666656    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9122c2fd-f29f-4e74-9615-15078517a3c0-service-ca-bundle\") pod \"periodic-gathering-b2z6s-vxz9h\" (UID: \"9122c2fd-f29f-4e74-9615-15078517a3c0\") " pod="openshift-insights/periodic-gathering-b2z6s-vxz9h"
Oct 31 01:03:47.667124 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:47.667104    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9122c2fd-f29f-4e74-9615-15078517a3c0-trusted-ca-bundle\") pod \"periodic-gathering-b2z6s-vxz9h\" (UID: \"9122c2fd-f29f-4e74-9615-15078517a3c0\") " pod="openshift-insights/periodic-gathering-b2z6s-vxz9h"
Oct 31 01:03:47.712141 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:47.712102    2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-7dpwv" podStartSLOduration=2.467930682 podStartE2EDuration="8.71208416s" totalImagesPullingTime="6.244153478s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:03:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:03:47.710766262 +0000 UTC m=+39.396245676" watchObservedRunningTime="2026-10-31 01:03:47.71208416 +0000 UTC m=+39.397563605"
Oct 31 01:03:47.729954 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:47.729926    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r4l7n\" (UniqueName: \"kubernetes.io/projected/9122c2fd-f29f-4e74-9615-15078517a3c0-kube-api-access-r4l7n\") pod \"periodic-gathering-b2z6s-vxz9h\" (UID: \"9122c2fd-f29f-4e74-9615-15078517a3c0\") " pod="openshift-insights/periodic-gathering-b2z6s-vxz9h"
Oct 31 01:03:47.781733 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:47.781705    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-b2z6s-vxz9h"
Oct 31 01:03:47.870286 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:47.869464    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7dd98c21-e0b1-4289-8fe8-036bf413d444-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-q56x5\" (UID: \"7dd98c21-e0b1-4289-8fe8-036bf413d444\") " pod="openshift-insights/insights-runtime-extractor-q56x5"
Oct 31 01:03:47.870286 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:47.869690    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7dd98c21-e0b1-4289-8fe8-036bf413d444-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-q56x5\" (UID: \"7dd98c21-e0b1-4289-8fe8-036bf413d444\") " pod="openshift-insights/insights-runtime-extractor-q56x5"
Oct 31 01:03:47.870286 ip-10-0-133-197 kubenswrapper[2578]: E1031 01:03:47.869841    2578 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 31 01:03:47.870286 ip-10-0-133-197 kubenswrapper[2578]: E1031 01:03:47.869906    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7dd98c21-e0b1-4289-8fe8-036bf413d444-insights-runtime-extractor-tls podName:7dd98c21-e0b1-4289-8fe8-036bf413d444 nodeName:}" failed. No retries permitted until 2026-10-31 01:03:48.869887501 +0000 UTC m=+40.555366906 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/7dd98c21-e0b1-4289-8fe8-036bf413d444-insights-runtime-extractor-tls") pod "insights-runtime-extractor-q56x5" (UID: "7dd98c21-e0b1-4289-8fe8-036bf413d444") : secret "insights-runtime-extractor-tls" not found
Oct 31 01:03:47.870286 ip-10-0-133-197 kubenswrapper[2578]: E1031 01:03:47.870037    2578 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Oct 31 01:03:47.870286 ip-10-0-133-197 kubenswrapper[2578]: E1031 01:03:47.870152    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/7dd98c21-e0b1-4289-8fe8-036bf413d444-kube-rbac-proxy-cm podName:7dd98c21-e0b1-4289-8fe8-036bf413d444 nodeName:}" failed. No retries permitted until 2026-10-31 01:03:48.870141273 +0000 UTC m=+40.555620678 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/7dd98c21-e0b1-4289-8fe8-036bf413d444-kube-rbac-proxy-cm") pod "insights-runtime-extractor-q56x5" (UID: "7dd98c21-e0b1-4289-8fe8-036bf413d444") : configmap "kube-rbac-proxy" not found
Oct 31 01:03:47.876587 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:47.876565    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:03:48.072749 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:48.072718    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:03:48.111218 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:48.111171    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-bpmjc" event={"ID":"69ff2b62-4588-4cb0-bdad-747c3692138c","Type":"ContainerStarted","Data":"4be6dfeb2e3438a23631d13d636b3ad5f5ded0f358f9686023bd18604ca85ed4"}
Oct 31 01:03:48.205512 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:48.204260    2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-b2z6s-vxz9h"]
Oct 31 01:03:48.206241 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:48.206219    2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9122c2fd_f29f_4e74_9615_15078517a3c0.slice/crio-91c64ba91f2d616e39faaef286d3e721f03f6ac5c7d010d6b674016160e7aa6e WatchSource:0}: Error finding container 91c64ba91f2d616e39faaef286d3e721f03f6ac5c7d010d6b674016160e7aa6e: Status 404 returned error can't find the container with id 91c64ba91f2d616e39faaef286d3e721f03f6ac5c7d010d6b674016160e7aa6e
Oct 31 01:03:48.239636 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:48.238927    2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-bpmjc" podStartSLOduration=386.255321613 podStartE2EDuration="6m33.238910092s" totalImagesPullingTime="6.983588479s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:57:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:03:48.237772996 +0000 UTC m=+39.923252412" watchObservedRunningTime="2026-10-31 01:03:48.238910092 +0000 UTC m=+39.924389506"
Oct 31 01:03:48.292811 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:48.292739    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:03:48.354112 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:48.353583    2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-s2r9l"]
Oct 31 01:03:48.371496 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:48.371466    2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-cqpmt"]
Oct 31 01:03:48.371655 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:48.371509    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-s2r9l"
Oct 31 01:03:48.384280 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:48.384004    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-cqpmt"
Oct 31 01:03:48.413808 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:48.413784    2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-s2r9l"]
Oct 31 01:03:48.423923 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:48.423897    2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-cqpmt"]
Oct 31 01:03:48.425462 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:48.425442    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Oct 31 01:03:48.425583 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:48.425546    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-cw2dc\""
Oct 31 01:03:48.425647 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:48.425632    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Oct 31 01:03:48.474676 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:48.474653    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qzhpr\" (UniqueName: \"kubernetes.io/projected/b5975856-8eb3-4601-a7e1-d1b1b3a957e8-kube-api-access-qzhpr\") pod \"migrator-66df5d496b-s2r9l\" (UID: \"b5975856-8eb3-4601-a7e1-d1b1b3a957e8\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-s2r9l"
Oct 31 01:03:48.474858 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:48.474835    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bspb5\" (UniqueName: \"kubernetes.io/projected/cb657366-6696-4c3e-a3a3-603ce00cf11a-kube-api-access-bspb5\") pod \"migrator-66df5d496b-cqpmt\" (UID: \"cb657366-6696-4c3e-a3a3-603ce00cf11a\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-cqpmt"
Oct 31 01:03:48.521072 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:48.521008    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:03:48.575937 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:48.575914    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bspb5\" (UniqueName: \"kubernetes.io/projected/cb657366-6696-4c3e-a3a3-603ce00cf11a-kube-api-access-bspb5\") pod \"migrator-66df5d496b-cqpmt\" (UID: \"cb657366-6696-4c3e-a3a3-603ce00cf11a\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-cqpmt"
Oct 31 01:03:48.576085 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:48.576057    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qzhpr\" (UniqueName: \"kubernetes.io/projected/b5975856-8eb3-4601-a7e1-d1b1b3a957e8-kube-api-access-qzhpr\") pod \"migrator-66df5d496b-s2r9l\" (UID: \"b5975856-8eb3-4601-a7e1-d1b1b3a957e8\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-s2r9l"
Oct 31 01:03:48.635250 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:48.635190    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qzhpr\" (UniqueName: \"kubernetes.io/projected/b5975856-8eb3-4601-a7e1-d1b1b3a957e8-kube-api-access-qzhpr\") pod \"migrator-66df5d496b-s2r9l\" (UID: \"b5975856-8eb3-4601-a7e1-d1b1b3a957e8\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-s2r9l"
Oct 31 01:03:48.635439 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:48.635376    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bspb5\" (UniqueName: \"kubernetes.io/projected/cb657366-6696-4c3e-a3a3-603ce00cf11a-kube-api-access-bspb5\") pod \"migrator-66df5d496b-cqpmt\" (UID: \"cb657366-6696-4c3e-a3a3-603ce00cf11a\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-cqpmt"
Oct 31 01:03:48.694920 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:48.694889    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-s2r9l"
Oct 31 01:03:48.715866 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:48.715841    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-cqpmt"
Oct 31 01:03:48.716867 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:48.716843    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:03:48.878290 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:48.878136    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7dd98c21-e0b1-4289-8fe8-036bf413d444-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-q56x5\" (UID: \"7dd98c21-e0b1-4289-8fe8-036bf413d444\") " pod="openshift-insights/insights-runtime-extractor-q56x5"
Oct 31 01:03:48.878460 ip-10-0-133-197 kubenswrapper[2578]: E1031 01:03:48.878265    2578 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 31 01:03:48.878460 ip-10-0-133-197 kubenswrapper[2578]: E1031 01:03:48.878392    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7dd98c21-e0b1-4289-8fe8-036bf413d444-insights-runtime-extractor-tls podName:7dd98c21-e0b1-4289-8fe8-036bf413d444 nodeName:}" failed. No retries permitted until 2026-10-31 01:03:50.878374069 +0000 UTC m=+42.563853462 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/7dd98c21-e0b1-4289-8fe8-036bf413d444-insights-runtime-extractor-tls") pod "insights-runtime-extractor-q56x5" (UID: "7dd98c21-e0b1-4289-8fe8-036bf413d444") : secret "insights-runtime-extractor-tls" not found
Oct 31 01:03:48.878582 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:48.878470    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7dd98c21-e0b1-4289-8fe8-036bf413d444-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-q56x5\" (UID: \"7dd98c21-e0b1-4289-8fe8-036bf413d444\") " pod="openshift-insights/insights-runtime-extractor-q56x5"
Oct 31 01:03:48.878582 ip-10-0-133-197 kubenswrapper[2578]: E1031 01:03:48.878537    2578 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Oct 31 01:03:48.878582 ip-10-0-133-197 kubenswrapper[2578]: E1031 01:03:48.878570    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/7dd98c21-e0b1-4289-8fe8-036bf413d444-kube-rbac-proxy-cm podName:7dd98c21-e0b1-4289-8fe8-036bf413d444 nodeName:}" failed. No retries permitted until 2026-10-31 01:03:50.878559734 +0000 UTC m=+42.564039125 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/7dd98c21-e0b1-4289-8fe8-036bf413d444-kube-rbac-proxy-cm") pod "insights-runtime-extractor-q56x5" (UID: "7dd98c21-e0b1-4289-8fe8-036bf413d444") : configmap "kube-rbac-proxy" not found
Oct 31 01:03:49.102097 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:49.102064    2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-s2r9l"]
Oct 31 01:03:49.105135 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:49.105099    2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb5975856_8eb3_4601_a7e1_d1b1b3a957e8.slice/crio-306f16787f3392fc8b7ace0026e1c8efe17179d370cdd50b03d02cb649070aaa WatchSource:0}: Error finding container 306f16787f3392fc8b7ace0026e1c8efe17179d370cdd50b03d02cb649070aaa: Status 404 returned error can't find the container with id 306f16787f3392fc8b7ace0026e1c8efe17179d370cdd50b03d02cb649070aaa
Oct 31 01:03:49.117124 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:49.117044    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-s2r9l" event={"ID":"b5975856-8eb3-4601-a7e1-d1b1b3a957e8","Type":"ContainerStarted","Data":"306f16787f3392fc8b7ace0026e1c8efe17179d370cdd50b03d02cb649070aaa"}
Oct 31 01:03:49.119126 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:49.119099    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-8mqbd" event={"ID":"565fefaf-82aa-42a8-9cc7-6517c4bddf1e","Type":"ContainerStarted","Data":"16ec198ad8922f815a5858c2fa2a8c733912237b2de000424d9a65388b504be6"}
Oct 31 01:03:49.119239 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:49.119223    2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-8mqbd"
Oct 31 01:03:49.120800 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:49.120779    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-b2z6s-vxz9h" event={"ID":"9122c2fd-f29f-4e74-9615-15078517a3c0","Type":"ContainerStarted","Data":"9532152a746063f8ad7d07b9839b23f02eedf132bfc3900c57a3d6cf1f9597ec"}
Oct 31 01:03:49.120913 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:49.120821    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-b2z6s-vxz9h" event={"ID":"9122c2fd-f29f-4e74-9615-15078517a3c0","Type":"ContainerStarted","Data":"91c64ba91f2d616e39faaef286d3e721f03f6ac5c7d010d6b674016160e7aa6e"}
Oct 31 01:03:49.155792 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:49.155770    2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-cqpmt"]
Oct 31 01:03:49.158570 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:49.158539    2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb657366_6696_4c3e_a3a3_603ce00cf11a.slice/crio-7182622ef6dda99f8aed4b16c806aa974d9e90326009d6dc3149e9bd8b4f5cb2 WatchSource:0}: Error finding container 7182622ef6dda99f8aed4b16c806aa974d9e90326009d6dc3149e9bd8b4f5cb2: Status 404 returned error can't find the container with id 7182622ef6dda99f8aed4b16c806aa974d9e90326009d6dc3149e9bd8b4f5cb2
Oct 31 01:03:49.179225 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:49.179203    2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-xhsdf"]
Oct 31 01:03:49.195870 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:49.195849    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-5fffc97bdf-xhsdf"
Oct 31 01:03:49.239202 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:49.239155    2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-8mqbd" podStartSLOduration=34.513738763 podStartE2EDuration="40.239143331s" totalImagesPullingTime="5.725404568s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:03:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:03:49.238025549 +0000 UTC m=+40.923504964" watchObservedRunningTime="2026-10-31 01:03:49.239143331 +0000 UTC m=+40.924622783"
Oct 31 01:03:49.258191 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:49.258153    2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-xhsdf"]
Oct 31 01:03:49.258513 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:49.258495    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-42gnc\""
Oct 31 01:03:49.258634 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:49.258620    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Oct 31 01:03:49.279084 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:49.279064    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Oct 31 01:03:49.279233 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:49.279065    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Oct 31 01:03:49.282521 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:49.282493    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/6fffc15e-94bd-4939-b38f-17d1fe6cc0b5-signing-cabundle\") pod \"service-ca-5fffc97bdf-xhsdf\" (UID: \"6fffc15e-94bd-4939-b38f-17d1fe6cc0b5\") " pod="openshift-service-ca/service-ca-5fffc97bdf-xhsdf"
Oct 31 01:03:49.282690 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:49.282666    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dtjj5\" (UniqueName: \"kubernetes.io/projected/6fffc15e-94bd-4939-b38f-17d1fe6cc0b5-kube-api-access-dtjj5\") pod \"service-ca-5fffc97bdf-xhsdf\" (UID: \"6fffc15e-94bd-4939-b38f-17d1fe6cc0b5\") " pod="openshift-service-ca/service-ca-5fffc97bdf-xhsdf"
Oct 31 01:03:49.282821 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:49.282724    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/6fffc15e-94bd-4939-b38f-17d1fe6cc0b5-signing-key\") pod \"service-ca-5fffc97bdf-xhsdf\" (UID: \"6fffc15e-94bd-4939-b38f-17d1fe6cc0b5\") " pod="openshift-service-ca/service-ca-5fffc97bdf-xhsdf"
Oct 31 01:03:49.287965 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:49.287932    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Oct 31 01:03:49.383579 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:49.383534    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/6fffc15e-94bd-4939-b38f-17d1fe6cc0b5-signing-cabundle\") pod \"service-ca-5fffc97bdf-xhsdf\" (UID: \"6fffc15e-94bd-4939-b38f-17d1fe6cc0b5\") " pod="openshift-service-ca/service-ca-5fffc97bdf-xhsdf"
Oct 31 01:03:49.383755 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:49.383674    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dtjj5\" (UniqueName: \"kubernetes.io/projected/6fffc15e-94bd-4939-b38f-17d1fe6cc0b5-kube-api-access-dtjj5\") pod \"service-ca-5fffc97bdf-xhsdf\" (UID: \"6fffc15e-94bd-4939-b38f-17d1fe6cc0b5\") " pod="openshift-service-ca/service-ca-5fffc97bdf-xhsdf"
Oct 31 01:03:49.383755 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:49.383722    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/6fffc15e-94bd-4939-b38f-17d1fe6cc0b5-signing-key\") pod \"service-ca-5fffc97bdf-xhsdf\" (UID: \"6fffc15e-94bd-4939-b38f-17d1fe6cc0b5\") " pod="openshift-service-ca/service-ca-5fffc97bdf-xhsdf"
Oct 31 01:03:49.384349 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:49.384311    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/6fffc15e-94bd-4939-b38f-17d1fe6cc0b5-signing-cabundle\") pod \"service-ca-5fffc97bdf-xhsdf\" (UID: \"6fffc15e-94bd-4939-b38f-17d1fe6cc0b5\") " pod="openshift-service-ca/service-ca-5fffc97bdf-xhsdf"
Oct 31 01:03:49.387403 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:49.387377    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/6fffc15e-94bd-4939-b38f-17d1fe6cc0b5-signing-key\") pod \"service-ca-5fffc97bdf-xhsdf\" (UID: \"6fffc15e-94bd-4939-b38f-17d1fe6cc0b5\") " pod="openshift-service-ca/service-ca-5fffc97bdf-xhsdf"
Oct 31 01:03:49.432240 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:49.432197    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:03:49.444026 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:49.443998    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dtjj5\" (UniqueName: \"kubernetes.io/projected/6fffc15e-94bd-4939-b38f-17d1fe6cc0b5-kube-api-access-dtjj5\") pod \"service-ca-5fffc97bdf-xhsdf\" (UID: \"6fffc15e-94bd-4939-b38f-17d1fe6cc0b5\") " pod="openshift-service-ca/service-ca-5fffc97bdf-xhsdf"
Oct 31 01:03:49.469573 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:49.469498    2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/periodic-gathering-b2z6s-vxz9h" podStartSLOduration=2.469480896 podStartE2EDuration="2.469480896s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:03:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:03:49.469155358 +0000 UTC m=+41.154634772" watchObservedRunningTime="2026-10-31 01:03:49.469480896 +0000 UTC m=+41.154960310"
Oct 31 01:03:49.517851 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:49.517786    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-5fffc97bdf-xhsdf"
Oct 31 01:03:49.631420 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:49.631384    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:03:49.901361 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:49.901217    2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-xhsdf"]
Oct 31 01:03:49.904814 ip-10-0-133-197 kubenswrapper[2578]: W1031 01:03:49.904785    2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6fffc15e_94bd_4939_b38f_17d1fe6cc0b5.slice/crio-6e130a212380a09d1a1465bc747f921897753868d89543228dcfa0e8b911773c WatchSource:0}: Error finding container 6e130a212380a09d1a1465bc747f921897753868d89543228dcfa0e8b911773c: Status 404 returned error can't find the container with id 6e130a212380a09d1a1465bc747f921897753868d89543228dcfa0e8b911773c
Oct 31 01:03:50.126152 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:50.125969    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-5fffc97bdf-xhsdf" event={"ID":"6fffc15e-94bd-4939-b38f-17d1fe6cc0b5","Type":"ContainerStarted","Data":"cbeb7aef124efb9235a67fa09e7312ededa3950ad84eab6c23975bc2cda56d85"}
Oct 31 01:03:50.126698 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:50.126172    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-5fffc97bdf-xhsdf" event={"ID":"6fffc15e-94bd-4939-b38f-17d1fe6cc0b5","Type":"ContainerStarted","Data":"6e130a212380a09d1a1465bc747f921897753868d89543228dcfa0e8b911773c"}
Oct 31 01:03:50.127816 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:50.127427    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-cqpmt" event={"ID":"cb657366-6696-4c3e-a3a3-603ce00cf11a","Type":"ContainerStarted","Data":"7182622ef6dda99f8aed4b16c806aa974d9e90326009d6dc3149e9bd8b4f5cb2"}
Oct 31 01:03:50.184791 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:50.184726    2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-9257n"
Oct 31 01:03:50.184791 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:50.184774    2578 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-7cd56b8dbc-9257n"
Oct 31 01:03:50.185163 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:50.185144    2578 scope.go:122] "RemoveContainer" containerID="46df6877e0e8bb148e755797ac3b08c39be418ccb7326153e8ea50c8505fe54b"
Oct 31 01:03:50.185878 ip-10-0-133-197 kubenswrapper[2578]: E1031 01:03:50.185852    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-9257n_openshift-console-operator(f0c0ada7-7688-4dc2-97fc-813e4d0c4ef3)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-9257n" podUID="f0c0ada7-7688-4dc2-97fc-813e4d0c4ef3"
Oct 31 01:03:50.370170 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:50.370097    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:03:50.899724 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:50.899697    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7dd98c21-e0b1-4289-8fe8-036bf413d444-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-q56x5\" (UID: \"7dd98c21-e0b1-4289-8fe8-036bf413d444\") " pod="openshift-insights/insights-runtime-extractor-q56x5"
Oct 31 01:03:50.899859 ip-10-0-133-197 kubenswrapper[2578]: I1031 01:03:50.899752    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7dd98c21-e0b1-4289-8fe8-036bf413d444-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-q56x5\" (UID: \"7dd98c21-e0b1-4289-8fe8-036bf413d444\") " pod="openshift-insights/insights-runtime-extractor-q56x5"
Oct 31 01:03:50.899859 ip-10-0-133-197 kubenswrapper[2578]: E1031 01:03:50.899818    2578 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Oct 31 01:03:50.899923 ip-10-0-133-197 kubenswrapper[2578]: E1031 01:03:50.899890    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/7dd98c21-e0b1-4289-8fe8-036bf413d444-kube-rbac-proxy-cm podName:7dd98c21-e0b1-4289-8fe8-036bf413d444 nodeName:}" failed. No retries permitted until 2026-10-31 01:03:54.899872631 +0000 UTC m=+46.585352036 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/7dd98c21-e0b1-4289-8fe8-036bf413d444-kube-rbac-proxy-cm") pod "insights-runtime-extractor-q56x5" (UID: "7dd98c21-e0b1-4289-8fe8-036bf413d444") : configmap "kube-rbac-proxy" not found
Oct 31 01:03:50.899923 ip-10-0-133-197 kubenswrapper[2578]: E1031 01:03:50.899892    2578 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 31 01:03:50.899991 ip-10-0-133-197 kubenswrapper[2578]: E1031 01:03:50.899938    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7dd98c21-e0b1-4289-8fe8-036bf413d444-insights-runtime-extractor-tls podName:7dd98c21-e0b1-4289-8fe8-036bf413d444 nodeName:}" failed. No retries permitted until 2026-10-31 01:03:54.899923682 +0000 UTC m=+46.585403093 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/7dd98c21-e0b1-4289-8fe8-036bf413d444-insights-runtime-extractor-tls") pod "insights-runtime-extractor-q56x5" (UID: "7dd98c21-e0b1-4289-8fe8-036bf413d444") : secret "insights-runtime-extractor-tls" not found
Oct 31 01:03:51.443312 ip-10-0-133-197 systemd[1]: Stopping Kubernetes Kubelet...
Oct 31 01:03:51.463925 ip-10-0-133-197 systemd[1]: kubelet.service: Deactivated successfully.
Oct 31 01:03:51.464081 ip-10-0-133-197 systemd[1]: Stopped Kubernetes Kubelet.
Oct 31 01:03:51.464243 ip-10-0-133-197 systemd[1]: kubelet.service: Consumed 3.054s CPU time.
Oct 31 01:03:51.477829 ip-10-0-133-197 systemd[1]: Starting Kubernetes Kubelet...
Oct 31 01:03:51.578794 ip-10-0-133-197 kubenswrapper[4964]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 31 01:03:51.578794 ip-10-0-133-197 kubenswrapper[4964]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Oct 31 01:03:51.578794 ip-10-0-133-197 kubenswrapper[4964]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 31 01:03:51.578794 ip-10-0-133-197 kubenswrapper[4964]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 31 01:03:51.581204 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.581126    4964 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 01:03:51.581204 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.581203    4964 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 01:03:51.581204 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.581208    4964 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 01:03:51.581447 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.581212    4964 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 01:03:51.581447 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.581216    4964 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 01:03:51.581447 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.581220    4964 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 01:03:51.581447 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.581224    4964 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 01:03:51.581447 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.581228    4964 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 01:03:51.581447 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.581231    4964 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 01:03:51.581447 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.581234    4964 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 01:03:51.581447 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.581238    4964 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 01:03:51.581447 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.581242    4964 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 01:03:51.581447 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.581246    4964 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 01:03:51.581447 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.581252    4964 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 01:03:51.581447 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.581257    4964 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 01:03:51.581447 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.581261    4964 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 01:03:51.581447 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.581265    4964 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 01:03:51.581447 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.581269    4964 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 01:03:51.581447 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.581272    4964 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 01:03:51.581447 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.581276    4964 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 01:03:51.581447 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.581287    4964 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 01:03:51.581447 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.581292    4964 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 01:03:51.582248 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.581295    4964 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 01:03:51.582248 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.581307    4964 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 01:03:51.582248 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.581311    4964 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 01:03:51.582248 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.581339    4964 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 01:03:51.582248 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.581343    4964 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 01:03:51.582248 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.581347    4964 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 01:03:51.582248 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.581351    4964 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 01:03:51.582248 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.581355    4964 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 01:03:51.582248 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.581358    4964 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 01:03:51.582248 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.581362    4964 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 01:03:51.582248 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.581366    4964 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 01:03:51.582248 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.581371    4964 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 01:03:51.582248 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.581382    4964 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 01:03:51.582248 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.581386    4964 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 01:03:51.582248 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.581391    4964 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 01:03:51.582248 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.581396    4964 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 01:03:51.582248 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.581400    4964 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 01:03:51.582248 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.581404    4964 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 01:03:51.582248 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.581409    4964 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 01:03:51.583086 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.581412    4964 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 01:03:51.583086 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.581417    4964 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 01:03:51.583086 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.581421    4964 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 01:03:51.583086 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.581425    4964 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 01:03:51.583086 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.581429    4964 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 01:03:51.583086 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.581433    4964 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 01:03:51.583086 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.581438    4964 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 01:03:51.583086 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.581442    4964 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 01:03:51.583086 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.581445    4964 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 01:03:51.583086 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.581449    4964 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 01:03:51.583086 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.581453    4964 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 01:03:51.583086 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.581458    4964 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 01:03:51.583086 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.581463    4964 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 01:03:51.583086 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.581467    4964 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 01:03:51.583086 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.581471    4964 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 01:03:51.583086 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.581475    4964 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 01:03:51.583086 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.581479    4964 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 01:03:51.583086 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.581486    4964 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 01:03:51.583086 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.581492    4964 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 01:03:51.583928 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.581498    4964 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 01:03:51.583928 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.581503    4964 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 01:03:51.583928 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.581508    4964 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 01:03:51.583928 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.581512    4964 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 01:03:51.583928 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.581517    4964 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 01:03:51.583928 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.581521    4964 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 01:03:51.583928 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.581525    4964 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 01:03:51.583928 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.581530    4964 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 01:03:51.583928 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.581535    4964 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 01:03:51.583928 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.581539    4964 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 01:03:51.583928 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.581543    4964 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 01:03:51.583928 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.581547    4964 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 01:03:51.583928 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.581552    4964 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 01:03:51.583928 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.581557    4964 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 01:03:51.583928 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.581561    4964 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 01:03:51.583928 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.581566    4964 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 01:03:51.583928 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.581570    4964 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 01:03:51.583928 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.581575    4964 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 01:03:51.583928 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.581579    4964 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 01:03:51.583928 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.581583    4964 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 01:03:51.585033 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.581587    4964 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 01:03:51.585033 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.581591    4964 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 01:03:51.585033 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.581595    4964 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 01:03:51.585033 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.581599    4964 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 01:03:51.585033 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.581604    4964 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 01:03:51.585033 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.581608    4964 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 01:03:51.585033 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.581613    4964 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 01:03:51.585033 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.581618    4964 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 01:03:51.585033 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.581622    4964 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 01:03:51.585033 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.581626    4964 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 01:03:51.585033 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.581630    4964 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 01:03:51.585033 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.581634    4964 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 01:03:51.585033 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.581640    4964 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 01:03:51.585033 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.581644    4964 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 01:03:51.585033 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.581649    4964 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 01:03:51.585033 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.581652    4964 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 01:03:51.585033 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.581657    4964 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 01:03:51.585033 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.581661    4964 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 01:03:51.585033 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.581665    4964 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 01:03:51.585033 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.581669    4964 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 01:03:51.585929 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.581676    4964 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 01:03:51.585929 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.581680    4964 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 01:03:51.585929 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.581684    4964 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 01:03:51.585929 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.581688    4964 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 01:03:51.585929 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.581692    4964 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 01:03:51.585929 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.581698    4964 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 01:03:51.585929 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.582925    4964 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 01:03:51.585929 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.582935    4964 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 01:03:51.585929 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.582940    4964 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 01:03:51.585929 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.582945    4964 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 01:03:51.585929 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.582949    4964 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 01:03:51.585929 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.582953    4964 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 01:03:51.585929 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.582957    4964 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 01:03:51.585929 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.582961    4964 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 01:03:51.585929 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.582968    4964 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 01:03:51.585929 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.582975    4964 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 01:03:51.585929 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.582980    4964 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 01:03:51.585929 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.582984    4964 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 01:03:51.585929 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.582988    4964 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 01:03:51.586824 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.582992    4964 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 01:03:51.586824 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.582996    4964 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 01:03:51.586824 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.583002    4964 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 01:03:51.586824 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.583008    4964 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 01:03:51.586824 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.583012    4964 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 01:03:51.586824 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.583016    4964 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 01:03:51.586824 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.583020    4964 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 01:03:51.586824 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.583024    4964 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 01:03:51.586824 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.583029    4964 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 01:03:51.586824 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.583033    4964 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 01:03:51.586824 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.583037    4964 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 01:03:51.586824 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.583041    4964 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 01:03:51.586824 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.583045    4964 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 01:03:51.586824 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.583050    4964 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 01:03:51.586824 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.583054    4964 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 01:03:51.586824 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.583058    4964 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 01:03:51.586824 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.583062    4964 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 01:03:51.586824 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.583066    4964 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 01:03:51.586824 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.583070    4964 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 01:03:51.587582 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.583075    4964 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 01:03:51.587582 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.583079    4964 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 01:03:51.587582 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.583083    4964 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 01:03:51.587582 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.583087    4964 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 01:03:51.587582 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.583091    4964 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 01:03:51.587582 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.583095    4964 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 01:03:51.587582 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.583099    4964 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 01:03:51.587582 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.583103    4964 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 01:03:51.587582 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.583106    4964 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 01:03:51.587582 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.583110    4964 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 01:03:51.587582 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.583115    4964 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 01:03:51.587582 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.583120    4964 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 01:03:51.587582 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.583124    4964 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 01:03:51.587582 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.583128    4964 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 01:03:51.587582 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.583132    4964 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 01:03:51.587582 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.583136    4964 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 01:03:51.587582 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.583140    4964 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 01:03:51.587582 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.583144    4964 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 01:03:51.587582 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.583147    4964 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 01:03:51.587582 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.583151    4964 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 01:03:51.588118 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.583155    4964 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 01:03:51.588118 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.583159    4964 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 01:03:51.588118 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.583163    4964 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 01:03:51.588118 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.583167    4964 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 01:03:51.588118 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.583171    4964 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 01:03:51.588118 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.583175    4964 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 01:03:51.588118 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.583179    4964 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 01:03:51.588118 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.583184    4964 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 01:03:51.588118 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.583188    4964 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 01:03:51.588118 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.583192    4964 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 01:03:51.588118 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.583196    4964 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 01:03:51.588118 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.583200    4964 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 01:03:51.588118 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.583204    4964 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 01:03:51.588118 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.583209    4964 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 01:03:51.588118 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.583213    4964 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 01:03:51.588118 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.583217    4964 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 01:03:51.588118 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.583221    4964 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 01:03:51.588118 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.583225    4964 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 01:03:51.588118 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.583229    4964 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 01:03:51.588118 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.583233    4964 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 01:03:51.588679 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.583238    4964 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 01:03:51.588679 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.583242    4964 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 01:03:51.588679 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.583249    4964 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 01:03:51.588679 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.583253    4964 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 01:03:51.588679 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.583257    4964 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 01:03:51.588679 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.583261    4964 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 01:03:51.588679 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.583266    4964 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 01:03:51.588679 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.583270    4964 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 01:03:51.588679 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.583274    4964 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 01:03:51.588679 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.583278    4964 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 01:03:51.588679 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.583282    4964 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 01:03:51.588679 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.583286    4964 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 01:03:51.588679 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.583290    4964 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 01:03:51.588679 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.583294    4964 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 01:03:51.588679 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.583299    4964 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 01:03:51.588679 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.583303    4964 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 01:03:51.588679 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.583307    4964 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 01:03:51.588679 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.583311    4964 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 01:03:51.588679 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.583332    4964 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 01:03:51.589133 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.583336    4964 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 01:03:51.589133 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.583340    4964 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 01:03:51.589133 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.583344    4964 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 01:03:51.589133 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.583348    4964 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 01:03:51.589133 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.583353    4964 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 01:03:51.589133 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.583357    4964 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 01:03:51.589133 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.583361    4964 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 01:03:51.589133 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.583366    4964 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 01:03:51.589133 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.583370    4964 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 01:03:51.589133 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.583374    4964 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 01:03:51.589133 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.583378    4964 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 01:03:51.589133 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.583382    4964 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 01:03:51.589133 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.583386    4964 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 01:03:51.589133 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.583390    4964 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 01:03:51.589133 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.583395    4964 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 01:03:51.589133 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.584191    4964 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 01:03:51.589133 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.584203    4964 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 01:03:51.589133 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.584208    4964 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 01:03:51.589133 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.584212    4964 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 01:03:51.589643 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.584216    4964 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 01:03:51.589643 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.584220    4964 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 01:03:51.589643 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.584224    4964 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 01:03:51.589643 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.584228    4964 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 01:03:51.589643 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.584232    4964 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 01:03:51.589643 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.584235    4964 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 01:03:51.589643 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.584240    4964 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 01:03:51.589643 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.584244    4964 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 01:03:51.589643 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.584247    4964 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 01:03:51.589643 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.584251    4964 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 01:03:51.589643 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.584255    4964 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 01:03:51.589643 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.584261    4964 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 01:03:51.589643 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.584266    4964 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 01:03:51.589643 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.584271    4964 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 01:03:51.589643 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.584275    4964 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 01:03:51.589643 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.584278    4964 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 01:03:51.589643 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.584282    4964 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 01:03:51.589643 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.584285    4964 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 01:03:51.589643 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.584289    4964 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 01:03:51.590083 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.584293    4964 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 01:03:51.590083 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.584297    4964 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 01:03:51.590083 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.584301    4964 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 01:03:51.590083 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.584305    4964 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 01:03:51.590083 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.584310    4964 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 01:03:51.590083 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.584342    4964 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 01:03:51.590083 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.584347    4964 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 01:03:51.590083 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.584351    4964 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 01:03:51.590083 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.584358    4964 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 01:03:51.590083 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.584364    4964 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 01:03:51.590083 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.584368    4964 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 01:03:51.590083 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.584372    4964 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 01:03:51.590083 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.584376    4964 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 01:03:51.590083 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.584380    4964 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 01:03:51.590083 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.584384    4964 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 01:03:51.590083 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.584389    4964 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 01:03:51.590083 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.584393    4964 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 01:03:51.590083 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.584397    4964 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 01:03:51.590083 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.584401    4964 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 01:03:51.590555 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.584406    4964 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 01:03:51.590555 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.584410    4964 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 01:03:51.590555 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.584414    4964 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 01:03:51.590555 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.584418    4964 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 01:03:51.590555 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.584422    4964 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 01:03:51.590555 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.584426    4964 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 01:03:51.590555 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.584430    4964 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 01:03:51.590555 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.584434    4964 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 01:03:51.590555 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.584438    4964 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 01:03:51.590555 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.584442    4964 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 01:03:51.590555 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.584446    4964 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 01:03:51.590555 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.584451    4964 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 01:03:51.590555 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.584456    4964 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 01:03:51.590555 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.584460    4964 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 01:03:51.590555 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.584464    4964 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 01:03:51.590555 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.584469    4964 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 01:03:51.590555 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.584473    4964 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 01:03:51.590555 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.584477    4964 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 01:03:51.590555 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.584482    4964 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 01:03:51.590555 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.584488    4964 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 01:03:51.591097 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.584492    4964 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 01:03:51.591097 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.584498    4964 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 01:03:51.591097 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.584504    4964 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 01:03:51.591097 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.584508    4964 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 01:03:51.591097 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.584512    4964 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 01:03:51.591097 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.584516    4964 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 01:03:51.591097 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.584520    4964 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 01:03:51.591097 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.584524    4964 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 01:03:51.591097 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.584528    4964 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 01:03:51.591097 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.584532    4964 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 01:03:51.591097 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.584536    4964 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 01:03:51.591097 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.584540    4964 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 01:03:51.591097 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.584544    4964 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 01:03:51.591097 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.584548    4964 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 01:03:51.591097 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.584552    4964 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 01:03:51.591097 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.584556    4964 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 01:03:51.591097 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.584560    4964 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 01:03:51.591097 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.584564    4964 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 01:03:51.591097 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.584568    4964 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 01:03:51.591097 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.584572    4964 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 01:03:51.591097 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.584576    4964 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 01:03:51.591611 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.584580    4964 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 01:03:51.591611 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.584584    4964 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 01:03:51.591611 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.584588    4964 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 01:03:51.591611 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.584593    4964 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 01:03:51.591611 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.584597    4964 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 01:03:51.591611 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.584601    4964 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 01:03:51.591611 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.584605    4964 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 01:03:51.591611 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.584610    4964 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 01:03:51.591611 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.584614    4964 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 01:03:51.591611 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.584618    4964 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 01:03:51.591611 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.584622    4964 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 01:03:51.591611 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.584626    4964 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 01:03:51.591611 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.584632    4964 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 01:03:51.591611 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.584638    4964 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 01:03:51.591611 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.584642    4964 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 01:03:51.591611 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.584646    4964 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 01:03:51.591611 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.584650    4964 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 01:03:51.591611 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.584654    4964 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 01:03:51.591611 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.584658    4964 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 01:03:51.592056 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.584666    4964 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 01:03:51.592056 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.584671    4964 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 01:03:51.592056 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.584676    4964 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 01:03:51.592056 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.584680    4964 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 01:03:51.592056 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:51.584820    4964 flags.go:64] FLAG: --address="0.0.0.0"
Oct 31 01:03:51.592056 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:51.584835    4964 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Oct 31 01:03:51.592056 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:51.584847    4964 flags.go:64] FLAG: --anonymous-auth="true"
Oct 31 01:03:51.592056 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:51.584854    4964 flags.go:64] FLAG: --application-metrics-count-limit="100"
Oct 31 01:03:51.592056 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:51.584860    4964 flags.go:64] FLAG: --authentication-token-webhook="false"
Oct 31 01:03:51.592056 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:51.584865    4964 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Oct 31 01:03:51.592056 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:51.584871    4964 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Oct 31 01:03:51.592056 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:51.584877    4964 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Oct 31 01:03:51.592056 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:51.584882    4964 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Oct 31 01:03:51.592056 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:51.584887    4964 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Oct 31 01:03:51.592056 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:51.584892    4964 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Oct 31 01:03:51.592056 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:51.584897    4964 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Oct 31 01:03:51.592056 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:51.584902    4964 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Oct 31 01:03:51.592056 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:51.584907    4964 flags.go:64] FLAG: --cgroup-root=""
Oct 31 01:03:51.592056 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:51.584911    4964 flags.go:64] FLAG: --cgroups-per-qos="true"
Oct 31 01:03:51.592056 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:51.584915    4964 flags.go:64] FLAG: --client-ca-file=""
Oct 31 01:03:51.592056 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:51.584920    4964 flags.go:64] FLAG: --cloud-provider="external"
Oct 31 01:03:51.592056 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:51.584925    4964 flags.go:64] FLAG: --cluster-dns="[]"
Oct 31 01:03:51.592056 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:51.584930    4964 flags.go:64] FLAG: --cluster-domain=""
Oct 31 01:03:51.592612 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:51.584935    4964 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Oct 31 01:03:51.592612 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:51.584940    4964 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Oct 31 01:03:51.592612 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:51.584946    4964 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Oct 31 01:03:51.592612 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:51.584953    4964 flags.go:64] FLAG: --container-log-max-files="5"
Oct 31 01:03:51.592612 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:51.584959    4964 flags.go:64] FLAG: --container-log-max-size="10Mi"
Oct 31 01:03:51.592612 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:51.584964    4964 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Oct 31 01:03:51.592612 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:51.584969    4964 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Oct 31 01:03:51.592612 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:51.584974    4964 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Oct 31 01:03:51.592612 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:51.584979    4964 flags.go:64] FLAG: --contention-profiling="false"
Oct 31 01:03:51.592612 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:51.584983    4964 flags.go:64] FLAG: --cpu-cfs-quota="true"
Oct 31 01:03:51.592612 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:51.584988    4964 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Oct 31 01:03:51.592612 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:51.584992    4964 flags.go:64] FLAG: --cpu-manager-policy="none"
Oct 31 01:03:51.592612 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:51.584997    4964 flags.go:64] FLAG: --cpu-manager-policy-options=""
Oct 31 01:03:51.592612 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:51.585004    4964 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Oct 31 01:03:51.592612 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:51.585009    4964 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Oct 31 01:03:51.592612 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:51.585013    4964 flags.go:64] FLAG: --enable-debugging-handlers="true"
Oct 31 01:03:51.592612 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:51.585018    4964 flags.go:64] FLAG: --enable-load-reader="false"
Oct 31 01:03:51.592612 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:51.585022    4964 flags.go:64] FLAG: --enable-server="true"
Oct 31 01:03:51.592612 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:51.585027    4964 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Oct 31 01:03:51.592612 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:51.585061    4964 flags.go:64] FLAG: --event-burst="100"
Oct 31 01:03:51.592612 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:51.585067    4964 flags.go:64] FLAG: --event-qps="50"
Oct 31 01:03:51.592612 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:51.585072    4964 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Oct 31 01:03:51.592612 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:51.585077    4964 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Oct 31 01:03:51.592612 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:51.585081    4964 flags.go:64] FLAG: --eviction-hard=""
Oct 31 01:03:51.593173 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:51.585087    4964 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Oct 31 01:03:51.593173 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:51.585092    4964 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Oct 31 01:03:51.593173 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:51.585097    4964 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Oct 31 01:03:51.593173 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:51.585102    4964 flags.go:64] FLAG: --eviction-soft=""
Oct 31 01:03:51.593173 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:51.585106    4964 flags.go:64] FLAG: --eviction-soft-grace-period=""
Oct 31 01:03:51.593173 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:51.585110    4964 flags.go:64] FLAG: --exit-on-lock-contention="false"
Oct 31 01:03:51.593173 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:51.585115    4964 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Oct 31 01:03:51.593173 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:51.585120    4964 flags.go:64] FLAG: --experimental-mounter-path=""
Oct 31 01:03:51.593173 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:51.585125    4964 flags.go:64] FLAG: --fail-cgroupv1="true"
Oct 31 01:03:51.593173 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:51.585129    4964 flags.go:64] FLAG: --fail-swap-on="true"
Oct 31 01:03:51.593173 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:51.585135    4964 flags.go:64] FLAG: --feature-gates=""
Oct 31 01:03:51.593173 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:51.585143    4964 flags.go:64] FLAG: --file-check-frequency="20s"
Oct 31 01:03:51.593173 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:51.585147    4964 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Oct 31 01:03:51.593173 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:51.585152    4964 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Oct 31 01:03:51.593173 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:51.585157    4964 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Oct 31 01:03:51.593173 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:51.585162    4964 flags.go:64] FLAG: --healthz-port="10248"
Oct 31 01:03:51.593173 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:51.585167    4964 flags.go:64] FLAG: --help="false"
Oct 31 01:03:51.593173 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:51.585171    4964 flags.go:64] FLAG: --hostname-override="ip-10-0-133-197.us-west-1.compute.internal"
Oct 31 01:03:51.593173 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:51.585176    4964 flags.go:64] FLAG: --housekeeping-interval="10s"
Oct 31 01:03:51.593173 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:51.585181    4964 flags.go:64] FLAG: --http-check-frequency="20s"
Oct 31 01:03:51.593173 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:51.585185    4964 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Oct 31 01:03:51.593173 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:51.585190    4964 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Oct 31 01:03:51.593173 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:51.585196    4964 flags.go:64] FLAG: --image-gc-high-threshold="85"
Oct 31 01:03:51.593747 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:51.585200    4964 flags.go:64] FLAG: --image-gc-low-threshold="80"
Oct 31 01:03:51.593747 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:51.585205    4964 flags.go:64] FLAG: --image-service-endpoint=""
Oct 31 01:03:51.593747 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:51.585209    4964 flags.go:64] FLAG: --kernel-memcg-notification="false"
Oct 31 01:03:51.593747 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:51.585214    4964 flags.go:64] FLAG: --kube-api-burst="100"
Oct 31 01:03:51.593747 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:51.585218    4964 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Oct 31 01:03:51.593747 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:51.585223    4964 flags.go:64] FLAG: --kube-api-qps="50"
Oct 31 01:03:51.593747 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:51.585227    4964 flags.go:64] FLAG: --kube-reserved=""
Oct 31 01:03:51.593747 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:51.585231    4964 flags.go:64] FLAG: --kube-reserved-cgroup=""
Oct 31 01:03:51.593747 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:51.585236    4964 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Oct 31 01:03:51.593747 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:51.585241    4964 flags.go:64] FLAG: --kubelet-cgroups=""
Oct 31 01:03:51.593747 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:51.585245    4964 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Oct 31 01:03:51.593747 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:51.585249    4964 flags.go:64] FLAG: --lock-file=""
Oct 31 01:03:51.593747 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:51.585254    4964 flags.go:64] FLAG: --log-cadvisor-usage="false"
Oct 31 01:03:51.593747 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:51.585258    4964 flags.go:64] FLAG: --log-flush-frequency="5s"
Oct 31 01:03:51.593747 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:51.585263    4964 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Oct 31 01:03:51.593747 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:51.585271    4964 flags.go:64] FLAG: --log-json-split-stream="false"
Oct 31 01:03:51.593747 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:51.585276    4964 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Oct 31 01:03:51.593747 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:51.585289    4964 flags.go:64] FLAG: --log-text-split-stream="false"
Oct 31 01:03:51.593747 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:51.585293    4964 flags.go:64] FLAG: --logging-format="text"
Oct 31 01:03:51.593747 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:51.585299    4964 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Oct 31 01:03:51.593747 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:51.585306    4964 flags.go:64] FLAG: --make-iptables-util-chains="true"
Oct 31 01:03:51.593747 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:51.585311    4964 flags.go:64] FLAG: --manifest-url=""
Oct 31 01:03:51.593747 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:51.585335    4964 flags.go:64] FLAG: --manifest-url-header=""
Oct 31 01:03:51.593747 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:51.585342    4964 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Oct 31 01:03:51.593747 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:51.585347    4964 flags.go:64] FLAG: --max-open-files="1000000"
Oct 31 01:03:51.594574 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:51.585352    4964 flags.go:64] FLAG: --max-pods="110"
Oct 31 01:03:51.594574 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:51.585357    4964 flags.go:64] FLAG: --maximum-dead-containers="-1"
Oct 31 01:03:51.594574 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:51.585362    4964 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Oct 31 01:03:51.594574 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:51.585366    4964 flags.go:64] FLAG: --memory-manager-policy="None"
Oct 31 01:03:51.594574 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:51.585371    4964 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Oct 31 01:03:51.594574 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:51.585375    4964 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Oct 31 01:03:51.594574 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:51.585380    4964 flags.go:64] FLAG: --node-ip="0.0.0.0"
Oct 31 01:03:51.594574 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:51.585385    4964 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Oct 31 01:03:51.594574 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:51.585395    4964 flags.go:64] FLAG: --node-status-max-images="50"
Oct 31 01:03:51.594574 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:51.585400    4964 flags.go:64] FLAG: --node-status-update-frequency="10s"
Oct 31 01:03:51.594574 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:51.585405    4964 flags.go:64] FLAG: --oom-score-adj="-999"
Oct 31 01:03:51.594574 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:51.585409    4964 flags.go:64] FLAG: --pod-cidr=""
Oct 31 01:03:51.594574 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:51.585414    4964 flags.go:64] FLAG: --pod-manifest-path=""
Oct 31 01:03:51.594574 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:51.585418    4964 flags.go:64] FLAG: --pod-max-pids="-1"
Oct 31 01:03:51.594574 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:51.585422    4964 flags.go:64] FLAG: --pods-per-core="0"
Oct 31 01:03:51.594574 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:51.585427    4964 flags.go:64] FLAG: --port="10250"
Oct 31 01:03:51.594574 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:51.585431    4964 flags.go:64] FLAG: --protect-kernel-defaults="false"
Oct 31 01:03:51.594574 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:51.585436    4964 flags.go:64] FLAG: --provider-id="aws:///us-west-1a/i-0540feba47811ea65"
Oct 31 01:03:51.594574 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:51.585440    4964 flags.go:64] FLAG: --qos-reserved=""
Oct 31 01:03:51.594574 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:51.585445    4964 flags.go:64] FLAG: --read-only-port="10255"
Oct 31 01:03:51.594574 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:51.585450    4964 flags.go:64] FLAG: --register-node="true"
Oct 31 01:03:51.594574 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:51.585454    4964 flags.go:64] FLAG: --register-with-taints=""
Oct 31 01:03:51.594574 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:51.585460    4964 flags.go:64] FLAG: --registry-burst="10"
Oct 31 01:03:51.594574 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:51.585465    4964 flags.go:64] FLAG: --registry-qps="5"
Oct 31 01:03:51.594574 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:51.585469    4964 flags.go:64] FLAG: --reserved-cpus=""
Oct 31 01:03:51.594574 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:51.585474    4964 flags.go:64] FLAG: --reserved-memory=""
Oct 31 01:03:51.595553 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:51.585483    4964 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Oct 31 01:03:51.595553 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:51.585489    4964 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Oct 31 01:03:51.595553 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:51.585494    4964 flags.go:64] FLAG: --rotate-certificates="false"
Oct 31 01:03:51.595553 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:51.585498    4964 flags.go:64] FLAG: --rotate-server-certificates="false"
Oct 31 01:03:51.595553 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:51.585503    4964 flags.go:64] FLAG: --runonce="false"
Oct 31 01:03:51.595553 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:51.585507    4964 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Oct 31 01:03:51.595553 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:51.585512    4964 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Oct 31 01:03:51.595553 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:51.585517    4964 flags.go:64] FLAG: --seccomp-default="false"
Oct 31 01:03:51.595553 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:51.585521    4964 flags.go:64] FLAG: --serialize-image-pulls="true"
Oct 31 01:03:51.595553 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:51.585526    4964 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Oct 31 01:03:51.595553 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:51.585530    4964 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Oct 31 01:03:51.595553 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:51.585535    4964 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Oct 31 01:03:51.595553 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:51.585540    4964 flags.go:64] FLAG: --storage-driver-password="root"
Oct 31 01:03:51.595553 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:51.585544    4964 flags.go:64] FLAG: --storage-driver-secure="false"
Oct 31 01:03:51.595553 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:51.585548    4964 flags.go:64] FLAG: --storage-driver-table="stats"
Oct 31 01:03:51.595553 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:51.585552    4964 flags.go:64] FLAG: --storage-driver-user="root"
Oct 31 01:03:51.595553 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:51.585557    4964 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Oct 31 01:03:51.595553 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:51.585562    4964 flags.go:64] FLAG: --sync-frequency="1m0s"
Oct 31 01:03:51.595553 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:51.585566    4964 flags.go:64] FLAG: --system-cgroups=""
Oct 31 01:03:51.595553 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:51.585571    4964 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Oct 31 01:03:51.595553 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:51.585578    4964 flags.go:64] FLAG: --system-reserved-cgroup=""
Oct 31 01:03:51.595553 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:51.585583    4964 flags.go:64] FLAG: --tls-cert-file=""
Oct 31 01:03:51.595553 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:51.585587    4964 flags.go:64] FLAG: --tls-cipher-suites="[]"
Oct 31 01:03:51.595553 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:51.585593    4964 flags.go:64] FLAG: --tls-min-version=""
Oct 31 01:03:51.595553 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:51.585597    4964 flags.go:64] FLAG: --tls-private-key-file=""
Oct 31 01:03:51.596328 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:51.585602    4964 flags.go:64] FLAG: --topology-manager-policy="none"
Oct 31 01:03:51.596328 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:51.585606    4964 flags.go:64] FLAG: --topology-manager-policy-options=""
Oct 31 01:03:51.596328 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:51.585611    4964 flags.go:64] FLAG: --topology-manager-scope="container"
Oct 31 01:03:51.596328 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:51.585615    4964 flags.go:64] FLAG: --v="2"
Oct 31 01:03:51.596328 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:51.585621    4964 flags.go:64] FLAG: --version="false"
Oct 31 01:03:51.596328 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:51.585628    4964 flags.go:64] FLAG: --vmodule=""
Oct 31 01:03:51.596328 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:51.585633    4964 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Oct 31 01:03:51.596328 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:51.585640    4964 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Oct 31 01:03:51.596328 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.585840    4964 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 01:03:51.596328 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.585846    4964 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 01:03:51.596328 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.585851    4964 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 01:03:51.596328 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.585856    4964 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 01:03:51.596328 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.585860    4964 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 01:03:51.596328 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.585864    4964 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 01:03:51.596328 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.585869    4964 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 01:03:51.596328 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.585874    4964 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 01:03:51.596328 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.585878    4964 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 01:03:51.596328 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.585882    4964 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 01:03:51.596328 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.585886    4964 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 01:03:51.596328 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.585890    4964 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 01:03:51.596328 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.585894    4964 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 01:03:51.596980 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.585898    4964 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 01:03:51.596980 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.585902    4964 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 01:03:51.596980 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.585906    4964 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 01:03:51.596980 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.585911    4964 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 01:03:51.596980 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.585915    4964 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 01:03:51.596980 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.585919    4964 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 01:03:51.596980 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.585923    4964 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 01:03:51.596980 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.585927    4964 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 01:03:51.596980 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.585931    4964 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 01:03:51.596980 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.585935    4964 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 01:03:51.596980 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.585939    4964 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 01:03:51.596980 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.585943    4964 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 01:03:51.596980 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.585947    4964 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 01:03:51.596980 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.585951    4964 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 01:03:51.596980 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.585955    4964 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 01:03:51.596980 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.585959    4964 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 01:03:51.596980 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.585963    4964 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 01:03:51.596980 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.585968    4964 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 01:03:51.596980 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.585973    4964 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 01:03:51.596980 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.585979    4964 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 01:03:51.597679 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.585983    4964 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 01:03:51.597679 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.585988    4964 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 01:03:51.597679 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.585992    4964 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 01:03:51.597679 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.585996    4964 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 01:03:51.597679 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.586000    4964 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 01:03:51.597679 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.586004    4964 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 01:03:51.597679 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.586008    4964 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 01:03:51.597679 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.586012    4964 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 01:03:51.597679 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.586016    4964 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 01:03:51.597679 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.586020    4964 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 01:03:51.597679 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.586024    4964 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 01:03:51.597679 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.586028    4964 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 01:03:51.597679 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.586032    4964 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 01:03:51.597679 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.586039    4964 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 01:03:51.597679 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.586045    4964 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 01:03:51.597679 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.586049    4964 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 01:03:51.597679 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.586054    4964 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 01:03:51.597679 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.586058    4964 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 01:03:51.597679 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.586062    4964 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 01:03:51.598499 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.586066    4964 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 01:03:51.598499 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.586070    4964 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 01:03:51.598499 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.586074    4964 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 01:03:51.598499 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.586078    4964 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 01:03:51.598499 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.586082    4964 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 01:03:51.598499 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.586086    4964 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 01:03:51.598499 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.586090    4964 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 01:03:51.598499 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.586095    4964 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 01:03:51.598499 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.586100    4964 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 01:03:51.598499 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.586104    4964 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 01:03:51.598499 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.586108    4964 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 01:03:51.598499 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.586114    4964 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 01:03:51.598499 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.586119    4964 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 01:03:51.598499 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.586124    4964 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 01:03:51.598499 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.586128    4964 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 01:03:51.598499 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.586132    4964 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 01:03:51.598499 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.586136    4964 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 01:03:51.598499 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.586140    4964 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 01:03:51.598499 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.586144    4964 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 01:03:51.598499 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.586148    4964 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 01:03:51.599157 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.586152    4964 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 01:03:51.599157 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.586156    4964 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 01:03:51.599157 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.586161    4964 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 01:03:51.599157 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.586165    4964 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 01:03:51.599157 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.586169    4964 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 01:03:51.599157 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.586173    4964 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 01:03:51.599157 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.586177    4964 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 01:03:51.599157 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.586181    4964 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 01:03:51.599157 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.586185    4964 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 01:03:51.599157 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.586190    4964 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 01:03:51.599157 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.586194    4964 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 01:03:51.599157 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.586198    4964 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 01:03:51.599157 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.586202    4964 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 01:03:51.599157 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.586206    4964 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 01:03:51.599157 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.586212    4964 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 01:03:51.599157 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.586219    4964 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 01:03:51.599157 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.586224    4964 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 01:03:51.599157 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.586229    4964 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 01:03:51.599157 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.586234    4964 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 01:03:51.599972 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.586238    4964 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 01:03:51.599972 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.586242    4964 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 01:03:51.599972 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.586246    4964 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 01:03:51.599972 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.586250    4964 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 01:03:51.599972 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.586256    4964 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 01:03:51.599972 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.586260    4964 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 01:03:51.599972 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.586264    4964 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 01:03:51.599972 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.586268    4964 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 01:03:51.599972 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.586273    4964 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 01:03:51.599972 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.586277    4964 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 01:03:51.599972 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.586281    4964 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 01:03:51.599972 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.586285    4964 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 01:03:51.599972 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.586289    4964 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 01:03:51.599972 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.586293    4964 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 01:03:51.599972 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.586297    4964 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 01:03:51.599972 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:51.586360    4964 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 31 01:03:51.600516 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.586411    4964 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 01:03:51.600516 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.586417    4964 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 01:03:51.600516 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.586421    4964 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 01:03:51.600516 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.586426    4964 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 01:03:51.600516 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.586430    4964 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 01:03:51.600516 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.586435    4964 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 01:03:51.600516 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.586439    4964 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 01:03:51.600516 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.586443    4964 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 01:03:51.600516 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.586447    4964 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 01:03:51.600516 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.586451    4964 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 01:03:51.600516 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.586455    4964 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 01:03:51.600516 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.586459    4964 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 01:03:51.600516 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.586463    4964 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 01:03:51.600516 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.586467    4964 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 01:03:51.600516 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.586471    4964 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 01:03:51.600516 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.586478    4964 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 01:03:51.600516 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.586483    4964 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 01:03:51.600516 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.586488    4964 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 01:03:51.600516 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.586493    4964 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 01:03:51.601206 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.586498    4964 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 01:03:51.601206 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.586503    4964 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 01:03:51.601206 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.586510    4964 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 01:03:51.601206 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.586514    4964 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 01:03:51.601206 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.586519    4964 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 01:03:51.601206 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.586525    4964 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 01:03:51.601206 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.586529    4964 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 01:03:51.601206 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.586533    4964 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 01:03:51.601206 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.586537    4964 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 01:03:51.601206 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.586541    4964 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 01:03:51.601206 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.586545    4964 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 01:03:51.601206 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.586549    4964 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 01:03:51.601206 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.586554    4964 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 01:03:51.601206 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.586558    4964 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 01:03:51.601206 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.586562    4964 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 01:03:51.601206 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.586566    4964 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 01:03:51.601206 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.586570    4964 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 01:03:51.601206 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.586574    4964 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 01:03:51.601206 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.586580    4964 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 01:03:51.601206 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.586585    4964 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 01:03:51.601908 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.586589    4964 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 01:03:51.601908 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.586595    4964 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 01:03:51.601908 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.586599    4964 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 01:03:51.601908 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.586603    4964 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 01:03:51.601908 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.586607    4964 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 01:03:51.601908 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.586612    4964 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 01:03:51.601908 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.586616    4964 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 01:03:51.601908 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.586620    4964 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 01:03:51.601908 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.586625    4964 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 01:03:51.601908 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.586629    4964 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 01:03:51.601908 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.586633    4964 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 01:03:51.601908 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.586637    4964 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 01:03:51.601908 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.586642    4964 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 01:03:51.601908 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.586646    4964 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 01:03:51.601908 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.586652    4964 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 01:03:51.601908 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.586657    4964 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 01:03:51.601908 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.586661    4964 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 01:03:51.601908 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.586665    4964 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 01:03:51.601908 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.586671    4964 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 01:03:51.601908 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.586676    4964 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 01:03:51.602895 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.586680    4964 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 01:03:51.602895 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.586685    4964 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 01:03:51.602895 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.586689    4964 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 01:03:51.602895 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.586693    4964 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 01:03:51.602895 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.586697    4964 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 01:03:51.602895 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.586701    4964 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 01:03:51.602895 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.586705    4964 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 01:03:51.602895 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.586709    4964 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 01:03:51.602895 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.586714    4964 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 01:03:51.602895 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.586718    4964 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 01:03:51.602895 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.586724    4964 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 01:03:51.602895 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.586728    4964 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 01:03:51.602895 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.586732    4964 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 01:03:51.602895 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.586736    4964 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 01:03:51.602895 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.586740    4964 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 01:03:51.602895 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.586744    4964 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 01:03:51.602895 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.586749    4964 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 01:03:51.602895 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.586753    4964 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 01:03:51.602895 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.586757    4964 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 01:03:51.603620 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.586761    4964 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 01:03:51.603620 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.586765    4964 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 01:03:51.603620 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.586769    4964 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 01:03:51.603620 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.586775    4964 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 01:03:51.603620 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.586781    4964 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 01:03:51.603620 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.586786    4964 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 01:03:51.603620 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.586791    4964 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 01:03:51.603620 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.586796    4964 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 01:03:51.603620 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.586801    4964 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 01:03:51.603620 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.586805    4964 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 01:03:51.603620 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.586809    4964 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 01:03:51.603620 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.586813    4964 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 01:03:51.603620 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.586818    4964 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 01:03:51.603620 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.586822    4964 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 01:03:51.603620 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.586827    4964 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 01:03:51.603620 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.586831    4964 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 01:03:51.603620 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.586835    4964 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 01:03:51.603620 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.586839    4964 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 01:03:51.603620 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.586843    4964 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 01:03:51.604255 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.586846    4964 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 01:03:51.604255 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.586850    4964 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 01:03:51.604255 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.586855    4964 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 01:03:51.604255 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.586859    4964 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 01:03:51.604255 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.586864    4964 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 01:03:51.604255 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.586869    4964 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 01:03:51.604255 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.586873    4964 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 01:03:51.604255 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.586877    4964 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 01:03:51.604255 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.586881    4964 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 01:03:51.604255 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:51.592670    4964 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Oct 31 01:03:51.604255 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:51.592690    4964 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Oct 31 01:03:51.604255 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:51.592706    4964 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Oct 31 01:03:51.604255 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:51.592711    4964 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Oct 31 01:03:51.604255 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.592768    4964 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 01:03:51.604255 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.592773    4964 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 01:03:51.604255 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.592783    4964 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 01:03:51.604255 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.592786    4964 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 01:03:51.604255 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.592789    4964 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 01:03:51.604255 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.592792    4964 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 01:03:51.604255 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.592794    4964 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 01:03:51.604811 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.592797    4964 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 01:03:51.604811 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.592799    4964 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 01:03:51.604811 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.592802    4964 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 01:03:51.604811 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.592805    4964 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 01:03:51.604811 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.592809    4964 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 01:03:51.604811 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.592813    4964 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 01:03:51.604811 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.592816    4964 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 01:03:51.604811 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.592819    4964 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 01:03:51.604811 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.592821    4964 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 01:03:51.604811 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.592824    4964 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 01:03:51.604811 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.592827    4964 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 01:03:51.604811 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.592829    4964 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 01:03:51.604811 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.592832    4964 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 01:03:51.604811 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.592834    4964 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 01:03:51.604811 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.592837    4964 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 01:03:51.604811 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.592839    4964 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 01:03:51.604811 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.592842    4964 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 01:03:51.604811 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.592845    4964 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 01:03:51.604811 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.592847    4964 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 01:03:51.605394 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.592850    4964 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 01:03:51.605394 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.592852    4964 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 01:03:51.605394 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.592854    4964 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 01:03:51.605394 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.592862    4964 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 01:03:51.605394 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.592865    4964 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 01:03:51.605394 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.592867    4964 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 01:03:51.605394 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.592870    4964 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 01:03:51.605394 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.592872    4964 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 01:03:51.605394 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.592875    4964 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 01:03:51.605394 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.592877    4964 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 01:03:51.605394 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.592880    4964 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 01:03:51.605394 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.592882    4964 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 01:03:51.605394 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.592885    4964 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 01:03:51.605394 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.592888    4964 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 01:03:51.605394 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.592890    4964 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 01:03:51.605394 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.592893    4964 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 01:03:51.605394 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.592896    4964 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 01:03:51.605394 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.592898    4964 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 01:03:51.605394 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.592901    4964 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 01:03:51.605970 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.592903    4964 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 01:03:51.605970 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.592906    4964 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 01:03:51.605970 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.592909    4964 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 01:03:51.605970 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.592911    4964 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 01:03:51.605970 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.592914    4964 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 01:03:51.605970 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.592916    4964 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 01:03:51.605970 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.592918    4964 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 01:03:51.605970 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.592921    4964 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 01:03:51.605970 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.592923    4964 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 01:03:51.605970 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.592926    4964 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 01:03:51.605970 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.592929    4964 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 01:03:51.605970 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.592931    4964 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 01:03:51.605970 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.592934    4964 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 01:03:51.605970 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.592936    4964 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 01:03:51.605970 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.592939    4964 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 01:03:51.605970 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.592941    4964 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 01:03:51.605970 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.592943    4964 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 01:03:51.605970 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.592951    4964 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 01:03:51.605970 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.592954    4964 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 01:03:51.605970 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.592957    4964 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 01:03:51.606666 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.592959    4964 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 01:03:51.606666 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.592962    4964 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 01:03:51.606666 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.592964    4964 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 01:03:51.606666 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.592967    4964 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 01:03:51.606666 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.592969    4964 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 01:03:51.606666 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.592972    4964 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 01:03:51.606666 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.592974    4964 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 01:03:51.606666 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.592977    4964 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 01:03:51.606666 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.592979    4964 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 01:03:51.606666 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.592982    4964 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 01:03:51.606666 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.592984    4964 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 01:03:51.606666 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.592987    4964 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 01:03:51.606666 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.592989    4964 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 01:03:51.606666 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.592991    4964 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 01:03:51.606666 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.592994    4964 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 01:03:51.606666 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.592997    4964 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 01:03:51.606666 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.592999    4964 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 01:03:51.606666 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.593002    4964 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 01:03:51.606666 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.593004    4964 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 01:03:51.606666 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.593006    4964 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 01:03:51.607281 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.593009    4964 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 01:03:51.607281 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.593011    4964 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 01:03:51.607281 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.593013    4964 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 01:03:51.607281 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.593016    4964 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 01:03:51.607281 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.593019    4964 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 01:03:51.607281 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.593021    4964 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 01:03:51.607281 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.593024    4964 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 01:03:51.607281 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.593026    4964 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 01:03:51.607281 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.593029    4964 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 01:03:51.607281 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.593031    4964 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 01:03:51.607281 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.593039    4964 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 01:03:51.607281 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.593042    4964 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 01:03:51.607281 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.593044    4964 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 01:03:51.607281 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.593046    4964 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 01:03:51.607281 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.593049    4964 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 01:03:51.607281 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.593051    4964 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 01:03:51.607281 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.593054    4964 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 01:03:51.607281 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.593056    4964 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 01:03:51.607281 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.593059    4964 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 01:03:51.607281 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.593063    4964 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 01:03:51.607896 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.593066    4964 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 01:03:51.607896 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:51.593093    4964 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 31 01:03:51.607896 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.593276    4964 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 01:03:51.607896 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.593292    4964 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 01:03:51.607896 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.593295    4964 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 01:03:51.607896 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.593298    4964 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 01:03:51.607896 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.593301    4964 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 01:03:51.607896 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.593304    4964 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 01:03:51.607896 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.593306    4964 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 01:03:51.607896 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.593309    4964 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 01:03:51.607896 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.593312    4964 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 01:03:51.607896 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.593331    4964 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 01:03:51.607896 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.593335    4964 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 01:03:51.607896 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.593338    4964 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 01:03:51.607896 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.593343    4964 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 01:03:51.607896 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.593347    4964 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 01:03:51.608295 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.593351    4964 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 01:03:51.608295 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.593355    4964 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 01:03:51.608295 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.593359    4964 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 01:03:51.608295 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.593362    4964 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 01:03:51.608295 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.593364    4964 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 01:03:51.608295 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.593367    4964 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 01:03:51.608295 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.593369    4964 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 01:03:51.608295 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.593380    4964 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 01:03:51.608295 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.593382    4964 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 01:03:51.608295 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.593385    4964 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 01:03:51.608295 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.593387    4964 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 01:03:51.608295 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.593389    4964 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 01:03:51.608295 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.593392    4964 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 01:03:51.608295 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.593394    4964 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 01:03:51.608295 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.593397    4964 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 01:03:51.608295 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.593400    4964 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 01:03:51.608295 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.593402    4964 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 01:03:51.608295 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.593405    4964 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 01:03:51.608295 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.593407    4964 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 01:03:51.608876 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.593410    4964 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 01:03:51.608876 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.593412    4964 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 01:03:51.608876 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.593429    4964 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 01:03:51.608876 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.593432    4964 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 01:03:51.608876 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.593436    4964 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 01:03:51.608876 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.593441    4964 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 01:03:51.608876 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.593445    4964 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 01:03:51.608876 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.593448    4964 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 01:03:51.608876 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.593451    4964 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 01:03:51.608876 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.593454    4964 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 01:03:51.608876 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.593456    4964 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 01:03:51.608876 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.593459    4964 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 01:03:51.608876 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.593461    4964 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 01:03:51.608876 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.593464    4964 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 01:03:51.608876 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.593467    4964 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 01:03:51.608876 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.593469    4964 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 01:03:51.608876 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.593472    4964 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 01:03:51.608876 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.593474    4964 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 01:03:51.608876 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.593477    4964 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 01:03:51.609417 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.593479    4964 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 01:03:51.609417 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.593481    4964 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 01:03:51.609417 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.593492    4964 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 01:03:51.609417 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.593495    4964 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 01:03:51.609417 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.593498    4964 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 01:03:51.609417 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.593500    4964 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 01:03:51.609417 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.593503    4964 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 01:03:51.609417 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.593505    4964 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 01:03:51.609417 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.593508    4964 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 01:03:51.609417 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.593510    4964 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 01:03:51.609417 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.593512    4964 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 01:03:51.609417 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.593515    4964 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 01:03:51.609417 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.593518    4964 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 01:03:51.609417 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.593520    4964 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 01:03:51.609417 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.593523    4964 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 01:03:51.609417 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.593525    4964 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 01:03:51.609417 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.593527    4964 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 01:03:51.609417 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.593530    4964 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 01:03:51.609417 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.593532    4964 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 01:03:51.609417 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.593535    4964 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 01:03:51.610060 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.593537    4964 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 01:03:51.610060 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.593540    4964 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 01:03:51.610060 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.593542    4964 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 01:03:51.610060 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.593544    4964 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 01:03:51.610060 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.593547    4964 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 01:03:51.610060 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.593549    4964 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 01:03:51.610060 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.593552    4964 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 01:03:51.610060 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.593554    4964 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 01:03:51.610060 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.593557    4964 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 01:03:51.610060 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.593560    4964 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 01:03:51.610060 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.593563    4964 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 01:03:51.610060 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.593567    4964 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 01:03:51.610060 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.593569    4964 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 01:03:51.610060 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.593572    4964 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 01:03:51.610060 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.593574    4964 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 01:03:51.610060 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.593582    4964 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 01:03:51.610060 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.593585    4964 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 01:03:51.610060 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.593587    4964 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 01:03:51.610060 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.593590    4964 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 01:03:51.610060 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.593592    4964 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 01:03:51.610806 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.593595    4964 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 01:03:51.610806 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.593597    4964 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 01:03:51.610806 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.593599    4964 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 01:03:51.610806 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.593602    4964 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 01:03:51.610806 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.593605    4964 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 01:03:51.610806 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.593607    4964 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 01:03:51.610806 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.593610    4964 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 01:03:51.610806 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.593612    4964 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 01:03:51.610806 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.593615    4964 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 01:03:51.610806 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.593617    4964 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 01:03:51.610806 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.593620    4964 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 01:03:51.610806 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.593622    4964 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 01:03:51.610806 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.593625    4964 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 01:03:51.610806 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.593627    4964 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 01:03:51.610806 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:51.593651    4964 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 31 01:03:51.610806 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.593689    4964 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 01:03:51.611365 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.593693    4964 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 01:03:51.611365 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.593696    4964 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 01:03:51.611365 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.593699    4964 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 01:03:51.611365 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.593701    4964 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 01:03:51.611365 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.593703    4964 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 01:03:51.611365 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.593706    4964 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 01:03:51.611365 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.593708    4964 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 01:03:51.611365 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.593711    4964 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 01:03:51.611365 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.593713    4964 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 01:03:51.611365 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.593716    4964 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 01:03:51.611365 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.593719    4964 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 01:03:51.611365 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.593723    4964 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 01:03:51.611365 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.593737    4964 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 01:03:51.611365 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.593740    4964 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 01:03:51.611365 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.593743    4964 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 01:03:51.611365 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.593746    4964 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 01:03:51.611365 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.593749    4964 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 01:03:51.611365 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.593751    4964 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 01:03:51.611365 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.593754    4964 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 01:03:51.611890 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.593757    4964 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 01:03:51.611890 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.593759    4964 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 01:03:51.611890 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.593762    4964 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 01:03:51.611890 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.593764    4964 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 01:03:51.611890 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.593767    4964 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 01:03:51.611890 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.593769    4964 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 01:03:51.611890 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.593771    4964 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 01:03:51.611890 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.593774    4964 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 01:03:51.611890 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.593776    4964 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 01:03:51.611890 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.593779    4964 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 01:03:51.611890 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.593781    4964 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 01:03:51.611890 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.593783    4964 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 01:03:51.611890 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.593786    4964 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 01:03:51.611890 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.593788    4964 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 01:03:51.611890 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.593791    4964 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 01:03:51.611890 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.593793    4964 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 01:03:51.611890 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.593796    4964 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 01:03:51.611890 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.593798    4964 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 01:03:51.611890 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.593801    4964 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 01:03:51.611890 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.593803    4964 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 01:03:51.612583 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.593806    4964 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 01:03:51.612583 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.593808    4964 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 01:03:51.612583 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.593810    4964 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 01:03:51.612583 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.593813    4964 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 01:03:51.612583 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.593815    4964 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 01:03:51.612583 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.593817    4964 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 01:03:51.612583 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.593826    4964 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 01:03:51.612583 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.593829    4964 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 01:03:51.612583 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.593832    4964 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 01:03:51.612583 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.593834    4964 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 01:03:51.612583 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.593836    4964 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 01:03:51.612583 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.593839    4964 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 01:03:51.612583 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.593841    4964 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 01:03:51.612583 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.593844    4964 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 01:03:51.612583 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.593846    4964 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 01:03:51.612583 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.593849    4964 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 01:03:51.612583 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.593853    4964 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 01:03:51.612583 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.593856    4964 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 01:03:51.612583 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.593858    4964 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 01:03:51.612583 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.593861    4964 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 01:03:51.613198 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.593864    4964 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 01:03:51.613198 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.593866    4964 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 01:03:51.613198 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.593868    4964 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 01:03:51.613198 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.593871    4964 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 01:03:51.613198 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.593874    4964 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 01:03:51.613198 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.593876    4964 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 01:03:51.613198 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.593878    4964 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 01:03:51.613198 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.593881    4964 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 01:03:51.613198 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.593883    4964 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 01:03:51.613198 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.593886    4964 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 01:03:51.613198 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.593888    4964 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 01:03:51.613198 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.593891    4964 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 01:03:51.613198 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.593893    4964 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 01:03:51.613198 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.593895    4964 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 01:03:51.613198 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.593898    4964 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 01:03:51.613198 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.593900    4964 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 01:03:51.613198 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.593903    4964 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 01:03:51.613198 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.593905    4964 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 01:03:51.613198 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.593908    4964 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 01:03:51.614028 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.593915    4964 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 01:03:51.614028 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.593918    4964 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 01:03:51.614028 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.593920    4964 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 01:03:51.614028 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.593922    4964 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 01:03:51.614028 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.593925    4964 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 01:03:51.614028 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.593928    4964 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 01:03:51.614028 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.593930    4964 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 01:03:51.614028 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.593932    4964 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 01:03:51.614028 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.593935    4964 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 01:03:51.614028 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.593938    4964 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 01:03:51.614028 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.593940    4964 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 01:03:51.614028 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.593943    4964 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 01:03:51.614028 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.593945    4964 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 01:03:51.614028 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.593947    4964 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 01:03:51.614028 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.593950    4964 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 01:03:51.614028 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.593952    4964 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 01:03:51.614028 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.593954    4964 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 01:03:51.614028 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.593957    4964 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 01:03:51.614028 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.593959    4964 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 01:03:51.614028 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.593962    4964 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 01:03:51.614686 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.593964    4964 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 01:03:51.614686 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.593967    4964 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 01:03:51.614686 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.593969    4964 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 01:03:51.614686 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.593983    4964 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 01:03:51.614686 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.593987    4964 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 01:03:51.614686 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.593990    4964 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 01:03:51.614686 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:51.593993    4964 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 01:03:51.614686 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:51.594050    4964 server.go:957] "Client rotation is on, will bootstrap in background"
Oct 31 01:03:51.614686 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:51.596825    4964 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary"
Oct 31 01:03:51.614686 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:51.596892    4964 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem"
Oct 31 01:03:51.614686 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:51.597106    4964 server.go:1014] "Starting client certificate rotation"
Oct 31 01:03:51.614686 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:51.597284    4964 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Oct 31 01:03:51.614686 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:51.597368    4964 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-30 00:58:09 +0000 UTC" deadline="2028-05-10 06:18:10.924900611 +0000 UTC"
Oct 31 01:03:51.614686 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:51.597423    4964 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13373h14m19.32749724s"
Oct 31 01:03:51.614686 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:51.597776    4964 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 31 01:03:51.615134 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:51.599265    4964 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 31 01:03:51.615134 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:51.601199    4964 log.go:25] "Validated CRI v1 runtime API"
Oct 31 01:03:51.615134 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:51.603462    4964 log.go:25] "Validated CRI v1 image API"
Oct 31 01:03:51.615134 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:51.604433    4964 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Oct 31 01:03:51.615134 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:51.607510    4964 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 894c0bb4-949e-41b2-a0ad-fe010d3501de:/dev/nvme0n1p3 f763df3b-a4a8-4bc9-bc11-a55871c3415d:/dev/nvme0n1p4]
Oct 31 01:03:51.615328 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:51.607540    4964 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/073fc7fbb6f5887bd5ad9f67b0ea840f764095bda44fa96f2e5959f1d52f73d5/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/073fc7fbb6f5887bd5ad9f67b0ea840f764095bda44fa96f2e5959f1d52f73d5/userdata/shm major:0 minor:240 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/075a77de8990101732600f618149824f4c9470f0e266030d9b437c8c27451e87/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/075a77de8990101732600f618149824f4c9470f0e266030d9b437c8c27451e87/userdata/shm major:0 minor:219 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/0b3559dbcb56809941ca496ac90b08c271ef0b225c141267043d900f6e7d7564/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/0b3559dbcb56809941ca496ac90b08c271ef0b225c141267043d900f6e7d7564/userdata/shm major:0 minor:83 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/105e79f277b5491b41149e986f955c0083288f78b77703e37bbf1cd561db332c/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/105e79f277b5491b41149e986f955c0083288f78b77703e37bbf1cd561db332c/userdata/shm major:0 minor:88 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/12bf89ab2c442ecd84c7a7607d703dd213b5513e6a93c92ef2dde5bf06ad6f99/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/12bf89ab2c442ecd84c7a7607d703dd213b5513e6a93c92ef2dde5bf06ad6f99/userdata/shm major:0 minor:246 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/1bfb2f27afea8897b43d2d990b1e208f70c168e0be7c46e2c3a03b21c99b28b0/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/1bfb2f27afea8897b43d2d990b1e208f70c168e0be7c46e2c3a03b21c99b28b0/userdata/shm major:0 minor:72 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/306f16787f3392fc8b7ace0026e1c8efe17179d370cdd50b03d02cb649070aaa/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/306f16787f3392fc8b7ace0026e1c8efe17179d370cdd50b03d02cb649070aaa/userdata/shm major:0 minor:346 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/3497d23ca426dcc552e2336dd5013e9a8e2a9195710bceeb655f082d4aef724d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/3497d23ca426dcc552e2336dd5013e9a8e2a9195710bceeb655f082d4aef724d/userdata/shm major:0 minor:244 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/357678e8fb63a8473b0f2c085a256547f8cd0e54439948bcf9e6493824d0ff36/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/357678e8fb63a8473b0f2c085a256547f8cd0e54439948bcf9e6493824d0ff36/userdata/shm major:0 minor:210 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/45b221fc6e0aeea83b1d8e8c88b6e9db2827f1a4aa5b5c74bc6b6ea9efc46143/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/45b221fc6e0aeea83b1d8e8c88b6e9db2827f1a4aa5b5c74bc6b6ea9efc46143/userdata/shm major:0 minor:48 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/5073165fba6902e2ceb840e7ba07c727561dac833681c45f203b6cf37a9c42bd/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/5073165fba6902e2ceb840e7ba07c727561dac833681c45f203b6cf37a9c42bd/userdata/shm major:0 minor:235 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/5ad1aa46bf6437794ab135946309ee2068f696296572304aee0cbf7dfc6ea469/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/5ad1aa46bf6437794ab135946309ee2068f696296572304aee0cbf7dfc6ea469/userdata/shm major:0 minor:76 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/66674f7724debd180e9b8be4d79be7b77877df53417946aa834a4897afec5876/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/66674f7724debd180e9b8be4d79be7b77877df53417946aa834a4897afec5876/userdata/shm major:0 minor:80 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/6e130a212380a09d1a1465bc747f921897753868d89543228dcfa0e8b911773c/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/6e130a212380a09d1a1465bc747f921897753868d89543228dcfa0e8b911773c/userdata/shm major:0 minor:362 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/7182622ef6dda99f8aed4b16c806aa974d9e90326009d6dc3149e9bd8b4f5cb2/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/7182622ef6dda99f8aed4b16c806aa974d9e90326009d6dc3149e9bd8b4f5cb2/userdata/shm major:0 minor:348 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/74e94b63179e44eb28db4bd8dd65d786876ea1da8d701522187f36b22647470a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/74e94b63179e44eb28db4bd8dd65d786876ea1da8d701522187f36b22647470a/userdata/shm major:0 minor:68 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/7e90fa928e74f963359d215e04ec3bd27f51c1496663feae053c18839e060ba3/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/7e90fa928e74f963359d215e04ec3bd27f51c1496663feae053c18839e060ba3/userdata/shm major:0 minor:234 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/91c64ba91f2d616e39faaef286d3e721f03f6ac5c7d010d6b674016160e7aa6e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/91c64ba91f2d616e39faaef286d3e721f03f6ac5c7d010d6b674016160e7aa6e/userdata/shm major:0 minor:334 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/9b0ccc77fd551612f3962882d98c4ffc275de65a04563263a46cb4830ae83020/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/9b0ccc77fd551612f3962882d98c4ffc275de65a04563263a46cb4830ae83020/userdata/shm major:0 minor:221 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a146455d1f30eb2f7f4ad58212035a0c1f733cb997565d267edda0121121fa40/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a146455d1f30eb2f7f4ad58212035a0c1f733cb997565d267edda0121121fa40/userdata/shm major:0 minor:238 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a2d98ed6ecf479a4141b67729bc58f0489ca18f41040a2da19154c4317bcaac5/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a2d98ed6ecf479a4141b67729bc58f0489ca18f41040a2da19154c4317bcaac5/userdata/shm major:0 minor:98 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/bba607643ebf9639b0c8044c3e3694561fdacdd3731bcf15feae4311301c9f2b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/bba607643ebf9639b0c8044c3e3694561fdacdd3731bcf15feae4311301c9f2b/userdata/shm major:0 minor:290 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/ddade6cb4a39d5c69647385ce4c28b127550d12a98dfe2caade4862845ce5388/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/ddade6cb4a39d5c69647385ce4c28b127550d12a98dfe2caade4862845ce5388/userdata/shm major:0 minor:94 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e459951077da3523c5cc454f50947afe46c379d6e8c40321e83519e9f3c10d61/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e459951077da3523c5cc454f50947afe46c379d6e8c40321e83519e9f3c10d61/userdata/shm major:0 minor:242 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e5ff0a702058f6c4a964d51275a39426fdf7de02bd4756e66c20fde0c8b3a1d3/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e5ff0a702058f6c4a964d51275a39426fdf7de02bd4756e66c20fde0c8b3a1d3/userdata/shm major:0 minor:47 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e7be247cfa0d4a3703d5d9670caf05d05e034c7c4142a470ca2f537b939bc2b9/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e7be247cfa0d4a3703d5d9670caf05d05e034c7c4142a470ca2f537b939bc2b9/userdata/shm major:0 minor:288 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/08490837-b3b6-4c2b-a498-568af0d7086d/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/08490837-b3b6-4c2b-a498-568af0d7086d/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:61 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/08490837-b3b6-4c2b-a498-568af0d7086d/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/08490837-b3b6-4c2b-a498-568af0d7086d/volumes/kubernetes.io~empty-dir/tmp major:0 minor:60 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/08490837-b3b6-4c2b-a498-568af0d7086d/volumes/kubernetes.io~projected/kube-api-access-w9tzc:{mountpoint:/var/lib/kubelet/pods/08490837-b3b6-4c2b-a498-568af0d7086d/volumes/kubernetes.io~projected/kube-api-access-w9tzc major:0 minor:63 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0ab3954e-be64-4260-bca8-ff5d58bd6ea2/volumes/kubernetes.io~projected/kube-api-access-vms79:{mountpoint:/var/lib/kubelet/pods/0ab3954e-be64-4260-bca8-ff5d58bd6ea2/volumes/kubernetes.io~projected/kube-api-access-vms79 major:0 minor:203 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/147bb9af-dfc7-4785-bb5e-906f25d65dd1/volumes/kubernetes.io~projected/kube-api-access-g9wls:{mountpoint:/var/lib/kubelet/pods/147bb9af-dfc7-4785-bb5e-906f25d65dd1/volumes/kubernetes.io~projected/kube-api-access-g9wls major:0 minor:233 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/147bb9af-dfc7-4785-bb5e-906f25d65dd1/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/147bb9af-dfc7-4785-bb5e-906f25d65dd1/volumes/kubernetes.io~secret/cert major:0 minor:218 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/318b3bc6-6e59-4211-af63-b116526a325d/volumes/kubernetes.io~projected/kube-api-access-rms4s:{mountpoint:/var/lib/kubelet/pods/318b3bc6-6e59-4211-af63-b116526a325d/volumes/kubernetes.io~projected/kube-api-access-rms4s major:0 minor:62 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/565fefaf-82aa-42a8-9cc7-6517c4bddf1e/volumes/kubernetes.io~projected/kube-api-access-r6zx9:{mountpoint:/var/lib/kubelet/pods/565fefaf-82aa-42a8-9cc7-6517c4bddf1e/volumes/kubernetes.io~projected/kube-api-access-r6zx9 major:0 minor:287 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/57c2f196-0d8a-4bce-a5bc-3b6b5c0d808c/volumes/kubernetes.io~projected/kube-api-access-2dx97:{mountpoint:/var/lib/kubelet/pods/57c2f196-0d8a-4bce-a5bc-3b6b5c0d808c/volumes/kubernetes.io~projected/kube-api-access-2dx97 major:0 minor:66 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/58b0b75a-1568-4703-ac55-fa87f29225ad/volumes/kubernetes.io~projected/kube-api-access-t228s:{mountpoint:/var/lib/kubelet/pods/58b0b75a-1568-4703-ac55-fa87f29225ad/volumes/kubernetes.io~projected/kube-api-access-t228s major:0 minor:64 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/69ff2b62-4588-4cb0-bdad-747c3692138c/volumes/kubernetes.io~projected/kube-api-access-6qc9x:{mountpoint:/var/lib/kubelet/pods/69ff2b62-4588-4cb0-bdad-747c3692138c/volumes/kubernetes.io~projected/kube-api-access-6qc9x major:0 minor:224 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6fffc15e-94bd-4939-b38f-17d1fe6cc0b5/volumes/kubernetes.io~projected/kube-api-access-dtjj5:{mountpoint:/var/lib/kubelet/pods/6fffc15e-94bd-4939-b38f-17d1fe6cc0b5/volumes/kubernetes.io~projected/kube-api-access-dtjj5 major:0 minor:361 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6fffc15e-94bd-4939-b38f-17d1fe6cc0b5/volumes/kubernetes.io~secret/signing-key:{mountpoint:/var/lib/kubelet/pods/6fffc15e-94bd-4939-b38f-17d1fe6cc0b5/volumes/kubernetes.io~secret/signing-key major:0 minor:360 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7dd98c21-e0b1-4289-8fe8-036bf413d444/volumes/kubernetes.io~projected/kube-api-access-7fhrx:{mountpoint:/var/lib/kubelet/pods/7dd98c21-e0b1-4289-8fe8-036bf413d444/volumes/kubernetes.io~projected/kube-api-access-7fhrx major:0 minor:322 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/85bb69db-c69b-467e-adeb-950bf2947c94/volumes/kubernetes.io~projected/kube-api-access-8n8tt:{mountpoint:/var/lib/kubelet/pods/85bb69db-c69b-467e-adeb-950bf2947c94/volumes/kubernetes.io~projected/kube-api-access-8n8tt major:0 minor:228 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8952fc79-c5a6-42ba-a296-8cb7a776f718/volumes/kubernetes.io~projected/kube-api-access-srkbh:{mountpoint:/var/lib/kubelet/pods/8952fc79-c5a6-42ba-a296-8cb7a776f718/volumes/kubernetes.io~projected/kube-api-access-srkbh major:0 minor:65 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9122c2fd-f29f-4e74-9615-15078517a3c0/volumes/kubernetes.io~projected/kube-api-access-r4l7n:{mountpoint:/var/lib/kubelet/pods/9122c2fd-f29f-4e74-9615-15078517a3c0/volumes/kubernetes.io~projected/kube-api-access-r4l7n major:0 minor:329 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9cd1266e-c630-4259-b1f7-1b88dbde7385/volumes/kubernetes.io~projected/kube-api-access-jxmsl:{mountpoint:/var/lib/kubelet/pods/9cd1266e-c630-4259-b1f7-1b88dbde7385/volumes/kubernetes.io~projected/kube-api-access-jxmsl major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/aa112079-b749-404c-9081-113220707fcd/volumes/kubernetes.io~projected/bound-sa-token:{mountpoint:/var/lib/kubelet/pods/aa112079-b749-404c-9081-113220707fcd/volumes/kubernetes.io~projected/bound-sa-token major:0 minor:208 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/aa112079-b749-404c-9081-113220707fcd/volumes/kubernetes.io~projected/kube-api-access-rpmxj:{mountpoint:/var/lib/kubelet/pods/aa112079-b749-404c-9081-113220707fcd/volumes/kubernetes.io~projected/kube-api-access-rpmxj major:0 minor:204 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/aa112079-b749-404c-9081-113220707fcd/volumes/kubernetes.io~secret/image-registry-private-configuration:{mountpoint:/var/lib/kubelet/pods/aa112079-b749-404c-9081-113220707fcd/volumes/kubernetes.io~secret/image-registry-private-configuration major:0 minor:202 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/aa112079-b749-404c-9081-113220707fcd/volumes/kubernetes.io~secret/installation-pull-secrets:{mountpoint:/var/lib/kubelet/pods/aa112079-b749-404c-9081-113220707fcd/volumes/kubernetes.io~secret/installation-pull-secrets major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/aa4836a5-c71a-485d-82fc-f8b53dae7717/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/aa4836a5-c71a-485d-82fc-f8b53dae7717/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:286 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b5975856-8eb3-4601-a7e1-d1b1b3a957e8/volumes/kubernetes.io~projected/kube-api-access-qzhpr:{mountpoint:/var/lib/kubelet/pods/b5975856-8eb3-4601-a7e1-d1b1b3a957e8/volumes/kubernetes.io~projected/kube-api-access-qzhpr major:0 minor:340 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b66619db-fd81-4435-be2a-a5ea6111e204/volumes/kubernetes.io~projected/kube-api-access-59kdl:{mountpoint:/var/lib/kubelet/pods/b66619db-fd81-4435-be2a-a5ea6111e204/volumes/kubernetes.io~projected/kube-api-access-59kdl major:0 minor:230 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c2be658b-ec92-4782-a78b-d2dccf2249cb/volumes/kubernetes.io~projected/kube-api-access-bjgwm:{mountpoint:/var/lib/kubelet/pods/c2be658b-ec92-4782-a78b-d2dccf2249cb/volumes/kubernetes.io~projected/kube-api-access-bjgwm major:0 minor:227 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c2be658b-ec92-4782-a78b-d2dccf2249cb/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/c2be658b-ec92-4782-a78b-d2dccf2249cb/volumes/kubernetes.io~secret/serving-cert major:0 minor:212 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c8e98df5-b2f7-49dd-a565-ed60534da496/volumes/kubernetes.io~projected/kube-api-access-tlnlx:{mountpoint:/var/lib/kubelet/pods/c8e98df5-b2f7-49dd-a565-ed60534da496/volumes/kubernetes.io~projected/kube-api-access-tlnlx major:0 minor:225 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c8e98df5-b2f7-49dd-a565-ed60534da496/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/c8e98df5-b2f7-49dd-a565-ed60534da496/volumes/kubernetes.io~secret/serving-cert major:0 minor:216 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/cb657366-6696-4c3e-a3a3-603ce00cf11a/volumes/kubernetes.io~projected/kube-api-access-bspb5:{mountpoint:/var/lib/kubelet/pods/cb657366-6696-4c3e-a3a3-603ce00cf11a/volumes/kubernetes.io~projected/kube-api-access-bspb5 major:0 minor:345 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d55eb70a-f1a8-4346-b8df-ccd2c598dd6b/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/d55eb70a-f1a8-4346-b8df-ccd2c598dd6b/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d55eb70a-f1a8-4346-b8df-ccd2c598dd6b/volumes/kubernetes.io~projected/kube-api-access-ftdwt:{mountpoint:/var/lib/kubelet/pods/d55eb70a-f1a8-4346-b8df-ccd2c598dd6b/volumes/kubernetes.io~projected/kube-api-access-ftdwt major:0 minor:67 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d55eb70a-f1a8-4346-b8df-ccd2c598dd6b/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/d55eb70a-f1a8-4346-b8df-ccd2c598dd6b/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d719eda9-bc9f-4307-952a-99d99aa0f62b/volumes/kubernetes.io~projected/kube-api-access-rz545:{mountpoint:/var/lib/kubelet/pods/d719eda9-bc9f-4307-952a-99d99aa0f62b/volumes/kubernetes.io~projected/kube-api-access-rz545 major:0 minor:209 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/da8395ef-7464-410e-96ee-b27378a03fed/volumes/kubernetes.io~projected/kube-api-access-jkn67:{mountpoint:/var/lib/kubelet/pods/da8395ef-7464-410e-96ee-b27378a03fed/volumes/kubernetes.io~projected/kube-api-access-jkn67 major:0 minor:205 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/dd8134f9-b2a6-4542-82ec-c10edc3fcd72/volumes/kubernetes.io~projected/kube-api-access-wsrb6:{mountpoint:/var/lib/kubelet/pods/dd8134f9-b2a6-4542-82ec-c10edc3fcd72/volumes/kubernetes.io~projected/kube-api-access-wsrb6 major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e49453a8-c7d9-4248-a8ef-4a819b76806c/volumes/kubernetes.io~projected/kube-api-access-98n75:{mountpoint:/var/lib/kubelet/pods/e49453a8-c7d9-4248-a8ef-4a819b76806c/volumes/kubernetes.io~projected/kube-api-access-98n75 major:0 minor:232 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e49453a8-c7d9-4248-a8ef-4a819b76806c/volumes/kubernetes.io~secret/default-certificate:{mountpoint:/var/lib/kubelet/pods/e49453a8-c7d9-4248-a8ef-4a819b76806c/volumes/kubernetes.io~secret/default-certificate major:0 minor:214 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e49453a8-c7d9-4248-a8ef-4a819b76806c/volumes/kubernetes.io~secret/stats-auth:{mountpoint:/var/lib/kubelet/pods/e49453a8-c7d9-4248-a8ef-4a819b76806c/volumes/kubernetes.io~secret/stats-auth major:0 minor:217 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e54a9dc6-36e6-4f1b-9c3a-623a520e11da/volumes/kubernetes.io~projected/kube-api-access-ks6fn:{mountpoint:/var/lib/kubelet/pods/e54a9dc6-36e6-4f1b-9c3a-623a520e11da/volumes/kubernetes.io~projected/kube-api-access-ks6fn major:0 minor:231 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e54a9dc6-36e6-4f1b-9c3a-623a520e11da/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/e54a9dc6-36e6-4f1b-9c3a-623a520e11da/volumes/kubernetes.io~secret/serving-cert major:0 minor:215 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/eafc69c5-15f1-4e4c-97ca-be586e55fbae/volumes/kubernetes.io~projected/kube-api-access-j78lj:{mountpoint:/var/lib/kubelet/pods/eafc69c5-15f1-4e4c-97ca-be586e55fbae/volumes/kubernetes.io~projected/kube-api-access-j78lj major:0 minor:223 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f0c0ada7-7688-4dc2-97fc-813e4d0c4ef3/volumes/kubernetes.io~projected/kube-api-access-zlmlp:{mountpoint:/var/lib/kubelet/pods/f0c0ada7-7688-4dc2-97fc-813e4d0c4ef3/volumes/kubernetes.io~projected/kube-api-access-zlmlp major:0 minor:229 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f0c0ada7-7688-4dc2-97fc-813e4d0c4ef3/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/f0c0ada7-7688-4dc2-97fc-813e4d0c4ef3/volumes/kubernetes.io~secret/serving-cert major:0 minor:213 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f273b9e5-0419-4587-a9c7-a0c8bec02acf/volumes/kubernetes.io~projected/kube-api-access-lldxn:{mountpoint:/var/lib/kubelet/pods/f273b9e5-0419-4587-a9c7-a0c8bec02acf/volumes/kubernetes.io~projected/kube-api-access-lldxn major:0 minor:226 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f8855c2c-87b4-4b9e-b28a-227981205b11/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/f8855c2c-87b4-4b9e-b28a-227981205b11/volumes/kubernetes.io~secret/agent-certs major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fa085dd8-ca1e-4ec5-acb2-9b9dc878bf8a/volumes/kubernetes.io~projected/kube-api-access-nmlm7:{mountpoint:/var/lib/kubelet/pods/fa085dd8-ca1e-4ec5-acb2-9b9dc878bf8a/volumes/kubernetes.io~projected/kube-api-access-nmlm7 major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ff4dcba2-df8d-4324-9e0c-abb23a5e6982/volumes/kubernetes.io~projected/bound-sa-token:{mountpoint:/var/lib/kubelet/pods/ff4dcba2-df8d-4324-9e0c-abb23a5e6982/volumes/kubernetes.io~projected/bound-sa-token major:0 minor:206 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ff4dcba2-df8d-4324-9e0c-abb23a5e6982/volumes/kubernetes.io~projected/kube-api-access-v9dnm:{mountpoint:/var/lib/kubelet/pods/ff4dcba2-df8d-4324-9e0c-abb23a5e6982/volumes/kubernetes.io~projected/kube-api-access-v9dnm major:0 minor:207 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ff4dcba2-df8d-4324-9e0c-abb23a5e6982/volumes/kubernetes.io~secret/image-registry-private-configuration:{mountpoint:/var/lib/kubelet/pods/ff4dcba2-df8d-4324-9e0c-abb23a5e6982/volumes/kubernetes.io~secret/image-registry-private-configuration major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ff4dcba2-df8d-4324-9e0c-abb23a5e6982/volumes/kubernetes.io~secret/installation-pull-secrets:{mountpoint:/var/lib/kubelet/pods/ff4dcba2-df8d-4324-9e0c-abb23a5e6982/volumes/kubernetes.io~secret/installation-pull-secrets major:0 minor:201 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/30e1741efc41ff29440d4c57ec226c8fe47afa8b39450814632a755064800127/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/82663c638eee975d44c2785185ade1db88753df3e52287aa06faf12e1ce66e15/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-106:{mountpoint:/var/lib/containers/storage/overlay/602506c126f6c55897c51bab6b95481ccd24df3a040342048e3fa416db88c0f2/merged major:0 minor:106 fsType:overlay blockSize:0} overlay_0-111:{mountpoint:/var/lib/containers/storage/overlay/c3f019ae128919a6a3548d933b186e8a16dc9b7650aee24914f5c8fb4e5d7a7b/merged major:0 minor:111 fsType:overlay blockSize:0} overlay_0-114:{mountpoint:/var/lib/containers/storage/overlay/95d867d97b67b0056d3e0e2605826cf54c18367d786464f9789c1ea5a8c6bbf8/merged major:0 minor:114 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/df14b83297d98adb1bce8541d89322d3be9e45a2cf83e562aa6d6ac9d12b2cc9/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/1a5d86e4595f251b7216f33e80d83e58be46f530b6f0634187b67af2dfa49746/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/0766cedbfb3c4eb731b3346bf70be80b18d323a1a983dec5bb067837a229203b/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-129:{mountpoint:/var/lib/containers/storage/overlay/eaa564243a99ef7a81e3c4fe438c138da30895127c697f2a11b8cbd0f615cb87/merged major:0 minor:129 fsType:overlay blockSize:0} overlay_0-131:{mountpoint:/var/lib/containers/storage/overlay/5df479bc56516e50c0d06540029f52ccb61a53c96d58c0b15023acfbcb199dc4/merged major:0 minor:131 fsType:overlay blockSize:0} overlay_0-142:{mountpoint:/var/lib/containers/storage/overlay/e23784d7190b9047effb52edc0471eed4232dd64cc4b49263697cc70245f8247/merged major:0 minor:142 fsType:overlay blockSize:0} overlay_0-144:{mountpoint:/var/lib/containers/storage/overlay/4944640b4d500c99b684cb5da0636ebe432769faed03f88ac5c1d6b9b4f60d08/merged major:0 minor:144 fsType:overlay blockSize:0} overlay_0-152:{mountpoint:/var/lib/containers/storage/overlay/179a095a8dd72226a0cf65b3e312851a4593078625fa9f5e00cbfb2f7aaa4fb1/merged major:0 minor:152 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/810135ea87f8f9f25272341c4842d96e4077ae9e908f7aaccce22e807cb01ead/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-164:{mountpoint:/var/lib/containers/storage/overlay/1c1df72d2b67c1760b0234633849f9d31be2c9bf939ac045cf2e1de008f3f61d/merged major:0 minor:164 fsType:overlay blockSize:0} overlay_0-168:{mountpoint:/var/lib/containers/storage/overlay/a448be8871545842fa1002e754ec890455726fffee9a1b3ccb64d345999858ac/merged major:0 minor:168 fsType:overlay blockSize:0} overlay_0-171:{mountpoint:/var/lib/containers/storage/overlay/8bc776dbedaa00c9c94ef290a895a9caff19533516bf5c0a9b2c5a6ae5a14ba4/merged major:0 minor:171 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/538648b85771fceb469d1e50e3d5f509aaf5bd3f865f7da54bc57d8aefa20907/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/9c2cc83ddee4daa03db7a6987df5e86abfbd93ee267ed50229f5187a97353403/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/733ba7a4ab0a1c381a7d9fa215ff172750fb15dd0ce06e04d302d58670124b46/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/1862ca21081b30298596d3411a2b0bd45bd4fdf287b8b3e9355bccf13d767796/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-248:{mountpoint:/var/lib/containers/storage/overlay/5daed01d69da27c18e59da473ddc4061368012ce95965f42d881e65f749dd61e/merged major:0 minor:248 fsType:overlay blockSize:0} overlay_0-250:{mountpoint:/var/lib/containers/storage/overlay/f38c404be1cf9800b3c25b8c3698991f8be33929112f4f44ec8b58ed1788e1a9/merged major:0 minor:250 fsType:overlay blockSize:0} overlay_0-252:{mountpoint:/var/lib/containers/storage/overlay/4401341e666c112fb80b4f1beea1460c6af78a3b3b77dc079ea9d45d0a1aea67/merged major:0 minor:252 fsType:overlay blockSize:0} overlay_0-254:{mountpoint:/var/lib/containers/storage/overlay/58ef6cc6dd4323ac2a9bef0b7d0f46585679a57bcc2ee003303c83bef1eabbcd/merged major:0 minor:254 fsType:overlay blockSize:0} overlay_0-256:{mountpoint:/var/lib/containers/storage/overlay/cd8fbd7f18f4243fbbdbdf0f67e11613df31c0617d2adcfd1a5a58501bc30de0/merged major:0 minor:256 fsType:overlay blockSize:0} overlay_0-258:{mountpoint:/var/lib/containers/storage/overlay/e1fe63e05e0763eeab18807c1eed2609683f9e55b833ebdc2f9d1e10e37dd83a/merged major:0 minor:258 fsType:overlay blockSize:0} overlay_0-260:{mountpoint:/var/lib/containers/storage/overlay/073b33319041f86bff004e7f55f076734122d96d8a4cb15b1da00d7c1834fcef/merged major:0 minor:260 fsType:overlay blockSize:0} overlay_0-266:{mountpoint:/var/lib/containers/storage/overlay/57932cdbce2fab9a908b4e3b8966f954dc8e1a51d89f0ba9070ce8587447eca0/merged major:0 minor:266 fsType:overlay blockSize:0} overlay_0-268:{mountpoint:/var/lib/containers/storage/overlay/af7c9eead22261d5ebd0d981bc5e88c77ef779cb5ae76e1fbd53be9e0d9bad74/merged major:0 minor:268 fsType:overlay blockSize:0} overlay_0-270:{mountpoint:/var/lib/containers/storage/overlay/6310c8def43afe6d3072d92014a20179159ed0a4602d8fc79501c1ac541a3983/merged major:0 minor:270 fsType:overlay blockSize:0} overlay_0-272:{mountpoint:/var/lib/containers/storage/overlay/66ee9a335fd5d7377cfe1e915fdfbef2fdf6ac8fec10391ead729ec0da1edcda/merged major:0 minor:272 fsType:overlay blockSize:0} overlay_0-273:{mountpoint:/var/lib/containers/storage/overlay/f24135a0b7579fe4e42229d8b505f120712cff2592194c5637d1d8366a781d52/merged major:0 minor:273 fsType:overlay blockSize:0} overlay_0-284:{mountpoint:/var/lib/containers/storage/overlay/cdfbb220d788fda396ec7c2f1cae0b4eb53f5dbae91b7d3d564eaec868c0e386/merged major:0 minor:284 fsType:overlay blockSize:0} overlay_0-292:{mountpoint:/var/lib/containers/storage/overlay/5cc49de43e067af706313be7f3f123798b87f8b71fe298022772f4a669e30661/merged major:0 minor:292 fsType:overlay blockSize:0} overlay_0-294:{mountpoint:/var/lib/containers/storage/overlay/3374e40784b01ccb9e31e68f9135d71e751c839e4124a22b757debf62b7729f9/merged major:0 minor:294 fsType:overlay blockSize:0} overlay_0-300:{mountpoint:/var/lib/containers/storage/overlay/cbe9a74d58edaee5488c937f647ca05d035e2e0e46c537ec3cd43f4b6ceb4998/merged major:0 minor:300 fsType:overlay blockSize:0} overlay_0-302:{mountpoint:/var/lib/containers/storage/overlay/b59fc742a6e2f354375b0ed11d92b48c465e1ca3878f2de3c07b2c7cdabe68c7/merged major:0 minor:302 fsType:overlay blockSize:0} overlay_0-304:{mountpoint:/var/lib/containers/storage/overlay/d99c1c82c88defa4dfcedffa896fc11f8aaf49e5fc2b33bb1a32d4ebd01d1f22/merged major:0 minor:304 fsType:overlay blockSize:0} overlay_0-310:{mountpoint:/var/lib/containers/storage/overlay/6c256e603674b1a35c7a09115c4c94730f322f11fd7f5dc867cef82d45bdcfc7/merged major:0 minor:310 fsType:overlay blockSize:0} overlay_0-312:{mountpoint:/var/lib/containers/storage/overlay/b66d8cbc9ee8ec15bc80f9d7158369d57d24e4a43ab15e890da5952c917f5b57/merged major:0 minor:312 fsType:overlay blockSize:0} overlay_0-327:{mountpoint:/var/lib/containers/storage/overlay/96b4ab1585759e048d04c10d727ecc77d52c7e1b0db0a72897e885dd83650578/merged major:0 minor:327 fsType:overlay blockSize:0} overlay_0-336:{mountpoint:/var/lib/containers/storage/overlay/fe9ad119c71fd5a68968b662c08ee94071f49e3c153714c7caf64d5979d8665f/merged major:0 minor:336 fsType:overlay blockSize:0} overlay_0-338:{mountpoint:/var/lib/containers/storage/overlay/60e67db4b97393f1181bca7003ca45af524c3803b05a194f1ede2b58c4b10997/merged major:0 minor:338 fsType:overlay blockSize:0} overlay_0-350:{mountpoint:/var/lib/containers/storage/overlay/fac31a783ab8cb0a41d543727df694dcda1642cbdfa2ef6ea2ec9e776f8bb4df/merged major:0 minor:350 fsType:overlay blockSize:0} overlay_0-352:{mountpoint:/var/lib/containers/storage/overlay/cbb0d5b3b6335e7f64b55e4a04947f189a4d6e724891d2cbc58b49f0a864a162/merged major:0 minor:352 fsType:overlay blockSize:0} overlay_0-358:{mountpoint:/var/lib/containers/storage/overlay/5e06ced4a8c77fa3e3f379c91a85ee66abcc632d4cf32b44a55cef64153c1d13/merged major:0 minor:358 fsType:overlay blockSize:0} overlay_0-364:{mountpoint:/var/lib/containers/storage/overlay/dc0a14b3ce85c4d9875b700e1c7326a17ca47aa81f3424b23545f6ece3d955ca/merged major:0 minor:364 fsType:overlay blockSize:0} overlay_0-366:{mountpoint:/var/lib/containers/storage/overlay/45806a144044f43f53eb068a6ac0d32867ffa1ad4c653df29b01adec62d21271/merged major:0 minor:366 fsType:overlay blockSize:0} overlay_0-368:{mountpoint:/var/lib/containers/storage/overlay/6a0331bf31cdd3b52384c66a070c1fe5c2d5451ea907ff65b9ebb8966154a8cf/merged major:0 minor:368 fsType:overlay blockSize:0} overlay_0-51:{mountpoint:/var/lib/containers/storage/overlay/df5c992267f6dd87a9612869db35b4fb728609140bff295332123c3b637349b6/merged major:0 minor:51 fsType:overlay blockSize:0} overlay_0-53:{mountpoint:/var/lib/containers/storage/overlay/71cc12805afcd105316f5b2b1a8040a3cabbf5ee1c38bb0bba77ed56fa7663a6/merged major:0 minor:53 fsType:overlay blockSize:0} overlay_0-70:{mountpoint:/var/lib/containers/storage/overlay/e0e6040bfd7fafbe2d1a8f2eedf4dc242fea377fee2ef04d263d82c39a8c1846/merged major:0 minor:70 fsType:overlay blockSize:0} overlay_0-74:{mountpoint:/var/lib/containers/storage/overlay/bb2302753d4481fc672111e7574cee15485f547d06fd03168d6651653dd90fd4/merged major:0 minor:74 fsType:overlay blockSize:0} overlay_0-78:{mountpoint:/var/lib/containers/storage/overlay/b30c119bbc367a365241e2d6f34729d259fe181261d9ca230845c9a7c0dfdeb5/merged major:0 minor:78 fsType:overlay blockSize:0} overlay_0-82:{mountpoint:/var/lib/containers/storage/overlay/4fb292434521c91442031dbb2e07c55fd0ad0c0ea471bb4506be7fa6de662505/merged major:0 minor:82 fsType:overlay blockSize:0} overlay_0-86:{mountpoint:/var/lib/containers/storage/overlay/f9b16c41b917060d53babf886046e6c72302b51e5b442f9a5320c8ad8c2ccc7f/merged major:0 minor:86 fsType:overlay blockSize:0} overlay_0-90:{mountpoint:/var/lib/containers/storage/overlay/f404f91b7001b922949d97501c90adbdb3527897477f1ebee4693ff964362f72/merged major:0 minor:90 fsType:overlay blockSize:0} overlay_0-92:{mountpoint:/var/lib/containers/storage/overlay/f22441b1cde447d4d0b062b8c4581e3177054d39ca3ccbf4a7bf7df464b3ace2/merged major:0 minor:92 fsType:overlay blockSize:0} overlay_0-96:{mountpoint:/var/lib/containers/storage/overlay/4235e9352e873ea6257c9151e08160c487c418fc51f5ad3139411b2933fccfea/merged major:0 minor:96 fsType:overlay blockSize:0}]
Oct 31 01:03:51.621018 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:51.619990    4964 manager.go:217] Machine: {Timestamp:2026-10-31 01:03:51.61899909 +0000 UTC m=+0.095565462 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3200290 MemoryCapacity:32812019712 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec26fc72061aacb4e05cbcf025a13103 SystemUUID:ec26fc72-061a-acb4-e05c-bcf025a13103 BootID:734896fe-ba5a-4878-a994-4b46528ea61c Filesystems:[{Device:overlay_0-51 DeviceMajor:0 DeviceMinor:51 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-78 DeviceMajor:0 DeviceMinor:78 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-106 DeviceMajor:0 DeviceMinor:106 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/318b3bc6-6e59-4211-af63-b116526a325d/volumes/kubernetes.io~projected/kube-api-access-rms4s DeviceMajor:0 DeviceMinor:62 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/0ab3954e-be64-4260-bca8-ff5d58bd6ea2/volumes/kubernetes.io~projected/kube-api-access-vms79 DeviceMajor:0 DeviceMinor:203 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/d719eda9-bc9f-4307-952a-99d99aa0f62b/volumes/kubernetes.io~projected/kube-api-access-rz545 DeviceMajor:0 DeviceMinor:209 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-268 DeviceMajor:0 DeviceMinor:268 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/565fefaf-82aa-42a8-9cc7-6517c4bddf1e/volumes/kubernetes.io~projected/kube-api-access-r6zx9 DeviceMajor:0 DeviceMinor:287 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/7dd98c21-e0b1-4289-8fe8-036bf413d444/volumes/kubernetes.io~projected/kube-api-access-7fhrx DeviceMajor:0 DeviceMinor:322 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-53 DeviceMajor:0 DeviceMinor:53 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/85bb69db-c69b-467e-adeb-950bf2947c94/volumes/kubernetes.io~projected/kube-api-access-8n8tt DeviceMajor:0 DeviceMinor:228 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/5073165fba6902e2ceb840e7ba07c727561dac833681c45f203b6cf37a9c42bd/userdata/shm DeviceMajor:0 DeviceMinor:235 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-300 DeviceMajor:0 DeviceMinor:300 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/cb657366-6696-4c3e-a3a3-603ce00cf11a/volumes/kubernetes.io~projected/kube-api-access-bspb5 DeviceMajor:0 DeviceMinor:345 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/147bb9af-dfc7-4785-bb5e-906f25d65dd1/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:218 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/e54a9dc6-36e6-4f1b-9c3a-623a520e11da/volumes/kubernetes.io~projected/kube-api-access-ks6fn DeviceMajor:0 DeviceMinor:231 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/f8855c2c-87b4-4b9e-b28a-227981205b11/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:59 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/66674f7724debd180e9b8be4d79be7b77877df53417946aa834a4897afec5876/userdata/shm DeviceMajor:0 DeviceMinor:80 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/fa085dd8-ca1e-4ec5-acb2-9b9dc878bf8a/volumes/kubernetes.io~projected/kube-api-access-nmlm7 DeviceMajor:0 DeviceMinor:56 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-82 DeviceMajor:0 DeviceMinor:82 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/e49453a8-c7d9-4248-a8ef-4a819b76806c/volumes/kubernetes.io~secret/default-certificate DeviceMajor:0 DeviceMinor:214 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/e49453a8-c7d9-4248-a8ef-4a819b76806c/volumes/kubernetes.io~secret/stats-auth DeviceMajor:0 DeviceMinor:217 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/45b221fc6e0aeea83b1d8e8c88b6e9db2827f1a4aa5b5c74bc6b6ea9efc46143/userdata/shm DeviceMajor:0 DeviceMinor:48 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ff4dcba2-df8d-4324-9e0c-abb23a5e6982/volumes/kubernetes.io~secret/installation-pull-secrets DeviceMajor:0 DeviceMinor:201 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/9b0ccc77fd551612f3962882d98c4ffc275de65a04563263a46cb4830ae83020/userdata/shm DeviceMajor:0 DeviceMinor:221 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-250 DeviceMajor:0 DeviceMinor:250 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-310 DeviceMajor:0 DeviceMinor:310 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-358 DeviceMajor:0 DeviceMinor:358 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-338 DeviceMajor:0 DeviceMinor:338 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/5ad1aa46bf6437794ab135946309ee2068f696296572304aee0cbf7dfc6ea469/userdata/shm DeviceMajor:0 DeviceMinor:76 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-92 DeviceMajor:0 DeviceMinor:92 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-272 DeviceMajor:0 DeviceMinor:272 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/b5975856-8eb3-4601-a7e1-d1b1b3a957e8/volumes/kubernetes.io~projected/kube-api-access-qzhpr DeviceMajor:0 DeviceMinor:340 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/6e130a212380a09d1a1465bc747f921897753868d89543228dcfa0e8b911773c/userdata/shm DeviceMajor:0 DeviceMinor:362 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-364 DeviceMajor:0 DeviceMinor:364 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/57c2f196-0d8a-4bce-a5bc-3b6b5c0d808c/volumes/kubernetes.io~projected/kube-api-access-2dx97 DeviceMajor:0 DeviceMinor:66 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/aa112079-b749-404c-9081-113220707fcd/volumes/kubernetes.io~secret/installation-pull-secrets DeviceMajor:0 DeviceMinor:197 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/c8e98df5-b2f7-49dd-a565-ed60534da496/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:216 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-252 DeviceMajor:0 DeviceMinor:252 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-260 DeviceMajor:0 DeviceMinor:260 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e7be247cfa0d4a3703d5d9670caf05d05e034c7c4142a470ca2f537b939bc2b9/userdata/shm DeviceMajor:0 DeviceMinor:288 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-327 DeviceMajor:0 DeviceMinor:327 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/c8e98df5-b2f7-49dd-a565-ed60534da496/volumes/kubernetes.io~projected/kube-api-access-tlnlx DeviceMajor:0 DeviceMinor:225 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/8952fc79-c5a6-42ba-a296-8cb7a776f718/volumes/kubernetes.io~projected/kube-api-access-srkbh DeviceMajor:0 DeviceMinor:65 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-70 DeviceMajor:0 DeviceMinor:70 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-114 DeviceMajor:0 DeviceMinor:114 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-131 DeviceMajor:0 DeviceMinor:131 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/da8395ef-7464-410e-96ee-b27378a03fed/volumes/kubernetes.io~projected/kube-api-access-jkn67 DeviceMajor:0 DeviceMinor:205 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/aa112079-b749-404c-9081-113220707fcd/volumes/kubernetes.io~projected/bound-sa-token DeviceMajor:0 DeviceMinor:208 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-96 DeviceMajor:0 DeviceMinor:96 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-171 DeviceMajor:0 DeviceMinor:171 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/e49453a8-c7d9-4248-a8ef-4a819b76806c/volumes/kubernetes.io~projected/kube-api-access-98n75 DeviceMajor:0 DeviceMinor:232 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/357678e8fb63a8473b0f2c085a256547f8cd0e54439948bcf9e6493824d0ff36/userdata/shm DeviceMajor:0 DeviceMinor:210 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/9122c2fd-f29f-4e74-9615-15078517a3c0/volumes/kubernetes.io~projected/kube-api-access-r4l7n DeviceMajor:0 DeviceMinor:329 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/6fffc15e-94bd-4939-b38f-17d1fe6cc0b5/volumes/kubernetes.io~projected/kube-api-access-dtjj5 DeviceMajor:0 DeviceMinor:361 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/08490837-b3b6-4c2b-a498-568af0d7086d/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:61 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/ff4dcba2-df8d-4324-9e0c-abb23a5e6982/volumes/kubernetes.io~projected/kube-api-access-v9dnm DeviceMajor:0 DeviceMinor:207 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/bba607643ebf9639b0c8044c3e3694561fdacdd3731bcf15feae4311301c9f2b/userdata/shm DeviceMajor:0 DeviceMinor:290 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16406007808 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e5ff0a702058f6c4a964d51275a39426fdf7de02bd4756e66c20fde0c8b3a1d3/userdata/shm DeviceMajor:0 DeviceMinor:47 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/0b3559dbcb56809941ca496ac90b08c271ef0b225c141267043d900f6e7d7564/userdata/shm DeviceMajor:0 DeviceMinor:83 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-256 DeviceMajor:0 DeviceMinor:256 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f273b9e5-0419-4587-a9c7-a0c8bec02acf/volumes/kubernetes.io~projected/kube-api-access-lldxn DeviceMajor:0 DeviceMinor:226 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e459951077da3523c5cc454f50947afe46c379d6e8c40321e83519e9f3c10d61/userdata/shm DeviceMajor:0 DeviceMinor:242 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-248 DeviceMajor:0 DeviceMinor:248 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/7182622ef6dda99f8aed4b16c806aa974d9e90326009d6dc3149e9bd8b4f5cb2/userdata/shm DeviceMajor:0 DeviceMinor:348 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/dd8134f9-b2a6-4542-82ec-c10edc3fcd72/volumes/kubernetes.io~projected/kube-api-access-wsrb6 DeviceMajor:0 DeviceMinor:55 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/073fc7fbb6f5887bd5ad9f67b0ea840f764095bda44fa96f2e5959f1d52f73d5/userdata/shm DeviceMajor:0 DeviceMinor:240 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-294 DeviceMajor:0 DeviceMinor:294 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-273 DeviceMajor:0 DeviceMinor:273 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d55eb70a-f1a8-4346-b8df-ccd2c598dd6b/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:58 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/d55eb70a-f1a8-4346-b8df-ccd2c598dd6b/volumes/kubernetes.io~projected/kube-api-access-ftdwt DeviceMajor:0 DeviceMinor:67 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/105e79f277b5491b41149e986f955c0083288f78b77703e37bbf1cd561db332c/userdata/shm DeviceMajor:0 DeviceMinor:88 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-312 DeviceMajor:0 DeviceMinor:312 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-368 DeviceMajor:0 DeviceMinor:368 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a2d98ed6ecf479a4141b67729bc58f0489ca18f41040a2da19154c4317bcaac5/userdata/shm DeviceMajor:0 DeviceMinor:98 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/eafc69c5-15f1-4e4c-97ca-be586e55fbae/volumes/kubernetes.io~projected/kube-api-access-j78lj DeviceMajor:0 DeviceMinor:223 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a146455d1f30eb2f7f4ad58212035a0c1f733cb997565d267edda0121121fa40/userdata/shm DeviceMajor:0 DeviceMinor:238 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-270 DeviceMajor:0 DeviceMinor:270 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f0c0ada7-7688-4dc2-97fc-813e4d0c4ef3/volumes/kubernetes.io~projected/kube-api-access-zlmlp DeviceMajor:0 DeviceMinor:229 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/91c64ba91f2d616e39faaef286d3e721f03f6ac5c7d010d6b674016160e7aa6e/userdata/shm DeviceMajor:0 DeviceMinor:334 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-352 DeviceMajor:0 DeviceMinor:352 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-366 DeviceMajor:0 DeviceMinor:366 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/9cd1266e-c630-4259-b1f7-1b88dbde7385/volumes/kubernetes.io~projected/kube-api-access-jxmsl DeviceMajor:0 DeviceMinor:57 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/58b0b75a-1568-4703-ac55-fa87f29225ad/volumes/kubernetes.io~projected/kube-api-access-t228s DeviceMajor:0 DeviceMinor:64 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-86 DeviceMajor:0 DeviceMinor:86 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/aa112079-b749-404c-9081-113220707fcd/volumes/kubernetes.io~secret/image-registry-private-configuration DeviceMajor:0 DeviceMinor:202 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/7e90fa928e74f963359d215e04ec3bd27f51c1496663feae053c18839e060ba3/userdata/shm DeviceMajor:0 DeviceMinor:234 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-284 DeviceMajor:0 DeviceMinor:284 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-292 DeviceMajor:0 DeviceMinor:292 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/12bf89ab2c442ecd84c7a7607d703dd213b5513e6a93c92ef2dde5bf06ad6f99/userdata/shm DeviceMajor:0 DeviceMinor:246 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-266 DeviceMajor:0 DeviceMinor:266 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-168 DeviceMajor:0 DeviceMinor:168 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/aa4836a5-c71a-485d-82fc-f8b53dae7717/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:286 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-129 DeviceMajor:0 DeviceMinor:129 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-142 DeviceMajor:0 DeviceMinor:142 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-144 DeviceMajor:0 DeviceMinor:144 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d55eb70a-f1a8-4346-b8df-ccd2c598dd6b/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6562406400 Type:vfs Inodes:819200 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16406011904 Type:vfs Inodes:1048576 HasInodes:true} {Device:/run/containers/storage/overlay-containers/ddade6cb4a39d5c69647385ce4c28b127550d12a98dfe2caade4862845ce5388/userdata/shm DeviceMajor:0 DeviceMinor:94 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/aa112079-b749-404c-9081-113220707fcd/volumes/kubernetes.io~projected/kube-api-access-rpmxj DeviceMajor:0 DeviceMinor:204 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/b66619db-fd81-4435-be2a-a5ea6111e204/volumes/kubernetes.io~projected/kube-api-access-59kdl DeviceMajor:0 DeviceMinor:230 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/3497d23ca426dcc552e2336dd5013e9a8e2a9195710bceeb655f082d4aef724d/userdata/shm DeviceMajor:0 DeviceMinor:244 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-302 DeviceMajor:0 DeviceMinor:302 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-304 DeviceMajor:0 DeviceMinor:304 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/6fffc15e-94bd-4939-b38f-17d1fe6cc0b5/volumes/kubernetes.io~secret/signing-key DeviceMajor:0 DeviceMinor:360 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/74e94b63179e44eb28db4bd8dd65d786876ea1da8d701522187f36b22647470a/userdata/shm DeviceMajor:0 DeviceMinor:68 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/f0c0ada7-7688-4dc2-97fc-813e4d0c4ef3/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:213 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/147bb9af-dfc7-4785-bb5e-906f25d65dd1/volumes/kubernetes.io~projected/kube-api-access-g9wls DeviceMajor:0 DeviceMinor:233 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-254 DeviceMajor:0 DeviceMinor:254 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/e54a9dc6-36e6-4f1b-9c3a-623a520e11da/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:215 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/08490837-b3b6-4c2b-a498-568af0d7086d/volumes/kubernetes.io~projected/kube-api-access-w9tzc DeviceMajor:0 DeviceMinor:63 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/1bfb2f27afea8897b43d2d990b1e208f70c168e0be7c46e2c3a03b21c99b28b0/userdata/shm DeviceMajor:0 DeviceMinor:72 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-74 DeviceMajor:0 DeviceMinor:74 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ff4dcba2-df8d-4324-9e0c-abb23a5e6982/volumes/kubernetes.io~projected/bound-sa-token DeviceMajor:0 DeviceMinor:206 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-336 DeviceMajor:0 DeviceMinor:336 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-111 DeviceMajor:0 DeviceMinor:111 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-152 DeviceMajor:0 DeviceMinor:152 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6562406400 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:overlay_0-90 DeviceMajor:0 DeviceMinor:90 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/run/containers/storage/overlay-containers/075a77de8990101732600f618149824f4c9470f0e266030d9b437c8c27451e87/userdata/shm DeviceMajor:0 DeviceMinor:219 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/c2be658b-ec92-4782-a78b-d2dccf2249cb/volumes/kubernetes.io~projected/kube-api-access-bjgwm DeviceMajor:0 DeviceMinor:227 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/08490837-b3b6-4c2b-a498-568af0d7086d/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:60 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-164 DeviceMajor:0 DeviceMinor:164 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ff4dcba2-df8d-4324-9e0c-abb23a5e6982/volumes/kubernetes.io~secret/image-registry-private-configuration DeviceMajor:0 DeviceMinor:45 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/c2be658b-ec92-4782-a78b-d2dccf2249cb/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:212 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/69ff2b62-4588-4cb0-bdad-747c3692138c/volumes/kubernetes.io~projected/kube-api-access-6qc9x DeviceMajor:0 DeviceMinor:224 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/306f16787f3392fc8b7ace0026e1c8efe17179d370cdd50b03d02cb649070aaa/userdata/shm DeviceMajor:0 DeviceMinor:346 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-350 DeviceMajor:0 DeviceMinor:350 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-258 DeviceMajor:0 DeviceMinor:258 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:073fc7fbb6f5887 MacAddress:fa:d2:e6:03:1f:5d Speed:10000 Mtu:8901} {Name:075a77de8990101 MacAddress:16:04:c4:bd:1d:6d Speed:10000 Mtu:8901} {Name:12bf89ab2c442ec MacAddress:de:f3:a9:1a:dd:36 Speed:10000 Mtu:8901} {Name:306f16787f3392f MacAddress:32:42:c9:a7:49:79 Speed:10000 Mtu:8901} {Name:3497d23ca426dcc MacAddress:92:10:52:95:2e:3e Speed:10000 Mtu:8901} {Name:357678e8fb63a84 MacAddress:46:97:dc:f5:2d:76 Speed:10000 Mtu:8901} {Name:5073165fba6902e MacAddress:22:5d:38:76:31:86 Speed:10000 Mtu:8901} {Name:6e130a212380a09 MacAddress:8e:53:c5:8a:5d:d3 Speed:10000 Mtu:8901} {Name:7182622ef6dda99 MacAddress:ae:42:18:c7:dd:82 Speed:10000 Mtu:8901} {Name:7e90fa928e74f96 MacAddress:8e:c1:10:ad:70:8d Speed:10000 Mtu:8901} {Name:91c64ba91f2d616 MacAddress:ce:bb:8d:ac:ff:7f Speed:10000 Mtu:8901} {Name:9b0ccc77fd55161 MacAddress:2a:1b:27:14:95:4b Speed:10000 Mtu:8901} {Name:a146455d1f30eb2 MacAddress:ce:4b:89:1a:e9:49 Speed:10000 Mtu:8901} {Name:bba607643ebf963 MacAddress:b2:cb:46:3d:62:b2 Speed:10000 Mtu:8901} {Name:br-ex MacAddress:02:32:9d:23:68:b5 Speed:0 Mtu:9001} {Name:br-int MacAddress:62:86:aa:a6:33:57 Speed:0 Mtu:8901} {Name:e459951077da352 MacAddress:8e:c2:23:d0:4f:a0 Speed:10000 Mtu:8901} {Name:e7be247cfa0d4a3 MacAddress:22:f7:4e:5a:12:a0 Speed:10000 Mtu:8901} {Name:ens5 MacAddress:02:32:9d:23:68:b5 Speed:0 Mtu:9001} {Name:genev_sys_6081 MacAddress:12:d4:2c:78:02:9e Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:84:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:ea:76:2d:1d:e3:7a Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812019712 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Oct 31 01:03:51.621018 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:51.621020    4964 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Oct 31 01:03:51.621457 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:51.621173    4964 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Oct 31 01:03:51.621626 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:51.621589    4964 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Oct 31 01:03:51.621937 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:51.621628    4964 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-133-197.us-west-1.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Oct 31 01:03:51.622005 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:51.621942    4964 topology_manager.go:143] "Creating topology manager with none policy"
Oct 31 01:03:51.622005 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:51.621955    4964 container_manager_linux.go:311] "Creating device plugin manager"
Oct 31 01:03:51.622005 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:51.621972    4964 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 31 01:03:51.622005 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:51.621999    4964 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 31 01:03:51.622173 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:51.622072    4964 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Oct 31 01:03:51.622333 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:51.622303    4964 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Oct 31 01:03:51.622680 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:51.622669    4964 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Oct 31 01:03:51.622724 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:51.622712    4964 kubelet.go:493] "Attempting to sync node with API server"
Oct 31 01:03:51.622766 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:51.622727    4964 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Oct 31 01:03:51.622766 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:51.622747    4964 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Oct 31 01:03:51.622766 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:51.622760    4964 kubelet.go:405] "Adding apiserver pod source"
Oct 31 01:03:51.622884 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:51.622771    4964 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Oct 31 01:03:51.625891 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:51.623804    4964 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Oct 31 01:03:51.625891 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:51.623819    4964 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 31 01:03:51.629515 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:51.626222    4964 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Oct 31 01:03:51.716767 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:51.716749    4964 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Oct 31 01:03:51.716767 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:51.716765    4964 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 31 01:03:51.716921 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:51.716875    4964 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem"
Oct 31 01:03:51.717076 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:51.717065    4964 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Oct 31 01:03:51.717123 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:51.717094    4964 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Oct 31 01:03:51.717225 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:51.717210    4964 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Oct 31 01:03:51.717300 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:51.717234    4964 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Oct 31 01:03:51.717300 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:51.717244    4964 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Oct 31 01:03:51.717300 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:51.717250    4964 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Oct 31 01:03:51.717300 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:51.717256    4964 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Oct 31 01:03:51.717300 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:51.717262    4964 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Oct 31 01:03:51.717300 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:51.717267    4964 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Oct 31 01:03:51.717300 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:51.717274    4964 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Oct 31 01:03:51.717300 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:51.717292    4964 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Oct 31 01:03:51.717300 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:51.717298    4964 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Oct 31 01:03:51.717607 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:51.717335    4964 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Oct 31 01:03:51.717607 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:51.717347    4964 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Oct 31 01:03:51.717607 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:51.717377    4964 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Oct 31 01:03:51.717607 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:51.717385    4964 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Oct 31 01:03:51.724209 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:51.721366    4964 server.go:1263] "Started kubelet"
Oct 31 01:03:51.724209 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:51.721807    4964 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Oct 31 01:03:51.724209 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:51.721803    4964 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Oct 31 01:03:51.724209 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:51.721877    4964 server_v1.go:49] "podresources" method="list" useActivePods=true
Oct 31 01:03:51.724209 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:51.722279    4964 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Oct 31 01:03:51.724209 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:51.722986    4964 server.go:317] "Adding debug handlers to kubelet server"
Oct 31 01:03:51.722593 ip-10-0-133-197 systemd[1]: Started Kubernetes Kubelet.
Oct 31 01:03:51.725300 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:51.725274    4964 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Oct 31 01:03:51.728770 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:51.726090    4964 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Oct 31 01:03:51.728770 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:51.726267    4964 volume_manager.go:309] "The desired_state_of_world populator starts"
Oct 31 01:03:51.728770 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:51.726280    4964 volume_manager.go:311] "Starting Kubelet Volume Manager"
Oct 31 01:03:51.728770 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:51.726430    4964 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Oct 31 01:03:51.728770 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:51.728731    4964 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-30 00:58:11 +0000 UTC" deadline="2028-05-30 21:40:50.658834181 +0000 UTC"
Oct 31 01:03:51.728770 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:51.728768    4964 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13868h36m58.930068965s"
Oct 31 01:03:51.729124 ip-10-0-133-197 kubenswrapper[4964]: E1031 01:03:51.729101    4964 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-197.us-west-1.compute.internal\" not found"
Oct 31 01:03:51.735128 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:51.735110    4964 factory.go:55] Registering systemd factory
Oct 31 01:03:51.735251 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:51.735241    4964 factory.go:223] Registration of the systemd container factory successfully
Oct 31 01:03:51.737286 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:51.736613    4964 factory.go:153] Registering CRI-O factory
Oct 31 01:03:51.737286 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:51.736633    4964 factory.go:223] Registration of the crio container factory successfully
Oct 31 01:03:51.737286 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:51.736682    4964 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Oct 31 01:03:51.737286 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:51.736702    4964 factory.go:103] Registering Raw factory
Oct 31 01:03:51.737286 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:51.736716    4964 manager.go:1199] Started watching for new ooms in manager
Oct 31 01:03:51.737286 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:51.737084    4964 manager.go:319] Starting recovery of all containers
Oct 31 01:03:51.738335 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:51.737789    4964 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c8e98df5-b2f7-49dd-a565-ed60534da496" volumeName="kubernetes.io/empty-dir/c8e98df5-b2f7-49dd-a565-ed60534da496-snapshots" seLinuxMountContext=""
Oct 31 01:03:51.738335 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:51.737826    4964 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e49453a8-c7d9-4248-a8ef-4a819b76806c" volumeName="kubernetes.io/secret/e49453a8-c7d9-4248-a8ef-4a819b76806c-stats-auth" seLinuxMountContext=""
Oct 31 01:03:51.738335 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:51.737846    4964 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ff4dcba2-df8d-4324-9e0c-abb23a5e6982" volumeName="kubernetes.io/empty-dir/ff4dcba2-df8d-4324-9e0c-abb23a5e6982-ca-trust-extracted" seLinuxMountContext=""
Oct 31 01:03:51.738335 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:51.737860    4964 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b5975856-8eb3-4601-a7e1-d1b1b3a957e8" volumeName="kubernetes.io/projected/b5975856-8eb3-4601-a7e1-d1b1b3a957e8-kube-api-access-qzhpr" seLinuxMountContext=""
Oct 31 01:03:51.738335 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:51.737876    4964 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b66619db-fd81-4435-be2a-a5ea6111e204" volumeName="kubernetes.io/configmap/b66619db-fd81-4435-be2a-a5ea6111e204-telemetry-config" seLinuxMountContext=""
Oct 31 01:03:51.738335 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:51.737887    4964 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d55eb70a-f1a8-4346-b8df-ccd2c598dd6b" volumeName="kubernetes.io/secret/d55eb70a-f1a8-4346-b8df-ccd2c598dd6b-ovn-node-metrics-cert" seLinuxMountContext=""
Oct 31 01:03:51.738335 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:51.737899    4964 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ff4dcba2-df8d-4324-9e0c-abb23a5e6982" volumeName="kubernetes.io/configmap/ff4dcba2-df8d-4324-9e0c-abb23a5e6982-trusted-ca" seLinuxMountContext=""
Oct 31 01:03:51.738335 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:51.737916    4964 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="58b0b75a-1568-4703-ac55-fa87f29225ad" volumeName="kubernetes.io/configmap/58b0b75a-1568-4703-ac55-fa87f29225ad-multus-daemon-config" seLinuxMountContext=""
Oct 31 01:03:51.738335 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:51.737931    4964 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6fffc15e-94bd-4939-b38f-17d1fe6cc0b5" volumeName="kubernetes.io/projected/6fffc15e-94bd-4939-b38f-17d1fe6cc0b5-kube-api-access-dtjj5" seLinuxMountContext=""
Oct 31 01:03:51.738335 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:51.737990    4964 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f273b9e5-0419-4587-a9c7-a0c8bec02acf" volumeName="kubernetes.io/configmap/f273b9e5-0419-4587-a9c7-a0c8bec02acf-samples-operator-config" seLinuxMountContext=""
Oct 31 01:03:51.738335 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:51.738006    4964 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d55eb70a-f1a8-4346-b8df-ccd2c598dd6b" volumeName="kubernetes.io/configmap/d55eb70a-f1a8-4346-b8df-ccd2c598dd6b-ovnkube-config" seLinuxMountContext=""
Oct 31 01:03:51.738335 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:51.738022    4964 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9122c2fd-f29f-4e74-9615-15078517a3c0" volumeName="kubernetes.io/empty-dir/9122c2fd-f29f-4e74-9615-15078517a3c0-archives-path" seLinuxMountContext=""
Oct 31 01:03:51.738335 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:51.738034    4964 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0ab3954e-be64-4260-bca8-ff5d58bd6ea2" volumeName="kubernetes.io/projected/0ab3954e-be64-4260-bca8-ff5d58bd6ea2-kube-api-access-vms79" seLinuxMountContext=""
Oct 31 01:03:51.738335 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:51.738056    4964 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7dd98c21-e0b1-4289-8fe8-036bf413d444" volumeName="kubernetes.io/empty-dir/7dd98c21-e0b1-4289-8fe8-036bf413d444-data-volume" seLinuxMountContext=""
Oct 31 01:03:51.738335 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:51.738102    4964 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8952fc79-c5a6-42ba-a296-8cb7a776f718" volumeName="kubernetes.io/projected/8952fc79-c5a6-42ba-a296-8cb7a776f718-kube-api-access-srkbh" seLinuxMountContext=""
Oct 31 01:03:51.738335 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:51.738130    4964 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="aa112079-b749-404c-9081-113220707fcd" volumeName="kubernetes.io/projected/aa112079-b749-404c-9081-113220707fcd-bound-sa-token" seLinuxMountContext=""
Oct 31 01:03:51.738335 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:51.738161    4964 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="aa112079-b749-404c-9081-113220707fcd" volumeName="kubernetes.io/secret/aa112079-b749-404c-9081-113220707fcd-image-registry-private-configuration" seLinuxMountContext=""
Oct 31 01:03:51.738335 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:51.738173    4964 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e54a9dc6-36e6-4f1b-9c3a-623a520e11da" volumeName="kubernetes.io/configmap/e54a9dc6-36e6-4f1b-9c3a-623a520e11da-config" seLinuxMountContext=""
Oct 31 01:03:51.738335 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:51.738189    4964 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="08490837-b3b6-4c2b-a498-568af0d7086d" volumeName="kubernetes.io/empty-dir/08490837-b3b6-4c2b-a498-568af0d7086d-etc-tuned" seLinuxMountContext=""
Oct 31 01:03:51.738335 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:51.738209    4964 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="57c2f196-0d8a-4bce-a5bc-3b6b5c0d808c" volumeName="kubernetes.io/projected/57c2f196-0d8a-4bce-a5bc-3b6b5c0d808c-kube-api-access-2dx97" seLinuxMountContext=""
Oct 31 01:03:51.738335 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:51.738225    4964 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9122c2fd-f29f-4e74-9615-15078517a3c0" volumeName="kubernetes.io/configmap/9122c2fd-f29f-4e74-9615-15078517a3c0-trusted-ca-bundle" seLinuxMountContext=""
Oct 31 01:03:51.738335 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:51.738256    4964 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c2be658b-ec92-4782-a78b-d2dccf2249cb" volumeName="kubernetes.io/configmap/c2be658b-ec92-4782-a78b-d2dccf2249cb-config" seLinuxMountContext=""
Oct 31 01:03:51.738335 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:51.738268    4964 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e49453a8-c7d9-4248-a8ef-4a819b76806c" volumeName="kubernetes.io/secret/e49453a8-c7d9-4248-a8ef-4a819b76806c-default-certificate" seLinuxMountContext=""
Oct 31 01:03:51.738335 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:51.738292    4964 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="eafc69c5-15f1-4e4c-97ca-be586e55fbae" volumeName="kubernetes.io/projected/eafc69c5-15f1-4e4c-97ca-be586e55fbae-kube-api-access-j78lj" seLinuxMountContext=""
Oct 31 01:03:51.738335 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:51.738304    4964 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="85bb69db-c69b-467e-adeb-950bf2947c94" volumeName="kubernetes.io/configmap/85bb69db-c69b-467e-adeb-950bf2947c94-config-volume" seLinuxMountContext=""
Oct 31 01:03:51.739690 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:51.739539    4964 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9cd1266e-c630-4259-b1f7-1b88dbde7385" volumeName="kubernetes.io/configmap/9cd1266e-c630-4259-b1f7-1b88dbde7385-iptables-alerter-script" seLinuxMountContext=""
Oct 31 01:03:51.739690 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:51.739572    4964 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="aa112079-b749-404c-9081-113220707fcd" volumeName="kubernetes.io/configmap/aa112079-b749-404c-9081-113220707fcd-registry-certificates" seLinuxMountContext=""
Oct 31 01:03:51.739690 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:51.739618    4964 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b66619db-fd81-4435-be2a-a5ea6111e204" volumeName="kubernetes.io/projected/b66619db-fd81-4435-be2a-a5ea6111e204-kube-api-access-59kdl" seLinuxMountContext=""
Oct 31 01:03:51.739690 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:51.739645    4964 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d719eda9-bc9f-4307-952a-99d99aa0f62b" volumeName="kubernetes.io/projected/d719eda9-bc9f-4307-952a-99d99aa0f62b-kube-api-access-rz545" seLinuxMountContext=""
Oct 31 01:03:51.741034 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:51.739891    4964 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f0c0ada7-7688-4dc2-97fc-813e4d0c4ef3" volumeName="kubernetes.io/projected/f0c0ada7-7688-4dc2-97fc-813e4d0c4ef3-kube-api-access-zlmlp" seLinuxMountContext=""
Oct 31 01:03:51.741034 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:51.739915    4964 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f8855c2c-87b4-4b9e-b28a-227981205b11" volumeName="kubernetes.io/configmap/f8855c2c-87b4-4b9e-b28a-227981205b11-konnectivity-ca" seLinuxMountContext=""
Oct 31 01:03:51.741034 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:51.739963    4964 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ff4dcba2-df8d-4324-9e0c-abb23a5e6982" volumeName="kubernetes.io/secret/ff4dcba2-df8d-4324-9e0c-abb23a5e6982-installation-pull-secrets" seLinuxMountContext=""
Oct 31 01:03:51.741034 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:51.739977    4964 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c8e98df5-b2f7-49dd-a565-ed60534da496" volumeName="kubernetes.io/empty-dir/c8e98df5-b2f7-49dd-a565-ed60534da496-tmp" seLinuxMountContext=""
Oct 31 01:03:51.741034 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:51.740015    4964 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cb657366-6696-4c3e-a3a3-603ce00cf11a" volumeName="kubernetes.io/projected/cb657366-6696-4c3e-a3a3-603ce00cf11a-kube-api-access-bspb5" seLinuxMountContext=""
Oct 31 01:03:51.741034 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:51.740033    4964 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e54a9dc6-36e6-4f1b-9c3a-623a520e11da" volumeName="kubernetes.io/projected/e54a9dc6-36e6-4f1b-9c3a-623a520e11da-kube-api-access-ks6fn" seLinuxMountContext=""
Oct 31 01:03:51.741034 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:51.740045    4964 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fa085dd8-ca1e-4ec5-acb2-9b9dc878bf8a" volumeName="kubernetes.io/configmap/fa085dd8-ca1e-4ec5-acb2-9b9dc878bf8a-cni-sysctl-allowlist" seLinuxMountContext=""
Oct 31 01:03:51.741034 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:51.740084    4964 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="85bb69db-c69b-467e-adeb-950bf2947c94" volumeName="kubernetes.io/projected/85bb69db-c69b-467e-adeb-950bf2947c94-kube-api-access-8n8tt" seLinuxMountContext=""
Oct 31 01:03:51.741034 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:51.740108    4964 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d55eb70a-f1a8-4346-b8df-ccd2c598dd6b" volumeName="kubernetes.io/configmap/d55eb70a-f1a8-4346-b8df-ccd2c598dd6b-ovnkube-script-lib" seLinuxMountContext=""
Oct 31 01:03:51.741034 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:51.740143    4964 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e54a9dc6-36e6-4f1b-9c3a-623a520e11da" volumeName="kubernetes.io/secret/e54a9dc6-36e6-4f1b-9c3a-623a520e11da-serving-cert" seLinuxMountContext=""
Oct 31 01:03:51.741034 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:51.740184    4964 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f8855c2c-87b4-4b9e-b28a-227981205b11" volumeName="kubernetes.io/secret/f8855c2c-87b4-4b9e-b28a-227981205b11-agent-certs" seLinuxMountContext=""
Oct 31 01:03:51.741034 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:51.740197    4964 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ff4dcba2-df8d-4324-9e0c-abb23a5e6982" volumeName="kubernetes.io/projected/ff4dcba2-df8d-4324-9e0c-abb23a5e6982-bound-sa-token" seLinuxMountContext=""
Oct 31 01:03:51.741034 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:51.740212    4964 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="565fefaf-82aa-42a8-9cc7-6517c4bddf1e" volumeName="kubernetes.io/projected/565fefaf-82aa-42a8-9cc7-6517c4bddf1e-kube-api-access-r6zx9" seLinuxMountContext=""
Oct 31 01:03:51.741034 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:51.740223    4964 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9122c2fd-f29f-4e74-9615-15078517a3c0" volumeName="kubernetes.io/projected/9122c2fd-f29f-4e74-9615-15078517a3c0-kube-api-access-r4l7n" seLinuxMountContext=""
Oct 31 01:03:51.741034 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:51.740247    4964 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9cd1266e-c630-4259-b1f7-1b88dbde7385" volumeName="kubernetes.io/projected/9cd1266e-c630-4259-b1f7-1b88dbde7385-kube-api-access-jxmsl" seLinuxMountContext=""
Oct 31 01:03:51.741034 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:51.740264    4964 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="aa112079-b749-404c-9081-113220707fcd" volumeName="kubernetes.io/configmap/aa112079-b749-404c-9081-113220707fcd-trusted-ca" seLinuxMountContext=""
Oct 31 01:03:51.741034 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:51.740286    4964 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="aa112079-b749-404c-9081-113220707fcd" volumeName="kubernetes.io/projected/aa112079-b749-404c-9081-113220707fcd-kube-api-access-rpmxj" seLinuxMountContext=""
Oct 31 01:03:51.741034 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:51.740302    4964 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c8e98df5-b2f7-49dd-a565-ed60534da496" volumeName="kubernetes.io/projected/c8e98df5-b2f7-49dd-a565-ed60534da496-kube-api-access-tlnlx" seLinuxMountContext=""
Oct 31 01:03:51.741034 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:51.740369    4964 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d55eb70a-f1a8-4346-b8df-ccd2c598dd6b" volumeName="kubernetes.io/configmap/d55eb70a-f1a8-4346-b8df-ccd2c598dd6b-env-overrides" seLinuxMountContext=""
Oct 31 01:03:51.741034 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:51.740383    4964 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f0c0ada7-7688-4dc2-97fc-813e4d0c4ef3" volumeName="kubernetes.io/secret/f0c0ada7-7688-4dc2-97fc-813e4d0c4ef3-serving-cert" seLinuxMountContext=""
Oct 31 01:03:51.741034 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:51.740414    4964 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="08490837-b3b6-4c2b-a498-568af0d7086d" volumeName="kubernetes.io/empty-dir/08490837-b3b6-4c2b-a498-568af0d7086d-tmp" seLinuxMountContext=""
Oct 31 01:03:51.741034 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:51.740426    4964 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="147bb9af-dfc7-4785-bb5e-906f25d65dd1" volumeName="kubernetes.io/projected/147bb9af-dfc7-4785-bb5e-906f25d65dd1-kube-api-access-g9wls" seLinuxMountContext=""
Oct 31 01:03:51.741034 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:51.740458    4964 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fa085dd8-ca1e-4ec5-acb2-9b9dc878bf8a" volumeName="kubernetes.io/configmap/fa085dd8-ca1e-4ec5-acb2-9b9dc878bf8a-cni-binary-copy" seLinuxMountContext=""
Oct 31 01:03:51.741034 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:51.740470    4964 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ff4dcba2-df8d-4324-9e0c-abb23a5e6982" volumeName="kubernetes.io/configmap/ff4dcba2-df8d-4324-9e0c-abb23a5e6982-registry-certificates" seLinuxMountContext=""
Oct 31 01:03:51.741034 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:51.740490    4964 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="318b3bc6-6e59-4211-af63-b116526a325d" volumeName="kubernetes.io/projected/318b3bc6-6e59-4211-af63-b116526a325d-kube-api-access-rms4s" seLinuxMountContext=""
Oct 31 01:03:51.741034 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:51.740504    4964 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="aa4836a5-c71a-485d-82fc-f8b53dae7717" volumeName="kubernetes.io/secret/aa4836a5-c71a-485d-82fc-f8b53dae7717-original-pull-secret" seLinuxMountContext=""
Oct 31 01:03:51.741034 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:51.740543    4964 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c2be658b-ec92-4782-a78b-d2dccf2249cb" volumeName="kubernetes.io/secret/c2be658b-ec92-4782-a78b-d2dccf2249cb-serving-cert" seLinuxMountContext=""
Oct 31 01:03:51.741034 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:51.740596    4964 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fa085dd8-ca1e-4ec5-acb2-9b9dc878bf8a" volumeName="kubernetes.io/configmap/fa085dd8-ca1e-4ec5-acb2-9b9dc878bf8a-whereabouts-flatfile-configmap" seLinuxMountContext=""
Oct 31 01:03:51.741034 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:51.740629    4964 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ff4dcba2-df8d-4324-9e0c-abb23a5e6982" volumeName="kubernetes.io/projected/ff4dcba2-df8d-4324-9e0c-abb23a5e6982-kube-api-access-v9dnm" seLinuxMountContext=""
Oct 31 01:03:51.741034 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:51.740666    4964 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6fffc15e-94bd-4939-b38f-17d1fe6cc0b5" volumeName="kubernetes.io/secret/6fffc15e-94bd-4939-b38f-17d1fe6cc0b5-signing-key" seLinuxMountContext=""
Oct 31 01:03:51.741034 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:51.740708    4964 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9dfae6d6-b9a6-4d85-b594-9d947a9e6383" volumeName="kubernetes.io/configmap/9dfae6d6-b9a6-4d85-b594-9d947a9e6383-nginx-conf" seLinuxMountContext=""
Oct 31 01:03:51.741034 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:51.740720    4964 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c2be658b-ec92-4782-a78b-d2dccf2249cb" volumeName="kubernetes.io/projected/c2be658b-ec92-4782-a78b-d2dccf2249cb-kube-api-access-bjgwm" seLinuxMountContext=""
Oct 31 01:03:51.741034 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:51.740743    4964 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f0c0ada7-7688-4dc2-97fc-813e4d0c4ef3" volumeName="kubernetes.io/configmap/f0c0ada7-7688-4dc2-97fc-813e4d0c4ef3-trusted-ca" seLinuxMountContext=""
Oct 31 01:03:51.741034 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:51.740759    4964 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="58b0b75a-1568-4703-ac55-fa87f29225ad" volumeName="kubernetes.io/projected/58b0b75a-1568-4703-ac55-fa87f29225ad-kube-api-access-t228s" seLinuxMountContext=""
Oct 31 01:03:51.741034 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:51.740792    4964 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="85bb69db-c69b-467e-adeb-950bf2947c94" volumeName="kubernetes.io/empty-dir/85bb69db-c69b-467e-adeb-950bf2947c94-tmp-dir" seLinuxMountContext=""
Oct 31 01:03:51.741034 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:51.740809    4964 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c8e98df5-b2f7-49dd-a565-ed60534da496" volumeName="kubernetes.io/configmap/c8e98df5-b2f7-49dd-a565-ed60534da496-trusted-ca-bundle" seLinuxMountContext=""
Oct 31 01:03:51.741034 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:51.740834    4964 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d55eb70a-f1a8-4346-b8df-ccd2c598dd6b" volumeName="kubernetes.io/projected/d55eb70a-f1a8-4346-b8df-ccd2c598dd6b-kube-api-access-ftdwt" seLinuxMountContext=""
Oct 31 01:03:51.741034 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:51.740850    4964 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="dd8134f9-b2a6-4542-82ec-c10edc3fcd72" volumeName="kubernetes.io/projected/dd8134f9-b2a6-4542-82ec-c10edc3fcd72-kube-api-access-wsrb6" seLinuxMountContext=""
Oct 31 01:03:51.741034 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:51.740873    4964 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e49453a8-c7d9-4248-a8ef-4a819b76806c" volumeName="kubernetes.io/projected/e49453a8-c7d9-4248-a8ef-4a819b76806c-kube-api-access-98n75" seLinuxMountContext=""
Oct 31 01:03:51.741034 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:51.740886    4964 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f273b9e5-0419-4587-a9c7-a0c8bec02acf" volumeName="kubernetes.io/projected/f273b9e5-0419-4587-a9c7-a0c8bec02acf-kube-api-access-lldxn" seLinuxMountContext=""
Oct 31 01:03:51.741034 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:51.740918    4964 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="aa112079-b749-404c-9081-113220707fcd" volumeName="kubernetes.io/secret/aa112079-b749-404c-9081-113220707fcd-installation-pull-secrets" seLinuxMountContext=""
Oct 31 01:03:51.741034 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:51.740933    4964 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c8e98df5-b2f7-49dd-a565-ed60534da496" volumeName="kubernetes.io/secret/c8e98df5-b2f7-49dd-a565-ed60534da496-serving-cert" seLinuxMountContext=""
Oct 31 01:03:51.741034 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:51.740962    4964 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="da8395ef-7464-410e-96ee-b27378a03fed" volumeName="kubernetes.io/projected/da8395ef-7464-410e-96ee-b27378a03fed-kube-api-access-jkn67" seLinuxMountContext=""
Oct 31 01:03:51.741034 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:51.740997    4964 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="dd8134f9-b2a6-4542-82ec-c10edc3fcd72" volumeName="kubernetes.io/configmap/dd8134f9-b2a6-4542-82ec-c10edc3fcd72-serviceca" seLinuxMountContext=""
Oct 31 01:03:51.744568 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:51.744359    4964 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f0c0ada7-7688-4dc2-97fc-813e4d0c4ef3" volumeName="kubernetes.io/configmap/f0c0ada7-7688-4dc2-97fc-813e4d0c4ef3-config" seLinuxMountContext=""
Oct 31 01:03:51.744568 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:51.744408    4964 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ff4dcba2-df8d-4324-9e0c-abb23a5e6982" volumeName="kubernetes.io/secret/ff4dcba2-df8d-4324-9e0c-abb23a5e6982-image-registry-private-configuration" seLinuxMountContext=""
Oct 31 01:03:51.744568 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:51.744444    4964 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6fffc15e-94bd-4939-b38f-17d1fe6cc0b5" volumeName="kubernetes.io/configmap/6fffc15e-94bd-4939-b38f-17d1fe6cc0b5-signing-cabundle" seLinuxMountContext=""
Oct 31 01:03:51.744568 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:51.744484    4964 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7dd98c21-e0b1-4289-8fe8-036bf413d444" volumeName="kubernetes.io/projected/7dd98c21-e0b1-4289-8fe8-036bf413d444-kube-api-access-7fhrx" seLinuxMountContext=""
Oct 31 01:03:51.744568 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:51.744487    4964 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-9257n_f0c0ada7-7688-4dc2-97fc-813e4d0c4ef3/console-operator/1.log"
Oct 31 01:03:51.744568 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:51.744499    4964 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="08490837-b3b6-4c2b-a498-568af0d7086d" volumeName="kubernetes.io/projected/08490837-b3b6-4c2b-a498-568af0d7086d-kube-api-access-w9tzc" seLinuxMountContext=""
Oct 31 01:03:51.744568 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:51.744522    4964 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="147bb9af-dfc7-4785-bb5e-906f25d65dd1" volumeName="kubernetes.io/secret/147bb9af-dfc7-4785-bb5e-906f25d65dd1-cert" seLinuxMountContext=""
Oct 31 01:03:51.744875 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:51.744572    4964 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="58b0b75a-1568-4703-ac55-fa87f29225ad" volumeName="kubernetes.io/configmap/58b0b75a-1568-4703-ac55-fa87f29225ad-cni-binary-copy" seLinuxMountContext=""
Oct 31 01:03:51.744875 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:51.743292    4964 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-9257n_f0c0ada7-7688-4dc2-97fc-813e4d0c4ef3/console-operator/1.log"
Oct 31 01:03:51.745005 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:51.744926    4964 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="69ff2b62-4588-4cb0-bdad-747c3692138c" volumeName="kubernetes.io/projected/69ff2b62-4588-4cb0-bdad-747c3692138c-kube-api-access-6qc9x" seLinuxMountContext=""
Oct 31 01:03:51.745005 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:51.744987    4964 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="aa112079-b749-404c-9081-113220707fcd" volumeName="kubernetes.io/empty-dir/aa112079-b749-404c-9081-113220707fcd-ca-trust-extracted" seLinuxMountContext=""
Oct 31 01:03:51.745113 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:51.745014    4964 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c8e98df5-b2f7-49dd-a565-ed60534da496" volumeName="kubernetes.io/configmap/c8e98df5-b2f7-49dd-a565-ed60534da496-service-ca-bundle" seLinuxMountContext=""
Oct 31 01:03:51.745113 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:51.745040    4964 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fa085dd8-ca1e-4ec5-acb2-9b9dc878bf8a" volumeName="kubernetes.io/projected/fa085dd8-ca1e-4ec5-acb2-9b9dc878bf8a-kube-api-access-nmlm7" seLinuxMountContext=""
Oct 31 01:03:51.745113 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:51.745093    4964 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="57c2f196-0d8a-4bce-a5bc-3b6b5c0d808c" volumeName="kubernetes.io/empty-dir/57c2f196-0d8a-4bce-a5bc-3b6b5c0d808c-tmp-dir" seLinuxMountContext=""
Oct 31 01:03:51.745242 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:51.745106    4964 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9122c2fd-f29f-4e74-9615-15078517a3c0" volumeName="kubernetes.io/configmap/9122c2fd-f29f-4e74-9615-15078517a3c0-service-ca-bundle" seLinuxMountContext=""
Oct 31 01:03:51.745242 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:51.745127    4964 reconstruct.go:97] "Volume reconstruction finished"
Oct 31 01:03:51.745242 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:51.745135    4964 reconciler.go:29] "Reconciler: start to sync state"
Oct 31 01:03:51.766899 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:51.766863    4964 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Oct 31 01:03:51.768569 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:51.768552    4964 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Oct 31 01:03:51.768569 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:51.768571    4964 status_manager.go:249] "Starting to sync pod status with apiserver"
Oct 31 01:03:51.768693 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:51.768593    4964 kubelet.go:2521] "Starting kubelet main sync loop"
Oct 31 01:03:51.768693 ip-10-0-133-197 kubenswrapper[4964]: E1031 01:03:51.768630    4964 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]"
Oct 31 01:03:51.783535 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:51.783513    4964 generic.go:356] "Generic (PLEG): container finished" podID="107fc5ddbeb783edeed66dfa339956e5" containerID="14ef005331ab078378b218b4578003c218440dedf09c4fc214b9e46021669c2e" exitCode=0
Oct 31 01:03:51.801633 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:51.801610    4964 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-9257n_f0c0ada7-7688-4dc2-97fc-813e4d0c4ef3/console-operator/1.log"
Oct 31 01:03:51.801712 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:51.801645    4964 generic.go:356] "Generic (PLEG): container finished" podID="f0c0ada7-7688-4dc2-97fc-813e4d0c4ef3" containerID="46df6877e0e8bb148e755797ac3b08c39be418ccb7326153e8ea50c8505fe54b" exitCode=255
Oct 31 01:03:51.829223 ip-10-0-133-197 kubenswrapper[4964]: E1031 01:03:51.829201    4964 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-197.us-west-1.compute.internal\" not found"
Oct 31 01:03:51.830557 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:51.830535    4964 generic.go:356] "Generic (PLEG): container finished" podID="fa085dd8-ca1e-4ec5-acb2-9b9dc878bf8a" containerID="3ea05b5bb5f802eca92bcb6694d410cc0e27110fcb501abbe2054d0a7463e944" exitCode=0
Oct 31 01:03:51.830557 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:51.830557    4964 generic.go:356] "Generic (PLEG): container finished" podID="fa085dd8-ca1e-4ec5-acb2-9b9dc878bf8a" containerID="4949c5e8a1cb38f44dc8723d4446f6fcd6389b7b18951b8ef2eddc740e55f22f" exitCode=0
Oct 31 01:03:51.830702 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:51.830565    4964 generic.go:356] "Generic (PLEG): container finished" podID="fa085dd8-ca1e-4ec5-acb2-9b9dc878bf8a" containerID="84d45cdffaa60e8b1f6749d33a39b43d58bf6beb65ed560f586126b5ba104f87" exitCode=0
Oct 31 01:03:51.830702 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:51.830571    4964 generic.go:356] "Generic (PLEG): container finished" podID="fa085dd8-ca1e-4ec5-acb2-9b9dc878bf8a" containerID="fad5072047ce37042e99f2f3cf2719e6c5dcc87e827fd9f0c3c0f1db6688b2ce" exitCode=0
Oct 31 01:03:51.830702 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:51.830576    4964 generic.go:356] "Generic (PLEG): container finished" podID="fa085dd8-ca1e-4ec5-acb2-9b9dc878bf8a" containerID="f7dfaca6eb74d3c92ab4a0a332937e5ade4d9a209a977efbc75c7f0312bac8f4" exitCode=0
Oct 31 01:03:51.830702 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:51.830583    4964 generic.go:356] "Generic (PLEG): container finished" podID="fa085dd8-ca1e-4ec5-acb2-9b9dc878bf8a" containerID="b8a7d02a9cbe67bba3e73c26cc7fa100da57b1359175a1e04d6d4f9c5b2763de" exitCode=0
Oct 31 01:03:51.835421 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:51.835397    4964 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 31 01:03:51.837186 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:51.837147    4964 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 31 01:03:51.844615 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:51.844597    4964 manager.go:324] Recovery completed
Oct 31 01:03:51.865810 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:51.865793    4964 cpu_manager.go:226] "Starting" policy="none"
Oct 31 01:03:51.865810 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:51.865807    4964 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Oct 31 01:03:51.865913 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:51.865822    4964 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Oct 31 01:03:51.866000 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:51.865958    4964 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet=""
Oct 31 01:03:51.866000 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:51.865971    4964 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={}
Oct 31 01:03:51.866000 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:51.865986    4964 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint"
Oct 31 01:03:51.866000 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:51.865991    4964 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet=""
Oct 31 01:03:51.866000 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:51.865997    4964 policy_none.go:50] "Start"
Oct 31 01:03:51.866174 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:51.866004    4964 memory_manager.go:187] "Starting memorymanager" policy="None"
Oct 31 01:03:51.866174 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:51.866012    4964 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Oct 31 01:03:51.869035 ip-10-0-133-197 kubenswrapper[4964]: E1031 01:03:51.869013    4964 kubelet.go:2545] "Skipping pod synchronization" err="container runtime status check may not have completed yet"
Oct 31 01:03:51.869118 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:51.869084    4964 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint"
Oct 31 01:03:51.869118 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:51.869095    4964 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint"
Oct 31 01:03:51.869118 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:51.869100    4964 policy_none.go:44] "Start"
Oct 31 01:03:51.873516 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:51.873494    4964 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Oct 31 01:03:51.876029 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:51.876017    4964 manager.go:341] "Starting Device Plugin manager"
Oct 31 01:03:51.876477 ip-10-0-133-197 kubenswrapper[4964]: E1031 01:03:51.876039    4964 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Oct 31 01:03:51.876522 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:51.876480    4964 server.go:87] "Starting device plugin registration server"
Oct 31 01:03:51.876696 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:51.876686    4964 eviction_manager.go:194] "Eviction manager: starting control loop"
Oct 31 01:03:51.876732 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:51.876697    4964 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Oct 31 01:03:51.876805 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:51.876789    4964 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Oct 31 01:03:51.876928 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:51.876891    4964 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Oct 31 01:03:51.876995 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:51.876938    4964 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Oct 31 01:03:51.876995 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:51.876945    4964 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Oct 31 01:03:51.877068 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:51.876995    4964 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-31T01:03:51.876927007Z","UUID":"1f446204-0ee7-46bd-8e68-a4a4a0198d63","Handler":null,"Name":"","Endpoint":""}
Oct 31 01:03:51.878273 ip-10-0-133-197 kubenswrapper[4964]: E1031 01:03:51.878246    4964 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Oct 31 01:03:51.878398 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:51.878293    4964 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Oct 31 01:03:51.878398 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:51.878331    4964 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Oct 31 01:03:51.887071 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:51.887018    4964 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 31 01:03:51.887766 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:51.887736    4964 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 31 01:03:51.887872 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:51.887846    4964 reconstruct.go:205] "DevicePaths of reconstructed volumes updated"
Oct 31 01:03:51.977369 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:51.977344    4964 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 31 01:03:51.978940 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:51.978921    4964 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-197.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Oct 31 01:03:51.979042 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:51.978953    4964 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-197.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Oct 31 01:03:51.979042 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:51.978965    4964 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-197.us-west-1.compute.internal" event="NodeHasSufficientPID"
Oct 31 01:03:51.979042 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:51.979003    4964 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-133-197.us-west-1.compute.internal"
Oct 31 01:03:52.069411 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:52.069372    4964 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-197.us-west-1.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-133-197.us-west-1.compute.internal"]
Oct 31 01:03:52.069989 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:52.069973    4964 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-197.us-west-1.compute.internal"
Oct 31 01:03:52.070090 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:52.069961    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-197.us-west-1.compute.internal" event={"ID":"107fc5ddbeb783edeed66dfa339956e5","Type":"ContainerStarted","Data":"eff8ed39a2c076a333e31fd286c4eab21b65ab2334b7add78f1945a3f83b013b"}
Oct 31 01:03:52.070158 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:52.070094    4964 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-197.us-west-1.compute.internal"
Oct 31 01:03:52.070158 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:52.070095    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-197.us-west-1.compute.internal" event={"ID":"107fc5ddbeb783edeed66dfa339956e5","Type":"ContainerDied","Data":"14ef005331ab078378b218b4578003c218440dedf09c4fc214b9e46021669c2e"}
Oct 31 01:03:52.070158 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:52.070124    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-197.us-west-1.compute.internal" event={"ID":"107fc5ddbeb783edeed66dfa339956e5","Type":"ContainerStarted","Data":"45b221fc6e0aeea83b1d8e8c88b6e9db2827f1a4aa5b5c74bc6b6ea9efc46143"}
Oct 31 01:03:52.070293 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:52.070171    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-197.us-west-1.compute.internal" event={"ID":"8dea7006d41de627c8e06bf18f518df6","Type":"ContainerStarted","Data":"e8f14a64a85206c9692c0ca2e99f02bf0f4093c4a2200a98780d31afc67d05d5"}
Oct 31 01:03:52.070293 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:52.070191    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-197.us-west-1.compute.internal" event={"ID":"8dea7006d41de627c8e06bf18f518df6","Type":"ContainerStarted","Data":"e5ff0a702058f6c4a964d51275a39426fdf7de02bd4756e66c20fde0c8b3a1d3"}
Oct 31 01:03:52.087602 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:52.087581    4964 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-133-197.us-west-1.compute.internal"
Oct 31 01:03:52.087704 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:52.087644    4964 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-133-197.us-west-1.compute.internal"
Oct 31 01:03:52.168716 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:52.168672    4964 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 31 01:03:52.168875 ip-10-0-133-197 kubenswrapper[4964]: E1031 01:03:52.168735    4964 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-133-197.us-west-1.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-197.us-west-1.compute.internal"
Oct 31 01:03:52.169450 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:52.169433    4964 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 31 01:03:52.169557 ip-10-0-133-197 kubenswrapper[4964]: E1031 01:03:52.169472    4964 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-133-197.us-west-1.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-197.us-west-1.compute.internal"
Oct 31 01:03:52.189047 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:52.189024    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/107fc5ddbeb783edeed66dfa339956e5-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-197.us-west-1.compute.internal\" (UID: \"107fc5ddbeb783edeed66dfa339956e5\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-197.us-west-1.compute.internal"
Oct 31 01:03:52.189123 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:52.189049    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/107fc5ddbeb783edeed66dfa339956e5-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-197.us-west-1.compute.internal\" (UID: \"107fc5ddbeb783edeed66dfa339956e5\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-197.us-west-1.compute.internal"
Oct 31 01:03:52.189123 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:52.189067    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/8dea7006d41de627c8e06bf18f518df6-config\") pod \"kube-apiserver-proxy-ip-10-0-133-197.us-west-1.compute.internal\" (UID: \"8dea7006d41de627c8e06bf18f518df6\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-197.us-west-1.compute.internal"
Oct 31 01:03:52.289378 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:52.289304    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/107fc5ddbeb783edeed66dfa339956e5-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-197.us-west-1.compute.internal\" (UID: \"107fc5ddbeb783edeed66dfa339956e5\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-197.us-west-1.compute.internal"
Oct 31 01:03:52.289378 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:52.289350    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/8dea7006d41de627c8e06bf18f518df6-config\") pod \"kube-apiserver-proxy-ip-10-0-133-197.us-west-1.compute.internal\" (UID: \"8dea7006d41de627c8e06bf18f518df6\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-197.us-west-1.compute.internal"
Oct 31 01:03:52.289378 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:52.289367    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/107fc5ddbeb783edeed66dfa339956e5-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-197.us-west-1.compute.internal\" (UID: \"107fc5ddbeb783edeed66dfa339956e5\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-197.us-west-1.compute.internal"
Oct 31 01:03:52.289527 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:52.289416    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/8dea7006d41de627c8e06bf18f518df6-config\") pod \"kube-apiserver-proxy-ip-10-0-133-197.us-west-1.compute.internal\" (UID: \"8dea7006d41de627c8e06bf18f518df6\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-197.us-west-1.compute.internal"
Oct 31 01:03:52.289527 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:52.289424    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/107fc5ddbeb783edeed66dfa339956e5-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-197.us-west-1.compute.internal\" (UID: \"107fc5ddbeb783edeed66dfa339956e5\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-197.us-west-1.compute.internal"
Oct 31 01:03:52.289527 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:52.289456    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/107fc5ddbeb783edeed66dfa339956e5-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-197.us-west-1.compute.internal\" (UID: \"107fc5ddbeb783edeed66dfa339956e5\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-197.us-west-1.compute.internal"
Oct 31 01:03:52.624975 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:52.624917    4964 apiserver.go:52] "Watching apiserver"
Oct 31 01:03:52.828643 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:52.828620    4964 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Oct 31 01:03:52.829517 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:52.829492    4964 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-cqpmt","openshift-multus/network-metrics-daemon-wdsmn","kube-system/kas-connection-checker-685895c7c9-jm6bm","openshift-console-operator/console-operator-7cd56b8dbc-9257n","openshift-ingress-canary/ingress-canary-7dpwv","kube-system/kube-apiserver-proxy-ip-10-0-133-197.us-west-1.compute.internal","openshift-ovn-kubernetes/ovnkube-node-pghw2","openshift-insights/insights-runtime-extractor-q56x5","openshift-service-ca-operator/service-ca-operator-544d749c8c-pdzj4","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-197.us-west-1.compute.internal","openshift-multus/multus-additional-cni-plugins-jwthj","openshift-kube-storage-version-migrator/migrator-66df5d496b-s2r9l","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-snxz4","openshift-image-registry/image-registry-75575b5db8-wtq4d","openshift-cluster-node-tuning-operator/tuned-45bvr","openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-n5q9v","openshift-network-operator/iptables-alerter-ssw8w","kube-system/kas-connection-checker-685895c7c9-vbbq6","openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-srcwl","openshift-network-diagnostics/network-check-target-8mqbd","openshift-ingress/router-default-5964fff8f9-8k6pp","openshift-insights/insights-operator-7c4986d449-jkk5x","openshift-network-console/networking-console-plugin-86c478bb88-wm4qt","openshift-dns/node-resolver-j46hv","kube-system/global-pull-secret-syncer-zs5g7","openshift-multus/multus-hxh6j","openshift-image-registry/image-registry-56d8464d7c-vpwjc","openshift-dns/dns-default-t7r6c","openshift-monitoring/cluster-monitoring-operator-884576c57-jccjk","kube-system/konnectivity-agent-78szg","openshift-network-diagnostics/network-check-source-6f78d59f99-bpmjc","openshift-insights/periodic-gathering-b2z6s-vxz9h","kube-system/kas-connection-checker-685895c7c9-q4z5f","openshift-service-ca/service-ca-5fffc97bdf-xhsdf","openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-v585z","openshift-image-registry/node-ca-rlxt2"]
Oct 31 01:03:52.830817 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:52.830794    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-srcwl"
Oct 31 01:03:52.835665 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:52.835644    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-5964fff8f9-8k6pp"
Oct 31 01:03:52.836795 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:52.836773    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-jccjk"
Oct 31 01:03:52.837295 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:52.837259    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-56d8464d7c-vpwjc"
Oct 31 01:03:52.837769 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:52.837751    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-wm4qt"
Oct 31 01:03:52.839259 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:52.839234    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wdsmn"
Oct 31 01:03:52.839698 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:52.839684    4964 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-197.us-west-1.compute.internal"
Oct 31 01:03:52.839882 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:52.839757    4964 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-197.us-west-1.compute.internal"
Oct 31 01:03:52.843213 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:52.843200    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-75575b5db8-wtq4d"
Oct 31 01:03:52.844248 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:52.844235    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-t7r6c"
Oct 31 01:03:52.846164 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:52.846143    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-q56x5"
Oct 31 01:03:52.886827 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:52.886772    4964 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\""
Oct 31 01:03:52.886827 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:52.886812    4964 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\""
Oct 31 01:03:52.886941 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:52.886900    4964 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\""
Oct 31 01:03:52.886978 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:52.886970    4964 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\""
Oct 31 01:03:52.887062 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:52.887049    4964 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"openshift-service-ca.crt\""
Oct 31 01:03:52.888103 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:52.888078    4964 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\""
Oct 31 01:03:52.888103 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:52.888090    4964 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\""
Oct 31 01:03:52.888226 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:52.888126    4964 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\""
Oct 31 01:03:52.888226 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:52.888131    4964 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"kube-root-ca.crt\""
Oct 31 01:03:52.888226 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:52.888189    4964 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\""
Oct 31 01:03:52.888626 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:52.888607    4964 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\""
Oct 31 01:03:52.888736 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:52.888688    4964 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\""
Oct 31 01:03:52.888832 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:52.888744    4964 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-config\""
Oct 31 01:03:52.889092 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:52.889073    4964 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\""
Oct 31 01:03:52.889176 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:52.889139    4964 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\""
Oct 31 01:03:52.889469 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:52.889448    4964 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\""
Oct 31 01:03:52.889574 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:52.889560    4964 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\""
Oct 31 01:03:52.890595 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:52.890576    4964 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\""
Oct 31 01:03:52.890687 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:52.890664    4964 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\""
Oct 31 01:03:52.928222 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:52.928201    4964 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Oct 31 01:03:52.933387 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:52.933364    4964 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\""
Oct 31 01:03:52.933468 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:52.933444    4964 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\""
Oct 31 01:03:52.935090 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:52.935069    4964 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\""
Oct 31 01:03:52.936411 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:52.936393    4964 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\""
Oct 31 01:03:52.936490 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:52.936417    4964 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Oct 31 01:03:52.936490 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:52.936438    4964 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\""
Oct 31 01:03:52.937123 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:52.937112    4964 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Oct 31 01:03:52.937547 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:52.937534    4964 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Oct 31 01:03:52.937661 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:52.937650    4964 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\""
Oct 31 01:03:52.937972 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:52.937960    4964 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Oct 31 01:03:52.938016 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:52.938003    4964 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\""
Oct 31 01:03:52.938085 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:52.938064    4964 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Oct 31 01:03:52.938618 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:52.938600    4964 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\""
Oct 31 01:03:52.938618 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:52.938615    4964 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Oct 31 01:03:52.938618 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:52.938609    4964 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Oct 31 01:03:52.938808 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:52.938640    4964 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-dockercfg-jsq9r\""
Oct 31 01:03:52.938808 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:52.938660    4964 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Oct 31 01:03:52.938808 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:52.938679    4964 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-h9z86\""
Oct 31 01:03:52.938808 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:52.938673    4964 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Oct 31 01:03:52.938808 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:52.938709    4964 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Oct 31 01:03:52.938808 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:52.938713    4964 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\""
Oct 31 01:03:52.938808 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:52.938709    4964 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Oct 31 01:03:52.938808 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:52.938754    4964 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-n7jgv\""
Oct 31 01:03:52.938808 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:52.938742    4964 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Oct 31 01:03:52.938808 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:52.938780    4964 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-tg5xh\""
Oct 31 01:03:52.938808 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:52.938725    4964 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Oct 31 01:03:52.938808 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:52.938762    4964 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Oct 31 01:03:52.941584 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:52.941571    4964 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Oct 31 01:03:52.941741 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:52.941728    4964 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Oct 31 01:03:52.941883 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:52.941873    4964 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Oct 31 01:03:52.982822 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:52.982803    4964 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Oct 31 01:03:52.982900 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:52.982832    4964 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Oct 31 01:03:52.984538 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:52.984525    4964 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Oct 31 01:03:52.985917 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:52.985903    4964 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Oct 31 01:03:52.987228 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:52.987211    4964 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Oct 31 01:03:52.987370 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:52.987214    4964 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Oct 31 01:03:52.987370 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:52.987279    4964 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-7g746\""
Oct 31 01:03:52.987526 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:52.987377    4964 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Oct 31 01:03:52.987526 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:52.987442    4964 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-42fft\""
Oct 31 01:03:52.987650 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:52.987539    4964 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-t8rwd\""
Oct 31 01:03:52.987650 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:52.987631    4964 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Oct 31 01:03:52.987779 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:52.987711    4964 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Oct 31 01:03:52.987779 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:52.987736    4964 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Oct 31 01:03:52.987779 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:52.987740    4964 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-qlxzn\""
Oct 31 01:03:52.987907 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:52.987778    4964 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Oct 31 01:03:52.987907 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:52.987782    4964 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Oct 31 01:03:52.987907 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:52.987807    4964 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-grw5x\""
Oct 31 01:03:52.987907 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:52.987835    4964 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-xgzbp\""
Oct 31 01:03:52.987907 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:52.987859    4964 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 31 01:03:52.987907 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:52.987873    4964 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 31 01:03:52.988285 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:52.987925    4964 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-55dd6\""
Oct 31 01:03:52.988285 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:52.987935    4964 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-kk499\""
Oct 31 01:03:52.988285 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:52.987978    4964 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\""
Oct 31 01:03:52.988285 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:52.988033    4964 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-cw2dc\""
Oct 31 01:03:52.988285 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:52.988064    4964 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Oct 31 01:03:52.988285 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:52.988076    4964 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-zzhf8\""
Oct 31 01:03:52.988285 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:52.988152    4964 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Oct 31 01:03:52.988285 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:52.988158    4964 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Oct 31 01:03:52.988285 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:52.988178    4964 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Oct 31 01:03:52.988285 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:52.988179    4964 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Oct 31 01:03:52.988285 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:52.988196    4964 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Oct 31 01:03:52.988285 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:52.988204    4964 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Oct 31 01:03:52.988285 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:52.988235    4964 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Oct 31 01:03:52.988285 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:52.988244    4964 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Oct 31 01:03:52.988285 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:52.988236    4964 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\""
Oct 31 01:03:52.988285 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:52.988255    4964 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Oct 31 01:03:52.988285 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:52.988278    4964 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Oct 31 01:03:52.988285 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:52.988289    4964 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Oct 31 01:03:52.988285 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:52.988292    4964 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Oct 31 01:03:52.988285 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:52.988278    4964 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Oct 31 01:03:52.989069 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:52.988300    4964 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-lc659\""
Oct 31 01:03:52.989069 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:52.988336    4964 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Oct 31 01:03:52.989069 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:52.988345    4964 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-5sjdn\""
Oct 31 01:03:52.989069 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:52.988347    4964 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Oct 31 01:03:52.989069 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:52.988358    4964 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Oct 31 01:03:52.989069 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:52.988366    4964 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-42gnc\""
Oct 31 01:03:52.989069 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:52.988376    4964 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Oct 31 01:03:52.989069 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:52.988366    4964 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Oct 31 01:03:52.989069 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:52.988368    4964 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-dvb7t\""
Oct 31 01:03:52.989069 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:52.988396    4964 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Oct 31 01:03:52.989069 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:52.988405    4964 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-rnpg9\""
Oct 31 01:03:52.989069 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:52.988399    4964 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Oct 31 01:03:52.989069 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:52.988419    4964 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Oct 31 01:03:52.989069 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:52.988424    4964 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Oct 31 01:03:52.989069 ip-10-0-133-197 kubenswrapper[4964]: E1031 01:03:52.988452    4964 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-133-197.us-west-1.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-197.us-west-1.compute.internal"
Oct 31 01:03:52.989069 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:52.988478    4964 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Oct 31 01:03:52.989069 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:52.988497    4964 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Oct 31 01:03:52.989069 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:52.988551    4964 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-fbmr7\""
Oct 31 01:03:52.989069 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:52.988651    4964 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Oct 31 01:03:52.989069 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:52.988664    4964 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Oct 31 01:03:52.989069 ip-10-0-133-197 kubenswrapper[4964]: E1031 01:03:52.988692    4964 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-133-197.us-west-1.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-197.us-west-1.compute.internal"
Oct 31 01:03:52.992899 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:52.992884    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c8e98df5-b2f7-49dd-a565-ed60534da496-tmp\") pod \"insights-operator-7c4986d449-jkk5x\" (UID: \"c8e98df5-b2f7-49dd-a565-ed60534da496\") " pod="openshift-insights/insights-operator-7c4986d449-jkk5x"
Oct 31 01:03:52.992939 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:52.992907    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-98n75\" (UniqueName: \"kubernetes.io/projected/e49453a8-c7d9-4248-a8ef-4a819b76806c-kube-api-access-98n75\") pod \"router-default-5964fff8f9-8k6pp\" (UID: \"e49453a8-c7d9-4248-a8ef-4a819b76806c\") " pod="openshift-ingress/router-default-5964fff8f9-8k6pp"
Oct 31 01:03:52.992939 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:52.992923    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/f8855c2c-87b4-4b9e-b28a-227981205b11-agent-certs\") pod \"konnectivity-agent-78szg\" (UID: \"f8855c2c-87b4-4b9e-b28a-227981205b11\") " pod="kube-system/konnectivity-agent-78szg"
Oct 31 01:03:52.993035 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:52.992941    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/58b0b75a-1568-4703-ac55-fa87f29225ad-hostroot\") pod \"multus-hxh6j\" (UID: \"58b0b75a-1568-4703-ac55-fa87f29225ad\") " pod="openshift-multus/multus-hxh6j"
Oct 31 01:03:52.993035 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:52.992956    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/58b0b75a-1568-4703-ac55-fa87f29225ad-system-cni-dir\") pod \"multus-hxh6j\" (UID: \"58b0b75a-1568-4703-ac55-fa87f29225ad\") " pod="openshift-multus/multus-hxh6j"
Oct 31 01:03:52.993035 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:52.992978    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/58b0b75a-1568-4703-ac55-fa87f29225ad-multus-socket-dir-parent\") pod \"multus-hxh6j\" (UID: \"58b0b75a-1568-4703-ac55-fa87f29225ad\") " pod="openshift-multus/multus-hxh6j"
Oct 31 01:03:52.993035 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:52.993001    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/e49453a8-c7d9-4248-a8ef-4a819b76806c-stats-auth\") pod \"router-default-5964fff8f9-8k6pp\" (UID: \"e49453a8-c7d9-4248-a8ef-4a819b76806c\") " pod="openshift-ingress/router-default-5964fff8f9-8k6pp"
Oct 31 01:03:52.993169 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:52.993037    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c8e98df5-b2f7-49dd-a565-ed60534da496-tmp\") pod \"insights-operator-7c4986d449-jkk5x\" (UID: \"c8e98df5-b2f7-49dd-a565-ed60534da496\") " pod="openshift-insights/insights-operator-7c4986d449-jkk5x"
Oct 31 01:03:52.993169 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:52.993046    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/aa112079-b749-404c-9081-113220707fcd-registry-tls\") pod \"image-registry-75575b5db8-wtq4d\" (UID: \"aa112079-b749-404c-9081-113220707fcd\") " pod="openshift-image-registry/image-registry-75575b5db8-wtq4d"
Oct 31 01:03:52.993267 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:52.993193    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8n8tt\" (UniqueName: \"kubernetes.io/projected/85bb69db-c69b-467e-adeb-950bf2947c94-kube-api-access-8n8tt\") pod \"dns-default-t7r6c\" (UID: \"85bb69db-c69b-467e-adeb-950bf2947c94\") " pod="openshift-dns/dns-default-t7r6c"
Oct 31 01:03:52.993267 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:52.993231    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/57c2f196-0d8a-4bce-a5bc-3b6b5c0d808c-hosts-file\") pod \"node-resolver-j46hv\" (UID: \"57c2f196-0d8a-4bce-a5bc-3b6b5c0d808c\") " pod="openshift-dns/node-resolver-j46hv"
Oct 31 01:03:52.993390 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:52.993266    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r4l7n\" (UniqueName: \"kubernetes.io/projected/9122c2fd-f29f-4e74-9615-15078517a3c0-kube-api-access-r4l7n\") pod \"periodic-gathering-b2z6s-vxz9h\" (UID: \"9122c2fd-f29f-4e74-9615-15078517a3c0\") " pod="openshift-insights/periodic-gathering-b2z6s-vxz9h"
Oct 31 01:03:52.993390 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:52.993302    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ftdwt\" (UniqueName: \"kubernetes.io/projected/d55eb70a-f1a8-4346-b8df-ccd2c598dd6b-kube-api-access-ftdwt\") pod \"ovnkube-node-pghw2\" (UID: \"d55eb70a-f1a8-4346-b8df-ccd2c598dd6b\") " pod="openshift-ovn-kubernetes/ovnkube-node-pghw2"
Oct 31 01:03:52.993390 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:52.993356    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/e49453a8-c7d9-4248-a8ef-4a819b76806c-stats-auth\") pod \"router-default-5964fff8f9-8k6pp\" (UID: \"e49453a8-c7d9-4248-a8ef-4a819b76806c\") " pod="openshift-ingress/router-default-5964fff8f9-8k6pp"
Oct 31 01:03:52.993390 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:52.993354    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/f273b9e5-0419-4587-a9c7-a0c8bec02acf-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-srcwl\" (UID: \"f273b9e5-0419-4587-a9c7-a0c8bec02acf\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-srcwl"
Oct 31 01:03:52.993636 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:52.993398    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/f8855c2c-87b4-4b9e-b28a-227981205b11-agent-certs\") pod \"konnectivity-agent-78szg\" (UID: \"f8855c2c-87b4-4b9e-b28a-227981205b11\") " pod="kube-system/konnectivity-agent-78szg"
Oct 31 01:03:52.993636 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:52.993412    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/58b0b75a-1568-4703-ac55-fa87f29225ad-host-run-k8s-cni-cncf-io\") pod \"multus-hxh6j\" (UID: \"58b0b75a-1568-4703-ac55-fa87f29225ad\") " pod="openshift-multus/multus-hxh6j"
Oct 31 01:03:52.993636 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:52.993434    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/b66619db-fd81-4435-be2a-a5ea6111e204-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-jccjk\" (UID: \"b66619db-fd81-4435-be2a-a5ea6111e204\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-jccjk"
Oct 31 01:03:52.993636 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:52.993463    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-59kdl\" (UniqueName: \"kubernetes.io/projected/b66619db-fd81-4435-be2a-a5ea6111e204-kube-api-access-59kdl\") pod \"cluster-monitoring-operator-884576c57-jccjk\" (UID: \"b66619db-fd81-4435-be2a-a5ea6111e204\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-jccjk"
Oct 31 01:03:52.993636 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:52.993519    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zlmlp\" (UniqueName: \"kubernetes.io/projected/f0c0ada7-7688-4dc2-97fc-813e4d0c4ef3-kube-api-access-zlmlp\") pod \"console-operator-7cd56b8dbc-9257n\" (UID: \"f0c0ada7-7688-4dc2-97fc-813e4d0c4ef3\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-9257n"
Oct 31 01:03:52.993636 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:52.993548    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/58b0b75a-1568-4703-ac55-fa87f29225ad-multus-conf-dir\") pod \"multus-hxh6j\" (UID: \"58b0b75a-1568-4703-ac55-fa87f29225ad\") " pod="openshift-multus/multus-hxh6j"
Oct 31 01:03:52.993636 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:52.993553    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/f273b9e5-0419-4587-a9c7-a0c8bec02acf-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-srcwl\" (UID: \"f273b9e5-0419-4587-a9c7-a0c8bec02acf\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-srcwl"
Oct 31 01:03:52.993636 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:52.993573    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/57c2f196-0d8a-4bce-a5bc-3b6b5c0d808c-tmp-dir\") pod \"node-resolver-j46hv\" (UID: \"57c2f196-0d8a-4bce-a5bc-3b6b5c0d808c\") " pod="openshift-dns/node-resolver-j46hv"
Oct 31 01:03:52.993636 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:52.993597    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rz545\" (UniqueName: \"kubernetes.io/projected/d719eda9-bc9f-4307-952a-99d99aa0f62b-kube-api-access-rz545\") pod \"kas-connection-checker-685895c7c9-jm6bm\" (UID: \"d719eda9-bc9f-4307-952a-99d99aa0f62b\") " pod="kube-system/kas-connection-checker-685895c7c9-jm6bm"
Oct 31 01:03:52.993636 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:52.993621    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/ff4dcba2-df8d-4324-9e0c-abb23a5e6982-image-registry-private-configuration\") pod \"image-registry-56d8464d7c-vpwjc\" (UID: \"ff4dcba2-df8d-4324-9e0c-abb23a5e6982\") " pod="openshift-image-registry/image-registry-56d8464d7c-vpwjc"
Oct 31 01:03:52.993636 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:52.993627    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/57c2f196-0d8a-4bce-a5bc-3b6b5c0d808c-tmp-dir\") pod \"node-resolver-j46hv\" (UID: \"57c2f196-0d8a-4bce-a5bc-3b6b5c0d808c\") " pod="openshift-dns/node-resolver-j46hv"
Oct 31 01:03:52.994146 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:52.993643    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/08490837-b3b6-4c2b-a498-568af0d7086d-etc-tuned\") pod \"tuned-45bvr\" (UID: \"08490837-b3b6-4c2b-a498-568af0d7086d\") " pod="openshift-cluster-node-tuning-operator/tuned-45bvr"
Oct 31 01:03:52.994146 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:52.993667    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/aa112079-b749-404c-9081-113220707fcd-trusted-ca\") pod \"image-registry-75575b5db8-wtq4d\" (UID: \"aa112079-b749-404c-9081-113220707fcd\") " pod="openshift-image-registry/image-registry-75575b5db8-wtq4d"
Oct 31 01:03:52.994146 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:52.993693    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c8e98df5-b2f7-49dd-a565-ed60534da496-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-jkk5x\" (UID: \"c8e98df5-b2f7-49dd-a565-ed60534da496\") " pod="openshift-insights/insights-operator-7c4986d449-jkk5x"
Oct 31 01:03:52.994146 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:52.993696    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/b66619db-fd81-4435-be2a-a5ea6111e204-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-jccjk\" (UID: \"b66619db-fd81-4435-be2a-a5ea6111e204\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-jccjk"
Oct 31 01:03:52.994146 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:52.993717    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/58b0b75a-1568-4703-ac55-fa87f29225ad-cnibin\") pod \"multus-hxh6j\" (UID: \"58b0b75a-1568-4703-ac55-fa87f29225ad\") " pod="openshift-multus/multus-hxh6j"
Oct 31 01:03:52.994146 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:52.993726    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/08490837-b3b6-4c2b-a498-568af0d7086d-etc-tuned\") pod \"tuned-45bvr\" (UID: \"08490837-b3b6-4c2b-a498-568af0d7086d\") " pod="openshift-cluster-node-tuning-operator/tuned-45bvr"
Oct 31 01:03:52.994146 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:52.993752    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/6fffc15e-94bd-4939-b38f-17d1fe6cc0b5-signing-key\") pod \"service-ca-5fffc97bdf-xhsdf\" (UID: \"6fffc15e-94bd-4939-b38f-17d1fe6cc0b5\") " pod="openshift-service-ca/service-ca-5fffc97bdf-xhsdf"
Oct 31 01:03:52.994146 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:52.993829    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/ff4dcba2-df8d-4324-9e0c-abb23a5e6982-image-registry-private-configuration\") pod \"image-registry-56d8464d7c-vpwjc\" (UID: \"ff4dcba2-df8d-4324-9e0c-abb23a5e6982\") " pod="openshift-image-registry/image-registry-56d8464d7c-vpwjc"
Oct 31 01:03:52.994146 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:52.993835    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/85bb69db-c69b-467e-adeb-950bf2947c94-config-volume\") pod \"dns-default-t7r6c\" (UID: \"85bb69db-c69b-467e-adeb-950bf2947c94\") " pod="openshift-dns/dns-default-t7r6c"
Oct 31 01:03:52.994146 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:52.993925    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jxmsl\" (UniqueName: \"kubernetes.io/projected/9cd1266e-c630-4259-b1f7-1b88dbde7385-kube-api-access-jxmsl\") pod \"iptables-alerter-ssw8w\" (UID: \"9cd1266e-c630-4259-b1f7-1b88dbde7385\") " pod="openshift-network-operator/iptables-alerter-ssw8w"
Oct 31 01:03:52.994146 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:52.994003    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/aa112079-b749-404c-9081-113220707fcd-trusted-ca\") pod \"image-registry-75575b5db8-wtq4d\" (UID: \"aa112079-b749-404c-9081-113220707fcd\") " pod="openshift-image-registry/image-registry-75575b5db8-wtq4d"
Oct 31 01:03:52.994146 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:52.994014    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/08490837-b3b6-4c2b-a498-568af0d7086d-etc-sysctl-d\") pod \"tuned-45bvr\" (UID: \"08490837-b3b6-4c2b-a498-568af0d7086d\") " pod="openshift-cluster-node-tuning-operator/tuned-45bvr"
Oct 31 01:03:52.994146 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:52.994029    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/6fffc15e-94bd-4939-b38f-17d1fe6cc0b5-signing-key\") pod \"service-ca-5fffc97bdf-xhsdf\" (UID: \"6fffc15e-94bd-4939-b38f-17d1fe6cc0b5\") " pod="openshift-service-ca/service-ca-5fffc97bdf-xhsdf"
Oct 31 01:03:52.994146 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:52.994042    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c8e98df5-b2f7-49dd-a565-ed60534da496-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-jkk5x\" (UID: \"c8e98df5-b2f7-49dd-a565-ed60534da496\") " pod="openshift-insights/insights-operator-7c4986d449-jkk5x"
Oct 31 01:03:52.994146 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:52.994046    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/85bb69db-c69b-467e-adeb-950bf2947c94-config-volume\") pod \"dns-default-t7r6c\" (UID: \"85bb69db-c69b-467e-adeb-950bf2947c94\") " pod="openshift-dns/dns-default-t7r6c"
Oct 31 01:03:52.994146 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:52.994120    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d55eb70a-f1a8-4346-b8df-ccd2c598dd6b-host-run-ovn-kubernetes\") pod \"ovnkube-node-pghw2\" (UID: \"d55eb70a-f1a8-4346-b8df-ccd2c598dd6b\") " pod="openshift-ovn-kubernetes/ovnkube-node-pghw2"
Oct 31 01:03:52.994810 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:52.994157    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/9dfae6d6-b9a6-4d85-b594-9d947a9e6383-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-wm4qt\" (UID: \"9dfae6d6-b9a6-4d85-b594-9d947a9e6383\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-wm4qt"
Oct 31 01:03:52.994810 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:52.994193    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e49453a8-c7d9-4248-a8ef-4a819b76806c-service-ca-bundle\") pod \"router-default-5964fff8f9-8k6pp\" (UID: \"e49453a8-c7d9-4248-a8ef-4a819b76806c\") " pod="openshift-ingress/router-default-5964fff8f9-8k6pp"
Oct 31 01:03:52.994810 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:52.994228    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/aa4836a5-c71a-485d-82fc-f8b53dae7717-original-pull-secret\") pod \"global-pull-secret-syncer-zs5g7\" (UID: \"aa4836a5-c71a-485d-82fc-f8b53dae7717\") " pod="kube-system/global-pull-secret-syncer-zs5g7"
Oct 31 01:03:52.994810 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:52.994251    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ff4dcba2-df8d-4324-9e0c-abb23a5e6982-registry-tls\") pod \"image-registry-56d8464d7c-vpwjc\" (UID: \"ff4dcba2-df8d-4324-9e0c-abb23a5e6982\") " pod="openshift-image-registry/image-registry-56d8464d7c-vpwjc"
Oct 31 01:03:52.994810 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:52.994369    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/ff4dcba2-df8d-4324-9e0c-abb23a5e6982-installation-pull-secrets\") pod \"image-registry-56d8464d7c-vpwjc\" (UID: \"ff4dcba2-df8d-4324-9e0c-abb23a5e6982\") " pod="openshift-image-registry/image-registry-56d8464d7c-vpwjc"
Oct 31 01:03:52.994810 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:52.994393    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9cd1266e-c630-4259-b1f7-1b88dbde7385-host-slash\") pod \"iptables-alerter-ssw8w\" (UID: \"9cd1266e-c630-4259-b1f7-1b88dbde7385\") " pod="openshift-network-operator/iptables-alerter-ssw8w"
Oct 31 01:03:52.994810 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:52.994408    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/08490837-b3b6-4c2b-a498-568af0d7086d-etc-sysctl-conf\") pod \"tuned-45bvr\" (UID: \"08490837-b3b6-4c2b-a498-568af0d7086d\") " pod="openshift-cluster-node-tuning-operator/tuned-45bvr"
Oct 31 01:03:52.994810 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:52.994423    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/f273b9e5-0419-4587-a9c7-a0c8bec02acf-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-srcwl\" (UID: \"f273b9e5-0419-4587-a9c7-a0c8bec02acf\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-srcwl"
Oct 31 01:03:52.994810 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:52.994446    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c8e98df5-b2f7-49dd-a565-ed60534da496-service-ca-bundle\") pod \"insights-operator-7c4986d449-jkk5x\" (UID: \"c8e98df5-b2f7-49dd-a565-ed60534da496\") " pod="openshift-insights/insights-operator-7c4986d449-jkk5x"
Oct 31 01:03:52.994810 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:52.994447    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/aa4836a5-c71a-485d-82fc-f8b53dae7717-original-pull-secret\") pod \"global-pull-secret-syncer-zs5g7\" (UID: \"aa4836a5-c71a-485d-82fc-f8b53dae7717\") " pod="kube-system/global-pull-secret-syncer-zs5g7"
Oct 31 01:03:52.994810 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:52.994470    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/f8855c2c-87b4-4b9e-b28a-227981205b11-konnectivity-ca\") pod \"konnectivity-agent-78szg\" (UID: \"f8855c2c-87b4-4b9e-b28a-227981205b11\") " pod="kube-system/konnectivity-agent-78szg"
Oct 31 01:03:52.994810 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:52.994560    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/fa085dd8-ca1e-4ec5-acb2-9b9dc878bf8a-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-jwthj\" (UID: \"fa085dd8-ca1e-4ec5-acb2-9b9dc878bf8a\") " pod="openshift-multus/multus-additional-cni-plugins-jwthj"
Oct 31 01:03:52.994810 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:52.994583    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/ff4dcba2-df8d-4324-9e0c-abb23a5e6982-installation-pull-secrets\") pod \"image-registry-56d8464d7c-vpwjc\" (UID: \"ff4dcba2-df8d-4324-9e0c-abb23a5e6982\") " pod="openshift-image-registry/image-registry-56d8464d7c-vpwjc"
Oct 31 01:03:52.994810 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:52.994596    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bspb5\" (UniqueName: \"kubernetes.io/projected/cb657366-6696-4c3e-a3a3-603ce00cf11a-kube-api-access-bspb5\") pod \"migrator-66df5d496b-cqpmt\" (UID: \"cb657366-6696-4c3e-a3a3-603ce00cf11a\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-cqpmt"
Oct 31 01:03:52.994810 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:52.994616    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c8e98df5-b2f7-49dd-a565-ed60534da496-service-ca-bundle\") pod \"insights-operator-7c4986d449-jkk5x\" (UID: \"c8e98df5-b2f7-49dd-a565-ed60534da496\") " pod="openshift-insights/insights-operator-7c4986d449-jkk5x"
Oct 31 01:03:52.994810 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:52.994626    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/08490837-b3b6-4c2b-a498-568af0d7086d-etc-sysconfig\") pod \"tuned-45bvr\" (UID: \"08490837-b3b6-4c2b-a498-568af0d7086d\") " pod="openshift-cluster-node-tuning-operator/tuned-45bvr"
Oct 31 01:03:52.994810 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:52.994651    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/58b0b75a-1568-4703-ac55-fa87f29225ad-host-run-netns\") pod \"multus-hxh6j\" (UID: \"58b0b75a-1568-4703-ac55-fa87f29225ad\") " pod="openshift-multus/multus-hxh6j"
Oct 31 01:03:52.994810 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:52.994682    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/6fffc15e-94bd-4939-b38f-17d1fe6cc0b5-signing-cabundle\") pod \"service-ca-5fffc97bdf-xhsdf\" (UID: \"6fffc15e-94bd-4939-b38f-17d1fe6cc0b5\") " pod="openshift-service-ca/service-ca-5fffc97bdf-xhsdf"
Oct 31 01:03:52.994810 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:52.994706    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f0c0ada7-7688-4dc2-97fc-813e4d0c4ef3-trusted-ca\") pod \"console-operator-7cd56b8dbc-9257n\" (UID: \"f0c0ada7-7688-4dc2-97fc-813e4d0c4ef3\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-9257n"
Oct 31 01:03:52.994810 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:52.994722    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/f8855c2c-87b4-4b9e-b28a-227981205b11-konnectivity-ca\") pod \"konnectivity-agent-78szg\" (UID: \"f8855c2c-87b4-4b9e-b28a-227981205b11\") " pod="kube-system/konnectivity-agent-78szg"
Oct 31 01:03:52.994810 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:52.994748    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/318b3bc6-6e59-4211-af63-b116526a325d-etc-selinux\") pod \"aws-ebs-csi-driver-node-snxz4\" (UID: \"318b3bc6-6e59-4211-af63-b116526a325d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-snxz4"
Oct 31 01:03:52.994810 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:52.994776    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6qc9x\" (UniqueName: \"kubernetes.io/projected/69ff2b62-4588-4cb0-bdad-747c3692138c-kube-api-access-6qc9x\") pod \"network-check-source-6f78d59f99-bpmjc\" (UID: \"69ff2b62-4588-4cb0-bdad-747c3692138c\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-bpmjc"
Oct 31 01:03:52.994810 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:52.994807    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/fa085dd8-ca1e-4ec5-acb2-9b9dc878bf8a-tuning-conf-dir\") pod \"multus-additional-cni-plugins-jwthj\" (UID: \"fa085dd8-ca1e-4ec5-acb2-9b9dc878bf8a\") " pod="openshift-multus/multus-additional-cni-plugins-jwthj"
Oct 31 01:03:52.995895 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:52.994838    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/08490837-b3b6-4c2b-a498-568af0d7086d-etc-kubernetes\") pod \"tuned-45bvr\" (UID: \"08490837-b3b6-4c2b-a498-568af0d7086d\") " pod="openshift-cluster-node-tuning-operator/tuned-45bvr"
Oct 31 01:03:52.995895 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:52.994844    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/fa085dd8-ca1e-4ec5-acb2-9b9dc878bf8a-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-jwthj\" (UID: \"fa085dd8-ca1e-4ec5-acb2-9b9dc878bf8a\") " pod="openshift-multus/multus-additional-cni-plugins-jwthj"
Oct 31 01:03:52.995895 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:52.994875    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/e49453a8-c7d9-4248-a8ef-4a819b76806c-default-certificate\") pod \"router-default-5964fff8f9-8k6pp\" (UID: \"e49453a8-c7d9-4248-a8ef-4a819b76806c\") " pod="openshift-ingress/router-default-5964fff8f9-8k6pp"
Oct 31 01:03:52.995895 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:52.994902    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/6fffc15e-94bd-4939-b38f-17d1fe6cc0b5-signing-cabundle\") pod \"service-ca-5fffc97bdf-xhsdf\" (UID: \"6fffc15e-94bd-4939-b38f-17d1fe6cc0b5\") " pod="openshift-service-ca/service-ca-5fffc97bdf-xhsdf"
Oct 31 01:03:52.995895 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:52.994907    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/08490837-b3b6-4c2b-a498-568af0d7086d-sys\") pod \"tuned-45bvr\" (UID: \"08490837-b3b6-4c2b-a498-568af0d7086d\") " pod="openshift-cluster-node-tuning-operator/tuned-45bvr"
Oct 31 01:03:52.995895 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:52.994972    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w9tzc\" (UniqueName: \"kubernetes.io/projected/08490837-b3b6-4c2b-a498-568af0d7086d-kube-api-access-w9tzc\") pod \"tuned-45bvr\" (UID: \"08490837-b3b6-4c2b-a498-568af0d7086d\") " pod="openshift-cluster-node-tuning-operator/tuned-45bvr"
Oct 31 01:03:52.995895 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:52.994993    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f0c0ada7-7688-4dc2-97fc-813e4d0c4ef3-trusted-ca\") pod \"console-operator-7cd56b8dbc-9257n\" (UID: \"f0c0ada7-7688-4dc2-97fc-813e4d0c4ef3\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-9257n"
Oct 31 01:03:52.995895 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:52.994999    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/d55eb70a-f1a8-4346-b8df-ccd2c598dd6b-host-kubelet\") pod \"ovnkube-node-pghw2\" (UID: \"d55eb70a-f1a8-4346-b8df-ccd2c598dd6b\") " pod="openshift-ovn-kubernetes/ovnkube-node-pghw2"
Oct 31 01:03:52.995895 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:52.995059    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/e49453a8-c7d9-4248-a8ef-4a819b76806c-default-certificate\") pod \"router-default-5964fff8f9-8k6pp\" (UID: \"e49453a8-c7d9-4248-a8ef-4a819b76806c\") " pod="openshift-ingress/router-default-5964fff8f9-8k6pp"
Oct 31 01:03:52.995895 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:52.995088    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d55eb70a-f1a8-4346-b8df-ccd2c598dd6b-var-lib-openvswitch\") pod \"ovnkube-node-pghw2\" (UID: \"d55eb70a-f1a8-4346-b8df-ccd2c598dd6b\") " pod="openshift-ovn-kubernetes/ovnkube-node-pghw2"
Oct 31 01:03:52.995895 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:52.995107    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/aa112079-b749-404c-9081-113220707fcd-installation-pull-secrets\") pod \"image-registry-75575b5db8-wtq4d\" (UID: \"aa112079-b749-404c-9081-113220707fcd\") " pod="openshift-image-registry/image-registry-75575b5db8-wtq4d"
Oct 31 01:03:52.995895 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:52.995123    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r6zx9\" (UniqueName: \"kubernetes.io/projected/565fefaf-82aa-42a8-9cc7-6517c4bddf1e-kube-api-access-r6zx9\") pod \"network-check-target-8mqbd\" (UID: \"565fefaf-82aa-42a8-9cc7-6517c4bddf1e\") " pod="openshift-network-diagnostics/network-check-target-8mqbd"
Oct 31 01:03:52.995895 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:52.995157    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/58b0b75a-1568-4703-ac55-fa87f29225ad-cni-binary-copy\") pod \"multus-hxh6j\" (UID: \"58b0b75a-1568-4703-ac55-fa87f29225ad\") " pod="openshift-multus/multus-hxh6j"
Oct 31 01:03:52.995895 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:52.995232    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/c2be658b-ec92-4782-a78b-d2dccf2249cb-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-n5q9v\" (UID: \"c2be658b-ec92-4782-a78b-d2dccf2249cb\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-n5q9v"
Oct 31 01:03:52.995895 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:52.995251    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j78lj\" (UniqueName: \"kubernetes.io/projected/eafc69c5-15f1-4e4c-97ca-be586e55fbae-kube-api-access-j78lj\") pod \"volume-data-source-validator-58c9f5f46d-v585z\" (UID: \"eafc69c5-15f1-4e4c-97ca-be586e55fbae\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-v585z"
Oct 31 01:03:52.995895 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:52.995270    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/d55eb70a-f1a8-4346-b8df-ccd2c598dd6b-ovnkube-config\") pod \"ovnkube-node-pghw2\" (UID: \"d55eb70a-f1a8-4346-b8df-ccd2c598dd6b\") " pod="openshift-ovn-kubernetes/ovnkube-node-pghw2"
Oct 31 01:03:52.995895 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:52.995306    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2dx97\" (UniqueName: \"kubernetes.io/projected/57c2f196-0d8a-4bce-a5bc-3b6b5c0d808c-kube-api-access-2dx97\") pod \"node-resolver-j46hv\" (UID: \"57c2f196-0d8a-4bce-a5bc-3b6b5c0d808c\") " pod="openshift-dns/node-resolver-j46hv"
Oct 31 01:03:52.995895 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:52.995338    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/aa112079-b749-404c-9081-113220707fcd-installation-pull-secrets\") pod \"image-registry-75575b5db8-wtq4d\" (UID: \"aa112079-b749-404c-9081-113220707fcd\") " pod="openshift-image-registry/image-registry-75575b5db8-wtq4d"
Oct 31 01:03:52.995895 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:52.995349    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/58b0b75a-1568-4703-ac55-fa87f29225ad-cni-binary-copy\") pod \"multus-hxh6j\" (UID: \"58b0b75a-1568-4703-ac55-fa87f29225ad\") " pod="openshift-multus/multus-hxh6j"
Oct 31 01:03:52.995895 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:52.995356    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/7dd98c21-e0b1-4289-8fe8-036bf413d444-crio-socket\") pod \"insights-runtime-extractor-q56x5\" (UID: \"7dd98c21-e0b1-4289-8fe8-036bf413d444\") " pod="openshift-insights/insights-runtime-extractor-q56x5"
Oct 31 01:03:52.995895 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:52.995390    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d55eb70a-f1a8-4346-b8df-ccd2c598dd6b-run-openvswitch\") pod \"ovnkube-node-pghw2\" (UID: \"d55eb70a-f1a8-4346-b8df-ccd2c598dd6b\") " pod="openshift-ovn-kubernetes/ovnkube-node-pghw2"
Oct 31 01:03:52.995895 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:52.995437    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/aa112079-b749-404c-9081-113220707fcd-ca-trust-extracted\") pod \"image-registry-75575b5db8-wtq4d\" (UID: \"aa112079-b749-404c-9081-113220707fcd\") " pod="openshift-image-registry/image-registry-75575b5db8-wtq4d"
Oct 31 01:03:52.995895 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:52.995454    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/c2be658b-ec92-4782-a78b-d2dccf2249cb-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-n5q9v\" (UID: \"c2be658b-ec92-4782-a78b-d2dccf2249cb\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-n5q9v"
Oct 31 01:03:52.995895 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:52.995463    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tlnlx\" (UniqueName: \"kubernetes.io/projected/c8e98df5-b2f7-49dd-a565-ed60534da496-kube-api-access-tlnlx\") pod \"insights-operator-7c4986d449-jkk5x\" (UID: \"c8e98df5-b2f7-49dd-a565-ed60534da496\") " pod="openshift-insights/insights-operator-7c4986d449-jkk5x"
Oct 31 01:03:52.995895 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:52.995486    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/85bb69db-c69b-467e-adeb-950bf2947c94-tmp-dir\") pod \"dns-default-t7r6c\" (UID: \"85bb69db-c69b-467e-adeb-950bf2947c94\") " pod="openshift-dns/dns-default-t7r6c"
Oct 31 01:03:52.995895 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:52.995501    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ff4dcba2-df8d-4324-9e0c-abb23a5e6982-trusted-ca\") pod \"image-registry-56d8464d7c-vpwjc\" (UID: \"ff4dcba2-df8d-4324-9e0c-abb23a5e6982\") " pod="openshift-image-registry/image-registry-56d8464d7c-vpwjc"
Oct 31 01:03:52.995895 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:52.995510    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/aa112079-b749-404c-9081-113220707fcd-ca-trust-extracted\") pod \"image-registry-75575b5db8-wtq4d\" (UID: \"aa112079-b749-404c-9081-113220707fcd\") " pod="openshift-image-registry/image-registry-75575b5db8-wtq4d"
Oct 31 01:03:52.995895 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:52.995517    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/fa085dd8-ca1e-4ec5-acb2-9b9dc878bf8a-cnibin\") pod \"multus-additional-cni-plugins-jwthj\" (UID: \"fa085dd8-ca1e-4ec5-acb2-9b9dc878bf8a\") " pod="openshift-multus/multus-additional-cni-plugins-jwthj"
Oct 31 01:03:52.995895 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:52.995554    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/08490837-b3b6-4c2b-a498-568af0d7086d-etc-systemd\") pod \"tuned-45bvr\" (UID: \"08490837-b3b6-4c2b-a498-568af0d7086d\") " pod="openshift-cluster-node-tuning-operator/tuned-45bvr"
Oct 31 01:03:52.995895 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:52.995579    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/58b0b75a-1568-4703-ac55-fa87f29225ad-host-run-multus-certs\") pod \"multus-hxh6j\" (UID: \"58b0b75a-1568-4703-ac55-fa87f29225ad\") " pod="openshift-multus/multus-hxh6j"
Oct 31 01:03:52.995895 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:52.995584    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/d55eb70a-f1a8-4346-b8df-ccd2c598dd6b-ovnkube-config\") pod \"ovnkube-node-pghw2\" (UID: \"d55eb70a-f1a8-4346-b8df-ccd2c598dd6b\") " pod="openshift-ovn-kubernetes/ovnkube-node-pghw2"
Oct 31 01:03:52.995895 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:52.995607    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9122c2fd-f29f-4e74-9615-15078517a3c0-trusted-ca-bundle\") pod \"periodic-gathering-b2z6s-vxz9h\" (UID: \"9122c2fd-f29f-4e74-9615-15078517a3c0\") " pod="openshift-insights/periodic-gathering-b2z6s-vxz9h"
Oct 31 01:03:52.995895 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:52.995608    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/85bb69db-c69b-467e-adeb-950bf2947c94-tmp-dir\") pod \"dns-default-t7r6c\" (UID: \"85bb69db-c69b-467e-adeb-950bf2947c94\") " pod="openshift-dns/dns-default-t7r6c"
Oct 31 01:03:52.995895 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:52.995639    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/08490837-b3b6-4c2b-a498-568af0d7086d-etc-modprobe-d\") pod \"tuned-45bvr\" (UID: \"08490837-b3b6-4c2b-a498-568af0d7086d\") " pod="openshift-cluster-node-tuning-operator/tuned-45bvr"
Oct 31 01:03:52.995895 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:52.995732    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/aa112079-b749-404c-9081-113220707fcd-image-registry-private-configuration\") pod \"image-registry-75575b5db8-wtq4d\" (UID: \"aa112079-b749-404c-9081-113220707fcd\") " pod="openshift-image-registry/image-registry-75575b5db8-wtq4d"
Oct 31 01:03:52.995895 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:52.995749    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/e54a9dc6-36e6-4f1b-9c3a-623a520e11da-serving-cert\") pod \"service-ca-operator-544d749c8c-pdzj4\" (UID: \"e54a9dc6-36e6-4f1b-9c3a-623a520e11da\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-pdzj4"
Oct 31 01:03:52.995895 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:52.995805    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bjgwm\" (UniqueName: \"kubernetes.io/projected/c2be658b-ec92-4782-a78b-d2dccf2249cb-kube-api-access-bjgwm\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-n5q9v\" (UID: \"c2be658b-ec92-4782-a78b-d2dccf2249cb\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-n5q9v"
Oct 31 01:03:52.995895 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:52.995832    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/9122c2fd-f29f-4e74-9615-15078517a3c0-archives-path\") pod \"periodic-gathering-b2z6s-vxz9h\" (UID: \"9122c2fd-f29f-4e74-9615-15078517a3c0\") " pod="openshift-insights/periodic-gathering-b2z6s-vxz9h"
Oct 31 01:03:52.995895 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:52.995904    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/9122c2fd-f29f-4e74-9615-15078517a3c0-archives-path\") pod \"periodic-gathering-b2z6s-vxz9h\" (UID: \"9122c2fd-f29f-4e74-9615-15078517a3c0\") " pod="openshift-insights/periodic-gathering-b2z6s-vxz9h"
Oct 31 01:03:52.995895 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:52.995923    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/aa112079-b749-404c-9081-113220707fcd-image-registry-private-configuration\") pod \"image-registry-75575b5db8-wtq4d\" (UID: \"aa112079-b749-404c-9081-113220707fcd\") " pod="openshift-image-registry/image-registry-75575b5db8-wtq4d"
Oct 31 01:03:52.997391 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:52.995937    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/dd8134f9-b2a6-4542-82ec-c10edc3fcd72-host\") pod \"node-ca-rlxt2\" (UID: \"dd8134f9-b2a6-4542-82ec-c10edc3fcd72\") " pod="openshift-image-registry/node-ca-rlxt2"
Oct 31 01:03:52.997391 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:52.995968    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/d55eb70a-f1a8-4346-b8df-ccd2c598dd6b-host-cni-netd\") pod \"ovnkube-node-pghw2\" (UID: \"d55eb70a-f1a8-4346-b8df-ccd2c598dd6b\") " pod="openshift-ovn-kubernetes/ovnkube-node-pghw2"
Oct 31 01:03:52.997391 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:52.995975    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/e54a9dc6-36e6-4f1b-9c3a-623a520e11da-serving-cert\") pod \"service-ca-operator-544d749c8c-pdzj4\" (UID: \"e54a9dc6-36e6-4f1b-9c3a-623a520e11da\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-pdzj4"
Oct 31 01:03:52.997391 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:52.995992    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/58b0b75a-1568-4703-ac55-fa87f29225ad-host-var-lib-cni-multus\") pod \"multus-hxh6j\" (UID: \"58b0b75a-1568-4703-ac55-fa87f29225ad\") " pod="openshift-multus/multus-hxh6j"
Oct 31 01:03:52.997391 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:52.996025    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-srkbh\" (UniqueName: \"kubernetes.io/projected/8952fc79-c5a6-42ba-a296-8cb7a776f718-kube-api-access-srkbh\") pod \"network-metrics-daemon-wdsmn\" (UID: \"8952fc79-c5a6-42ba-a296-8cb7a776f718\") " pod="openshift-multus/network-metrics-daemon-wdsmn"
Oct 31 01:03:52.997391 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:52.996043    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/ff4dcba2-df8d-4324-9e0c-abb23a5e6982-ca-trust-extracted\") pod \"image-registry-56d8464d7c-vpwjc\" (UID: \"ff4dcba2-df8d-4324-9e0c-abb23a5e6982\") " pod="openshift-image-registry/image-registry-56d8464d7c-vpwjc"
Oct 31 01:03:52.997391 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:52.996080    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/fa085dd8-ca1e-4ec5-acb2-9b9dc878bf8a-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-jwthj\" (UID: \"fa085dd8-ca1e-4ec5-acb2-9b9dc878bf8a\") " pod="openshift-multus/multus-additional-cni-plugins-jwthj"
Oct 31 01:03:52.997391 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:52.996094    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/ff4dcba2-df8d-4324-9e0c-abb23a5e6982-ca-trust-extracted\") pod \"image-registry-56d8464d7c-vpwjc\" (UID: \"ff4dcba2-df8d-4324-9e0c-abb23a5e6982\") " pod="openshift-image-registry/image-registry-56d8464d7c-vpwjc"
Oct 31 01:03:52.997391 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:52.996192    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d55eb70a-f1a8-4346-b8df-ccd2c598dd6b-host-slash\") pod \"ovnkube-node-pghw2\" (UID: \"d55eb70a-f1a8-4346-b8df-ccd2c598dd6b\") " pod="openshift-ovn-kubernetes/ovnkube-node-pghw2"
Oct 31 01:03:52.997391 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:52.996228    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ff4dcba2-df8d-4324-9e0c-abb23a5e6982-trusted-ca\") pod \"image-registry-56d8464d7c-vpwjc\" (UID: \"ff4dcba2-df8d-4324-9e0c-abb23a5e6982\") " pod="openshift-image-registry/image-registry-56d8464d7c-vpwjc"
Oct 31 01:03:52.997391 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:52.996230    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/58b0b75a-1568-4703-ac55-fa87f29225ad-etc-kubernetes\") pod \"multus-hxh6j\" (UID: \"58b0b75a-1568-4703-ac55-fa87f29225ad\") " pod="openshift-multus/multus-hxh6j"
Oct 31 01:03:52.997391 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:52.996260    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rpmxj\" (UniqueName: \"kubernetes.io/projected/aa112079-b749-404c-9081-113220707fcd-kube-api-access-rpmxj\") pod \"image-registry-75575b5db8-wtq4d\" (UID: \"aa112079-b749-404c-9081-113220707fcd\") " pod="openshift-image-registry/image-registry-75575b5db8-wtq4d"
Oct 31 01:03:52.997391 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:52.996280    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/318b3bc6-6e59-4211-af63-b116526a325d-socket-dir\") pod \"aws-ebs-csi-driver-node-snxz4\" (UID: \"318b3bc6-6e59-4211-af63-b116526a325d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-snxz4"
Oct 31 01:03:52.997391 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:52.996296    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/aa4836a5-c71a-485d-82fc-f8b53dae7717-dbus\") pod \"global-pull-secret-syncer-zs5g7\" (UID: \"aa4836a5-c71a-485d-82fc-f8b53dae7717\") " pod="kube-system/global-pull-secret-syncer-zs5g7"
Oct 31 01:03:52.997391 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:52.996299    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/fa085dd8-ca1e-4ec5-acb2-9b9dc878bf8a-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-jwthj\" (UID: \"fa085dd8-ca1e-4ec5-acb2-9b9dc878bf8a\") " pod="openshift-multus/multus-additional-cni-plugins-jwthj"
Oct 31 01:03:52.997391 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:52.996310    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ff4dcba2-df8d-4324-9e0c-abb23a5e6982-bound-sa-token\") pod \"image-registry-56d8464d7c-vpwjc\" (UID: \"ff4dcba2-df8d-4324-9e0c-abb23a5e6982\") " pod="openshift-image-registry/image-registry-56d8464d7c-vpwjc"
Oct 31 01:03:52.997391 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:52.996338    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9122c2fd-f29f-4e74-9615-15078517a3c0-trusted-ca-bundle\") pod \"periodic-gathering-b2z6s-vxz9h\" (UID: \"9122c2fd-f29f-4e74-9615-15078517a3c0\") " pod="openshift-insights/periodic-gathering-b2z6s-vxz9h"
Oct 31 01:03:52.997391 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:52.996359    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ks6fn\" (UniqueName: \"kubernetes.io/projected/e54a9dc6-36e6-4f1b-9c3a-623a520e11da-kube-api-access-ks6fn\") pod \"service-ca-operator-544d749c8c-pdzj4\" (UID: \"e54a9dc6-36e6-4f1b-9c3a-623a520e11da\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-pdzj4"
Oct 31 01:03:52.997391 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:52.996472    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jkn67\" (UniqueName: \"kubernetes.io/projected/da8395ef-7464-410e-96ee-b27378a03fed-kube-api-access-jkn67\") pod \"kas-connection-checker-685895c7c9-vbbq6\" (UID: \"da8395ef-7464-410e-96ee-b27378a03fed\") " pod="kube-system/kas-connection-checker-685895c7c9-vbbq6"
Oct 31 01:03:52.997391 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:52.996500    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/dd8134f9-b2a6-4542-82ec-c10edc3fcd72-serviceca\") pod \"node-ca-rlxt2\" (UID: \"dd8134f9-b2a6-4542-82ec-c10edc3fcd72\") " pod="openshift-image-registry/node-ca-rlxt2"
Oct 31 01:03:52.997391 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:52.996524    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/aa112079-b749-404c-9081-113220707fcd-registry-certificates\") pod \"image-registry-75575b5db8-wtq4d\" (UID: \"aa112079-b749-404c-9081-113220707fcd\") " pod="openshift-image-registry/image-registry-75575b5db8-wtq4d"
Oct 31 01:03:52.997391 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:52.996553    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/c8e98df5-b2f7-49dd-a565-ed60534da496-snapshots\") pod \"insights-operator-7c4986d449-jkk5x\" (UID: \"c8e98df5-b2f7-49dd-a565-ed60534da496\") " pod="openshift-insights/insights-operator-7c4986d449-jkk5x"
Oct 31 01:03:52.997391 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:52.996584    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9122c2fd-f29f-4e74-9615-15078517a3c0-service-ca-bundle\") pod \"periodic-gathering-b2z6s-vxz9h\" (UID: \"9122c2fd-f29f-4e74-9615-15078517a3c0\") " pod="openshift-insights/periodic-gathering-b2z6s-vxz9h"
Oct 31 01:03:52.997391 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:52.996601    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/d55eb70a-f1a8-4346-b8df-ccd2c598dd6b-ovnkube-script-lib\") pod \"ovnkube-node-pghw2\" (UID: \"d55eb70a-f1a8-4346-b8df-ccd2c598dd6b\") " pod="openshift-ovn-kubernetes/ovnkube-node-pghw2"
Oct 31 01:03:52.997391 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:52.996618    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/318b3bc6-6e59-4211-af63-b116526a325d-kubelet-dir\") pod \"aws-ebs-csi-driver-node-snxz4\" (UID: \"318b3bc6-6e59-4211-af63-b116526a325d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-snxz4"
Oct 31 01:03:52.997391 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:52.996637    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/58b0b75a-1568-4703-ac55-fa87f29225ad-host-var-lib-kubelet\") pod \"multus-hxh6j\" (UID: \"58b0b75a-1568-4703-ac55-fa87f29225ad\") " pod="openshift-multus/multus-hxh6j"
Oct 31 01:03:52.997391 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:52.996687    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/58b0b75a-1568-4703-ac55-fa87f29225ad-multus-daemon-config\") pod \"multus-hxh6j\" (UID: \"58b0b75a-1568-4703-ac55-fa87f29225ad\") " pod="openshift-multus/multus-hxh6j"
Oct 31 01:03:52.997391 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:52.996691    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/aa112079-b749-404c-9081-113220707fcd-registry-certificates\") pod \"image-registry-75575b5db8-wtq4d\" (UID: \"aa112079-b749-404c-9081-113220707fcd\") " pod="openshift-image-registry/image-registry-75575b5db8-wtq4d"
Oct 31 01:03:52.997391 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:52.996718    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/dd8134f9-b2a6-4542-82ec-c10edc3fcd72-serviceca\") pod \"node-ca-rlxt2\" (UID: \"dd8134f9-b2a6-4542-82ec-c10edc3fcd72\") " pod="openshift-image-registry/node-ca-rlxt2"
Oct 31 01:03:52.997391 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:52.996726    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/c8e98df5-b2f7-49dd-a565-ed60534da496-snapshots\") pod \"insights-operator-7c4986d449-jkk5x\" (UID: \"c8e98df5-b2f7-49dd-a565-ed60534da496\") " pod="openshift-insights/insights-operator-7c4986d449-jkk5x"
Oct 31 01:03:52.997391 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:52.996731    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g9wls\" (UniqueName: \"kubernetes.io/projected/147bb9af-dfc7-4785-bb5e-906f25d65dd1-kube-api-access-g9wls\") pod \"ingress-canary-7dpwv\" (UID: \"147bb9af-dfc7-4785-bb5e-906f25d65dd1\") " pod="openshift-ingress-canary/ingress-canary-7dpwv"
Oct 31 01:03:52.997391 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:52.996777    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9122c2fd-f29f-4e74-9615-15078517a3c0-service-ca-bundle\") pod \"periodic-gathering-b2z6s-vxz9h\" (UID: \"9122c2fd-f29f-4e74-9615-15078517a3c0\") " pod="openshift-insights/periodic-gathering-b2z6s-vxz9h"
Oct 31 01:03:52.997391 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:52.996814    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vms79\" (UniqueName: \"kubernetes.io/projected/0ab3954e-be64-4260-bca8-ff5d58bd6ea2-kube-api-access-vms79\") pod \"kas-connection-checker-685895c7c9-q4z5f\" (UID: \"0ab3954e-be64-4260-bca8-ff5d58bd6ea2\") " pod="kube-system/kas-connection-checker-685895c7c9-q4z5f"
Oct 31 01:03:52.997391 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:52.996833    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/7dd98c21-e0b1-4289-8fe8-036bf413d444-data-volume\") pod \"insights-runtime-extractor-q56x5\" (UID: \"7dd98c21-e0b1-4289-8fe8-036bf413d444\") " pod="openshift-insights/insights-runtime-extractor-q56x5"
Oct 31 01:03:52.997391 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:52.996848    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wsrb6\" (UniqueName: \"kubernetes.io/projected/dd8134f9-b2a6-4542-82ec-c10edc3fcd72-kube-api-access-wsrb6\") pod \"node-ca-rlxt2\" (UID: \"dd8134f9-b2a6-4542-82ec-c10edc3fcd72\") " pod="openshift-image-registry/node-ca-rlxt2"
Oct 31 01:03:52.997391 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:52.996853    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/d55eb70a-f1a8-4346-b8df-ccd2c598dd6b-ovnkube-script-lib\") pod \"ovnkube-node-pghw2\" (UID: \"d55eb70a-f1a8-4346-b8df-ccd2c598dd6b\") " pod="openshift-ovn-kubernetes/ovnkube-node-pghw2"
Oct 31 01:03:52.997391 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:52.996862    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/c8e98df5-b2f7-49dd-a565-ed60534da496-serving-cert\") pod \"insights-operator-7c4986d449-jkk5x\" (UID: \"c8e98df5-b2f7-49dd-a565-ed60534da496\") " pod="openshift-insights/insights-operator-7c4986d449-jkk5x"
Oct 31 01:03:52.997391 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:52.996849    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/58b0b75a-1568-4703-ac55-fa87f29225ad-multus-daemon-config\") pod \"multus-hxh6j\" (UID: \"58b0b75a-1568-4703-ac55-fa87f29225ad\") " pod="openshift-multus/multus-hxh6j"
Oct 31 01:03:52.997391 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:52.996890    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/f0c0ada7-7688-4dc2-97fc-813e4d0c4ef3-config\") pod \"console-operator-7cd56b8dbc-9257n\" (UID: \"f0c0ada7-7688-4dc2-97fc-813e4d0c4ef3\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-9257n"
Oct 31 01:03:52.997391 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:52.996908    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/7dd98c21-e0b1-4289-8fe8-036bf413d444-data-volume\") pod \"insights-runtime-extractor-q56x5\" (UID: \"7dd98c21-e0b1-4289-8fe8-036bf413d444\") " pod="openshift-insights/insights-runtime-extractor-q56x5"
Oct 31 01:03:52.997391 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:52.996914    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t228s\" (UniqueName: \"kubernetes.io/projected/58b0b75a-1568-4703-ac55-fa87f29225ad-kube-api-access-t228s\") pod \"multus-hxh6j\" (UID: \"58b0b75a-1568-4703-ac55-fa87f29225ad\") " pod="openshift-multus/multus-hxh6j"
Oct 31 01:03:52.997391 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:52.996941    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/85bb69db-c69b-467e-adeb-950bf2947c94-metrics-tls\") pod \"dns-default-t7r6c\" (UID: \"85bb69db-c69b-467e-adeb-950bf2947c94\") " pod="openshift-dns/dns-default-t7r6c"
Oct 31 01:03:52.997391 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:52.996965    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/9cd1266e-c630-4259-b1f7-1b88dbde7385-iptables-alerter-script\") pod \"iptables-alerter-ssw8w\" (UID: \"9cd1266e-c630-4259-b1f7-1b88dbde7385\") " pod="openshift-network-operator/iptables-alerter-ssw8w"
Oct 31 01:03:52.997391 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:52.996990    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/58b0b75a-1568-4703-ac55-fa87f29225ad-multus-cni-dir\") pod \"multus-hxh6j\" (UID: \"58b0b75a-1568-4703-ac55-fa87f29225ad\") " pod="openshift-multus/multus-hxh6j"
Oct 31 01:03:52.997391 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:52.997000    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/c8e98df5-b2f7-49dd-a565-ed60534da496-serving-cert\") pod \"insights-operator-7c4986d449-jkk5x\" (UID: \"c8e98df5-b2f7-49dd-a565-ed60534da496\") " pod="openshift-insights/insights-operator-7c4986d449-jkk5x"
Oct 31 01:03:52.997391 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:52.997030    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/ff4dcba2-df8d-4324-9e0c-abb23a5e6982-registry-certificates\") pod \"image-registry-56d8464d7c-vpwjc\" (UID: \"ff4dcba2-df8d-4324-9e0c-abb23a5e6982\") " pod="openshift-image-registry/image-registry-56d8464d7c-vpwjc"
Oct 31 01:03:52.997391 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:52.997059    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/f0c0ada7-7688-4dc2-97fc-813e4d0c4ef3-config\") pod \"console-operator-7cd56b8dbc-9257n\" (UID: \"f0c0ada7-7688-4dc2-97fc-813e4d0c4ef3\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-9257n"
Oct 31 01:03:52.997391 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:52.997061    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/318b3bc6-6e59-4211-af63-b116526a325d-device-dir\") pod \"aws-ebs-csi-driver-node-snxz4\" (UID: \"318b3bc6-6e59-4211-af63-b116526a325d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-snxz4"
Oct 31 01:03:52.997391 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:52.997095    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/147bb9af-dfc7-4785-bb5e-906f25d65dd1-cert\") pod \"ingress-canary-7dpwv\" (UID: \"147bb9af-dfc7-4785-bb5e-906f25d65dd1\") " pod="openshift-ingress-canary/ingress-canary-7dpwv"
Oct 31 01:03:52.997391 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:52.997124    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nmlm7\" (UniqueName: \"kubernetes.io/projected/fa085dd8-ca1e-4ec5-acb2-9b9dc878bf8a-kube-api-access-nmlm7\") pod \"multus-additional-cni-plugins-jwthj\" (UID: \"fa085dd8-ca1e-4ec5-acb2-9b9dc878bf8a\") " pod="openshift-multus/multus-additional-cni-plugins-jwthj"
Oct 31 01:03:52.997391 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:52.997147    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dtjj5\" (UniqueName: \"kubernetes.io/projected/6fffc15e-94bd-4939-b38f-17d1fe6cc0b5-kube-api-access-dtjj5\") pod \"service-ca-5fffc97bdf-xhsdf\" (UID: \"6fffc15e-94bd-4939-b38f-17d1fe6cc0b5\") " pod="openshift-service-ca/service-ca-5fffc97bdf-xhsdf"
Oct 31 01:03:52.997391 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:52.997165    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lldxn\" (UniqueName: \"kubernetes.io/projected/f273b9e5-0419-4587-a9c7-a0c8bec02acf-kube-api-access-lldxn\") pod \"cluster-samples-operator-6ddb6c8b94-srcwl\" (UID: \"f273b9e5-0419-4587-a9c7-a0c8bec02acf\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-srcwl"
Oct 31 01:03:52.997391 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:52.997183    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/318b3bc6-6e59-4211-af63-b116526a325d-registration-dir\") pod \"aws-ebs-csi-driver-node-snxz4\" (UID: \"318b3bc6-6e59-4211-af63-b116526a325d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-snxz4"
Oct 31 01:03:52.997391 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:52.997146    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/ff4dcba2-df8d-4324-9e0c-abb23a5e6982-registry-certificates\") pod \"image-registry-56d8464d7c-vpwjc\" (UID: \"ff4dcba2-df8d-4324-9e0c-abb23a5e6982\") " pod="openshift-image-registry/image-registry-56d8464d7c-vpwjc"
Oct 31 01:03:52.997391 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:52.997210    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/aa4836a5-c71a-485d-82fc-f8b53dae7717-kubelet-config\") pod \"global-pull-secret-syncer-zs5g7\" (UID: \"aa4836a5-c71a-485d-82fc-f8b53dae7717\") " pod="kube-system/global-pull-secret-syncer-zs5g7"
Oct 31 01:03:52.997391 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:52.997222    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/9cd1266e-c630-4259-b1f7-1b88dbde7385-iptables-alerter-script\") pod \"iptables-alerter-ssw8w\" (UID: \"9cd1266e-c630-4259-b1f7-1b88dbde7385\") " pod="openshift-network-operator/iptables-alerter-ssw8w"
Oct 31 01:03:52.997391 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:52.997237    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d55eb70a-f1a8-4346-b8df-ccd2c598dd6b-host-run-netns\") pod \"ovnkube-node-pghw2\" (UID: \"d55eb70a-f1a8-4346-b8df-ccd2c598dd6b\") " pod="openshift-ovn-kubernetes/ovnkube-node-pghw2"
Oct 31 01:03:52.997391 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:52.997276    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/d55eb70a-f1a8-4346-b8df-ccd2c598dd6b-ovn-node-metrics-cert\") pod \"ovnkube-node-pghw2\" (UID: \"d55eb70a-f1a8-4346-b8df-ccd2c598dd6b\") " pod="openshift-ovn-kubernetes/ovnkube-node-pghw2"
Oct 31 01:03:52.997391 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:52.997334    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qzhpr\" (UniqueName: \"kubernetes.io/projected/b5975856-8eb3-4601-a7e1-d1b1b3a957e8-kube-api-access-qzhpr\") pod \"migrator-66df5d496b-s2r9l\" (UID: \"b5975856-8eb3-4601-a7e1-d1b1b3a957e8\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-s2r9l"
Oct 31 01:03:52.997391 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:52.997338    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/147bb9af-dfc7-4785-bb5e-906f25d65dd1-cert\") pod \"ingress-canary-7dpwv\" (UID: \"147bb9af-dfc7-4785-bb5e-906f25d65dd1\") " pod="openshift-ingress-canary/ingress-canary-7dpwv"
Oct 31 01:03:52.997391 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:52.997372    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/aa112079-b749-404c-9081-113220707fcd-bound-sa-token\") pod \"image-registry-75575b5db8-wtq4d\" (UID: \"aa112079-b749-404c-9081-113220707fcd\") " pod="openshift-image-registry/image-registry-75575b5db8-wtq4d"
Oct 31 01:03:52.997391 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:52.997403    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/f0c0ada7-7688-4dc2-97fc-813e4d0c4ef3-serving-cert\") pod \"console-operator-7cd56b8dbc-9257n\" (UID: \"f0c0ada7-7688-4dc2-97fc-813e4d0c4ef3\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-9257n"
Oct 31 01:03:52.997391 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:52.997429    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rms4s\" (UniqueName: \"kubernetes.io/projected/318b3bc6-6e59-4211-af63-b116526a325d-kube-api-access-rms4s\") pod \"aws-ebs-csi-driver-node-snxz4\" (UID: \"318b3bc6-6e59-4211-af63-b116526a325d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-snxz4"
Oct 31 01:03:52.999069 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:52.997436    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/d55eb70a-f1a8-4346-b8df-ccd2c598dd6b-ovn-node-metrics-cert\") pod \"ovnkube-node-pghw2\" (UID: \"d55eb70a-f1a8-4346-b8df-ccd2c598dd6b\") " pod="openshift-ovn-kubernetes/ovnkube-node-pghw2"
Oct 31 01:03:52.999069 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:52.997456    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/08490837-b3b6-4c2b-a498-568af0d7086d-run\") pod \"tuned-45bvr\" (UID: \"08490837-b3b6-4c2b-a498-568af0d7086d\") " pod="openshift-cluster-node-tuning-operator/tuned-45bvr"
Oct 31 01:03:52.999069 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:52.997479    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/08490837-b3b6-4c2b-a498-568af0d7086d-host\") pod \"tuned-45bvr\" (UID: \"08490837-b3b6-4c2b-a498-568af0d7086d\") " pod="openshift-cluster-node-tuning-operator/tuned-45bvr"
Oct 31 01:03:52.999069 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:52.997513    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/d55eb70a-f1a8-4346-b8df-ccd2c598dd6b-systemd-units\") pod \"ovnkube-node-pghw2\" (UID: \"d55eb70a-f1a8-4346-b8df-ccd2c598dd6b\") " pod="openshift-ovn-kubernetes/ovnkube-node-pghw2"
Oct 31 01:03:52.999069 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:52.997551    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/e54a9dc6-36e6-4f1b-9c3a-623a520e11da-config\") pod \"service-ca-operator-544d749c8c-pdzj4\" (UID: \"e54a9dc6-36e6-4f1b-9c3a-623a520e11da\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-pdzj4"
Oct 31 01:03:52.999069 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:52.997565    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/f0c0ada7-7688-4dc2-97fc-813e4d0c4ef3-serving-cert\") pod \"console-operator-7cd56b8dbc-9257n\" (UID: \"f0c0ada7-7688-4dc2-97fc-813e4d0c4ef3\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-9257n"
Oct 31 01:03:52.999069 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:52.997582    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/c2be658b-ec92-4782-a78b-d2dccf2249cb-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-n5q9v\" (UID: \"c2be658b-ec92-4782-a78b-d2dccf2249cb\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-n5q9v"
Oct 31 01:03:52.999069 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:52.997617    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8952fc79-c5a6-42ba-a296-8cb7a776f718-metrics-certs\") pod \"network-metrics-daemon-wdsmn\" (UID: \"8952fc79-c5a6-42ba-a296-8cb7a776f718\") " pod="openshift-multus/network-metrics-daemon-wdsmn"
Oct 31 01:03:52.999069 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:52.997639    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7fhrx\" (UniqueName: \"kubernetes.io/projected/7dd98c21-e0b1-4289-8fe8-036bf413d444-kube-api-access-7fhrx\") pod \"insights-runtime-extractor-q56x5\" (UID: \"7dd98c21-e0b1-4289-8fe8-036bf413d444\") " pod="openshift-insights/insights-runtime-extractor-q56x5"
Oct 31 01:03:52.999069 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:52.997711    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/08490837-b3b6-4c2b-a498-568af0d7086d-lib-modules\") pod \"tuned-45bvr\" (UID: \"08490837-b3b6-4c2b-a498-568af0d7086d\") " pod="openshift-cluster-node-tuning-operator/tuned-45bvr"
Oct 31 01:03:52.999069 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:52.997720    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/e54a9dc6-36e6-4f1b-9c3a-623a520e11da-config\") pod \"service-ca-operator-544d749c8c-pdzj4\" (UID: \"e54a9dc6-36e6-4f1b-9c3a-623a520e11da\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-pdzj4"
Oct 31 01:03:52.999069 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:52.997735    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/d55eb70a-f1a8-4346-b8df-ccd2c598dd6b-node-log\") pod \"ovnkube-node-pghw2\" (UID: \"d55eb70a-f1a8-4346-b8df-ccd2c598dd6b\") " pod="openshift-ovn-kubernetes/ovnkube-node-pghw2"
Oct 31 01:03:52.999069 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:52.997763    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/d55eb70a-f1a8-4346-b8df-ccd2c598dd6b-env-overrides\") pod \"ovnkube-node-pghw2\" (UID: \"d55eb70a-f1a8-4346-b8df-ccd2c598dd6b\") " pod="openshift-ovn-kubernetes/ovnkube-node-pghw2"
Oct 31 01:03:52.999069 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:52.997792    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/9dfae6d6-b9a6-4d85-b594-9d947a9e6383-nginx-conf\") pod \"networking-console-plugin-86c478bb88-wm4qt\" (UID: \"9dfae6d6-b9a6-4d85-b594-9d947a9e6383\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-wm4qt"
Oct 31 01:03:52.999069 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:52.997823    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/c2be658b-ec92-4782-a78b-d2dccf2249cb-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-n5q9v\" (UID: \"c2be658b-ec92-4782-a78b-d2dccf2249cb\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-n5q9v"
Oct 31 01:03:52.999069 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:52.997825    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/b66619db-fd81-4435-be2a-a5ea6111e204-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-jccjk\" (UID: \"b66619db-fd81-4435-be2a-a5ea6111e204\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-jccjk"
Oct 31 01:03:52.999069 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:52.997864    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v9dnm\" (UniqueName: \"kubernetes.io/projected/ff4dcba2-df8d-4324-9e0c-abb23a5e6982-kube-api-access-v9dnm\") pod \"image-registry-56d8464d7c-vpwjc\" (UID: \"ff4dcba2-df8d-4324-9e0c-abb23a5e6982\") " pod="openshift-image-registry/image-registry-56d8464d7c-vpwjc"
Oct 31 01:03:52.999069 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:52.997924    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fa085dd8-ca1e-4ec5-acb2-9b9dc878bf8a-system-cni-dir\") pod \"multus-additional-cni-plugins-jwthj\" (UID: \"fa085dd8-ca1e-4ec5-acb2-9b9dc878bf8a\") " pod="openshift-multus/multus-additional-cni-plugins-jwthj"
Oct 31 01:03:52.999069 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:52.997934    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/d55eb70a-f1a8-4346-b8df-ccd2c598dd6b-env-overrides\") pod \"ovnkube-node-pghw2\" (UID: \"d55eb70a-f1a8-4346-b8df-ccd2c598dd6b\") " pod="openshift-ovn-kubernetes/ovnkube-node-pghw2"
Oct 31 01:03:52.999069 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:52.997936    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/9dfae6d6-b9a6-4d85-b594-9d947a9e6383-nginx-conf\") pod \"networking-console-plugin-86c478bb88-wm4qt\" (UID: \"9dfae6d6-b9a6-4d85-b594-9d947a9e6383\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-wm4qt"
Oct 31 01:03:52.999069 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:52.997950    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/d55eb70a-f1a8-4346-b8df-ccd2c598dd6b-run-systemd\") pod \"ovnkube-node-pghw2\" (UID: \"d55eb70a-f1a8-4346-b8df-ccd2c598dd6b\") " pod="openshift-ovn-kubernetes/ovnkube-node-pghw2"
Oct 31 01:03:52.999069 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:52.997982    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d55eb70a-f1a8-4346-b8df-ccd2c598dd6b-host-cni-bin\") pod \"ovnkube-node-pghw2\" (UID: \"d55eb70a-f1a8-4346-b8df-ccd2c598dd6b\") " pod="openshift-ovn-kubernetes/ovnkube-node-pghw2"
Oct 31 01:03:52.999069 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:52.998003    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7dd98c21-e0b1-4289-8fe8-036bf413d444-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-q56x5\" (UID: \"7dd98c21-e0b1-4289-8fe8-036bf413d444\") " pod="openshift-insights/insights-runtime-extractor-q56x5"
Oct 31 01:03:52.999069 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:52.998022    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/fa085dd8-ca1e-4ec5-acb2-9b9dc878bf8a-cni-binary-copy\") pod \"multus-additional-cni-plugins-jwthj\" (UID: \"fa085dd8-ca1e-4ec5-acb2-9b9dc878bf8a\") " pod="openshift-multus/multus-additional-cni-plugins-jwthj"
Oct 31 01:03:52.999069 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:52.998036    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/d55eb70a-f1a8-4346-b8df-ccd2c598dd6b-run-ovn\") pod \"ovnkube-node-pghw2\" (UID: \"d55eb70a-f1a8-4346-b8df-ccd2c598dd6b\") " pod="openshift-ovn-kubernetes/ovnkube-node-pghw2"
Oct 31 01:03:52.999069 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:52.998056    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/d55eb70a-f1a8-4346-b8df-ccd2c598dd6b-log-socket\") pod \"ovnkube-node-pghw2\" (UID: \"d55eb70a-f1a8-4346-b8df-ccd2c598dd6b\") " pod="openshift-ovn-kubernetes/ovnkube-node-pghw2"
Oct 31 01:03:52.999069 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:52.998096    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d55eb70a-f1a8-4346-b8df-ccd2c598dd6b-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-pghw2\" (UID: \"d55eb70a-f1a8-4346-b8df-ccd2c598dd6b\") " pod="openshift-ovn-kubernetes/ovnkube-node-pghw2"
Oct 31 01:03:52.999069 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:52.998113    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/318b3bc6-6e59-4211-af63-b116526a325d-sys-fs\") pod \"aws-ebs-csi-driver-node-snxz4\" (UID: \"318b3bc6-6e59-4211-af63-b116526a325d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-snxz4"
Oct 31 01:03:52.999069 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:52.998147    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7dd98c21-e0b1-4289-8fe8-036bf413d444-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-q56x5\" (UID: \"7dd98c21-e0b1-4289-8fe8-036bf413d444\") " pod="openshift-insights/insights-runtime-extractor-q56x5"
Oct 31 01:03:52.999069 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:52.998172    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/08490837-b3b6-4c2b-a498-568af0d7086d-var-lib-kubelet\") pod \"tuned-45bvr\" (UID: \"08490837-b3b6-4c2b-a498-568af0d7086d\") " pod="openshift-cluster-node-tuning-operator/tuned-45bvr"
Oct 31 01:03:52.999069 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:52.998217    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/58b0b75a-1568-4703-ac55-fa87f29225ad-host-var-lib-cni-bin\") pod \"multus-hxh6j\" (UID: \"58b0b75a-1568-4703-ac55-fa87f29225ad\") " pod="openshift-multus/multus-hxh6j"
Oct 31 01:03:52.999069 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:52.998230    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/fa085dd8-ca1e-4ec5-acb2-9b9dc878bf8a-cni-binary-copy\") pod \"multus-additional-cni-plugins-jwthj\" (UID: \"fa085dd8-ca1e-4ec5-acb2-9b9dc878bf8a\") " pod="openshift-multus/multus-additional-cni-plugins-jwthj"
Oct 31 01:03:52.999069 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:52.998247    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e49453a8-c7d9-4248-a8ef-4a819b76806c-metrics-certs\") pod \"router-default-5964fff8f9-8k6pp\" (UID: \"e49453a8-c7d9-4248-a8ef-4a819b76806c\") " pod="openshift-ingress/router-default-5964fff8f9-8k6pp"
Oct 31 01:03:52.999069 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:52.998268    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/08490837-b3b6-4c2b-a498-568af0d7086d-tmp\") pod \"tuned-45bvr\" (UID: \"08490837-b3b6-4c2b-a498-568af0d7086d\") " pod="openshift-cluster-node-tuning-operator/tuned-45bvr"
Oct 31 01:03:52.999069 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:52.998291    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d55eb70a-f1a8-4346-b8df-ccd2c598dd6b-etc-openvswitch\") pod \"ovnkube-node-pghw2\" (UID: \"d55eb70a-f1a8-4346-b8df-ccd2c598dd6b\") " pod="openshift-ovn-kubernetes/ovnkube-node-pghw2"
Oct 31 01:03:52.999069 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:52.998334    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/08490837-b3b6-4c2b-a498-568af0d7086d-tmp\") pod \"tuned-45bvr\" (UID: \"08490837-b3b6-4c2b-a498-568af0d7086d\") " pod="openshift-cluster-node-tuning-operator/tuned-45bvr"
Oct 31 01:03:53.050667 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:53.050647    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8n8tt\" (UniqueName: \"kubernetes.io/projected/85bb69db-c69b-467e-adeb-950bf2947c94-kube-api-access-8n8tt\") pod \"dns-default-t7r6c\" (UID: \"85bb69db-c69b-467e-adeb-950bf2947c94\") " pod="openshift-dns/dns-default-t7r6c"
Oct 31 01:03:53.052023 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:53.052002    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-59kdl\" (UniqueName: \"kubernetes.io/projected/b66619db-fd81-4435-be2a-a5ea6111e204-kube-api-access-59kdl\") pod \"cluster-monitoring-operator-884576c57-jccjk\" (UID: \"b66619db-fd81-4435-be2a-a5ea6111e204\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-jccjk"
Oct 31 01:03:53.052137 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:53.052124    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ftdwt\" (UniqueName: \"kubernetes.io/projected/d55eb70a-f1a8-4346-b8df-ccd2c598dd6b-kube-api-access-ftdwt\") pod \"ovnkube-node-pghw2\" (UID: \"d55eb70a-f1a8-4346-b8df-ccd2c598dd6b\") " pod="openshift-ovn-kubernetes/ovnkube-node-pghw2"
Oct 31 01:03:53.052710 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:53.052694    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rz545\" (UniqueName: \"kubernetes.io/projected/d719eda9-bc9f-4307-952a-99d99aa0f62b-kube-api-access-rz545\") pod \"kas-connection-checker-685895c7c9-jm6bm\" (UID: \"d719eda9-bc9f-4307-952a-99d99aa0f62b\") " pod="kube-system/kas-connection-checker-685895c7c9-jm6bm"
Oct 31 01:03:53.053354 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:53.053336    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-98n75\" (UniqueName: \"kubernetes.io/projected/e49453a8-c7d9-4248-a8ef-4a819b76806c-kube-api-access-98n75\") pod \"router-default-5964fff8f9-8k6pp\" (UID: \"e49453a8-c7d9-4248-a8ef-4a819b76806c\") " pod="openshift-ingress/router-default-5964fff8f9-8k6pp"
Oct 31 01:03:53.053354 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:53.053346    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zlmlp\" (UniqueName: \"kubernetes.io/projected/f0c0ada7-7688-4dc2-97fc-813e4d0c4ef3-kube-api-access-zlmlp\") pod \"console-operator-7cd56b8dbc-9257n\" (UID: \"f0c0ada7-7688-4dc2-97fc-813e4d0c4ef3\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-9257n"
Oct 31 01:03:53.058020 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:53.058005    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r4l7n\" (UniqueName: \"kubernetes.io/projected/9122c2fd-f29f-4e74-9615-15078517a3c0-kube-api-access-r4l7n\") pod \"periodic-gathering-b2z6s-vxz9h\" (UID: \"9122c2fd-f29f-4e74-9615-15078517a3c0\") " pod="openshift-insights/periodic-gathering-b2z6s-vxz9h"
Oct 31 01:03:53.058134 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:53.058111    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6qc9x\" (UniqueName: \"kubernetes.io/projected/69ff2b62-4588-4cb0-bdad-747c3692138c-kube-api-access-6qc9x\") pod \"network-check-source-6f78d59f99-bpmjc\" (UID: \"69ff2b62-4588-4cb0-bdad-747c3692138c\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-bpmjc"
Oct 31 01:03:53.059642 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:53.059627    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jxmsl\" (UniqueName: \"kubernetes.io/projected/9cd1266e-c630-4259-b1f7-1b88dbde7385-kube-api-access-jxmsl\") pod \"iptables-alerter-ssw8w\" (UID: \"9cd1266e-c630-4259-b1f7-1b88dbde7385\") " pod="openshift-network-operator/iptables-alerter-ssw8w"
Oct 31 01:03:53.068488 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:53.065828    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tlnlx\" (UniqueName: \"kubernetes.io/projected/c8e98df5-b2f7-49dd-a565-ed60534da496-kube-api-access-tlnlx\") pod \"insights-operator-7c4986d449-jkk5x\" (UID: \"c8e98df5-b2f7-49dd-a565-ed60534da496\") " pod="openshift-insights/insights-operator-7c4986d449-jkk5x"
Oct 31 01:03:53.069210 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:53.069191    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bspb5\" (UniqueName: \"kubernetes.io/projected/cb657366-6696-4c3e-a3a3-603ce00cf11a-kube-api-access-bspb5\") pod \"migrator-66df5d496b-cqpmt\" (UID: \"cb657366-6696-4c3e-a3a3-603ce00cf11a\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-cqpmt"
Oct 31 01:03:53.070735 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:53.070717    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vms79\" (UniqueName: \"kubernetes.io/projected/0ab3954e-be64-4260-bca8-ff5d58bd6ea2-kube-api-access-vms79\") pod \"kas-connection-checker-685895c7c9-q4z5f\" (UID: \"0ab3954e-be64-4260-bca8-ff5d58bd6ea2\") " pod="kube-system/kas-connection-checker-685895c7c9-q4z5f"
Oct 31 01:03:53.072011 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:53.071985    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bjgwm\" (UniqueName: \"kubernetes.io/projected/c2be658b-ec92-4782-a78b-d2dccf2249cb-kube-api-access-bjgwm\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-n5q9v\" (UID: \"c2be658b-ec92-4782-a78b-d2dccf2249cb\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-n5q9v"
Oct 31 01:03:53.072099 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:53.072046    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rms4s\" (UniqueName: \"kubernetes.io/projected/318b3bc6-6e59-4211-af63-b116526a325d-kube-api-access-rms4s\") pod \"aws-ebs-csi-driver-node-snxz4\" (UID: \"318b3bc6-6e59-4211-af63-b116526a325d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-snxz4"
Oct 31 01:03:53.072099 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:53.072078    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ff4dcba2-df8d-4324-9e0c-abb23a5e6982-bound-sa-token\") pod \"image-registry-56d8464d7c-vpwjc\" (UID: \"ff4dcba2-df8d-4324-9e0c-abb23a5e6982\") " pod="openshift-image-registry/image-registry-56d8464d7c-vpwjc"
Oct 31 01:03:53.072381 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:53.072365    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r6zx9\" (UniqueName: \"kubernetes.io/projected/565fefaf-82aa-42a8-9cc7-6517c4bddf1e-kube-api-access-r6zx9\") pod \"network-check-target-8mqbd\" (UID: \"565fefaf-82aa-42a8-9cc7-6517c4bddf1e\") " pod="openshift-network-diagnostics/network-check-target-8mqbd"
Oct 31 01:03:53.072442 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:53.072420    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rpmxj\" (UniqueName: \"kubernetes.io/projected/aa112079-b749-404c-9081-113220707fcd-kube-api-access-rpmxj\") pod \"image-registry-75575b5db8-wtq4d\" (UID: \"aa112079-b749-404c-9081-113220707fcd\") " pod="openshift-image-registry/image-registry-75575b5db8-wtq4d"
Oct 31 01:03:53.072577 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:53.072563    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2dx97\" (UniqueName: \"kubernetes.io/projected/57c2f196-0d8a-4bce-a5bc-3b6b5c0d808c-kube-api-access-2dx97\") pod \"node-resolver-j46hv\" (UID: \"57c2f196-0d8a-4bce-a5bc-3b6b5c0d808c\") " pod="openshift-dns/node-resolver-j46hv"
Oct 31 01:03:53.072680 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:53.072665    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jkn67\" (UniqueName: \"kubernetes.io/projected/da8395ef-7464-410e-96ee-b27378a03fed-kube-api-access-jkn67\") pod \"kas-connection-checker-685895c7c9-vbbq6\" (UID: \"da8395ef-7464-410e-96ee-b27378a03fed\") " pod="kube-system/kas-connection-checker-685895c7c9-vbbq6"
Oct 31 01:03:53.072735 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:53.072722    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qzhpr\" (UniqueName: \"kubernetes.io/projected/b5975856-8eb3-4601-a7e1-d1b1b3a957e8-kube-api-access-qzhpr\") pod \"migrator-66df5d496b-s2r9l\" (UID: \"b5975856-8eb3-4601-a7e1-d1b1b3a957e8\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-s2r9l"
Oct 31 01:03:53.073275 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:53.073252    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-srkbh\" (UniqueName: \"kubernetes.io/projected/8952fc79-c5a6-42ba-a296-8cb7a776f718-kube-api-access-srkbh\") pod \"network-metrics-daemon-wdsmn\" (UID: \"8952fc79-c5a6-42ba-a296-8cb7a776f718\") " pod="openshift-multus/network-metrics-daemon-wdsmn"
Oct 31 01:03:53.073450 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:53.073432    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g9wls\" (UniqueName: \"kubernetes.io/projected/147bb9af-dfc7-4785-bb5e-906f25d65dd1-kube-api-access-g9wls\") pod \"ingress-canary-7dpwv\" (UID: \"147bb9af-dfc7-4785-bb5e-906f25d65dd1\") " pod="openshift-ingress-canary/ingress-canary-7dpwv"
Oct 31 01:03:53.073510 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:53.073482    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ks6fn\" (UniqueName: \"kubernetes.io/projected/e54a9dc6-36e6-4f1b-9c3a-623a520e11da-kube-api-access-ks6fn\") pod \"service-ca-operator-544d749c8c-pdzj4\" (UID: \"e54a9dc6-36e6-4f1b-9c3a-623a520e11da\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-pdzj4"
Oct 31 01:03:53.073573 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:53.073512    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t228s\" (UniqueName: \"kubernetes.io/projected/58b0b75a-1568-4703-ac55-fa87f29225ad-kube-api-access-t228s\") pod \"multus-hxh6j\" (UID: \"58b0b75a-1568-4703-ac55-fa87f29225ad\") " pod="openshift-multus/multus-hxh6j"
Oct 31 01:03:53.073676 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:53.073660    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dtjj5\" (UniqueName: \"kubernetes.io/projected/6fffc15e-94bd-4939-b38f-17d1fe6cc0b5-kube-api-access-dtjj5\") pod \"service-ca-5fffc97bdf-xhsdf\" (UID: \"6fffc15e-94bd-4939-b38f-17d1fe6cc0b5\") " pod="openshift-service-ca/service-ca-5fffc97bdf-xhsdf"
Oct 31 01:03:53.073908 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:53.073891    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7fhrx\" (UniqueName: \"kubernetes.io/projected/7dd98c21-e0b1-4289-8fe8-036bf413d444-kube-api-access-7fhrx\") pod \"insights-runtime-extractor-q56x5\" (UID: \"7dd98c21-e0b1-4289-8fe8-036bf413d444\") " pod="openshift-insights/insights-runtime-extractor-q56x5"
Oct 31 01:03:53.074044 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:53.074026    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v9dnm\" (UniqueName: \"kubernetes.io/projected/ff4dcba2-df8d-4324-9e0c-abb23a5e6982-kube-api-access-v9dnm\") pod \"image-registry-56d8464d7c-vpwjc\" (UID: \"ff4dcba2-df8d-4324-9e0c-abb23a5e6982\") " pod="openshift-image-registry/image-registry-56d8464d7c-vpwjc"
Oct 31 01:03:53.074228 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:53.074213    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nmlm7\" (UniqueName: \"kubernetes.io/projected/fa085dd8-ca1e-4ec5-acb2-9b9dc878bf8a-kube-api-access-nmlm7\") pod \"multus-additional-cni-plugins-jwthj\" (UID: \"fa085dd8-ca1e-4ec5-acb2-9b9dc878bf8a\") " pod="openshift-multus/multus-additional-cni-plugins-jwthj"
Oct 31 01:03:53.074958 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:53.074937    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lldxn\" (UniqueName: \"kubernetes.io/projected/f273b9e5-0419-4587-a9c7-a0c8bec02acf-kube-api-access-lldxn\") pod \"cluster-samples-operator-6ddb6c8b94-srcwl\" (UID: \"f273b9e5-0419-4587-a9c7-a0c8bec02acf\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-srcwl"
Oct 31 01:03:53.075269 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:53.075247    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wsrb6\" (UniqueName: \"kubernetes.io/projected/dd8134f9-b2a6-4542-82ec-c10edc3fcd72-kube-api-access-wsrb6\") pod \"node-ca-rlxt2\" (UID: \"dd8134f9-b2a6-4542-82ec-c10edc3fcd72\") " pod="openshift-image-registry/node-ca-rlxt2"
Oct 31 01:03:53.075374 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:53.075354    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j78lj\" (UniqueName: \"kubernetes.io/projected/eafc69c5-15f1-4e4c-97ca-be586e55fbae-kube-api-access-j78lj\") pod \"volume-data-source-validator-58c9f5f46d-v585z\" (UID: \"eafc69c5-15f1-4e4c-97ca-be586e55fbae\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-v585z"
Oct 31 01:03:53.075679 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:53.075661    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w9tzc\" (UniqueName: \"kubernetes.io/projected/08490837-b3b6-4c2b-a498-568af0d7086d-kube-api-access-w9tzc\") pod \"tuned-45bvr\" (UID: \"08490837-b3b6-4c2b-a498-568af0d7086d\") " pod="openshift-cluster-node-tuning-operator/tuned-45bvr"
Oct 31 01:03:53.075821 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:53.075804    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/aa112079-b749-404c-9081-113220707fcd-bound-sa-token\") pod \"image-registry-75575b5db8-wtq4d\" (UID: \"aa112079-b749-404c-9081-113220707fcd\") " pod="openshift-image-registry/image-registry-75575b5db8-wtq4d"
Oct 31 01:03:53.080489 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:53.080459    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:03:53.099381 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:53.099358    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/7dd98c21-e0b1-4289-8fe8-036bf413d444-crio-socket\") pod \"insights-runtime-extractor-q56x5\" (UID: \"7dd98c21-e0b1-4289-8fe8-036bf413d444\") " pod="openshift-insights/insights-runtime-extractor-q56x5"
Oct 31 01:03:53.099456 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:53.099383    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d55eb70a-f1a8-4346-b8df-ccd2c598dd6b-run-openvswitch\") pod \"ovnkube-node-pghw2\" (UID: \"d55eb70a-f1a8-4346-b8df-ccd2c598dd6b\") " pod="openshift-ovn-kubernetes/ovnkube-node-pghw2"
Oct 31 01:03:53.099456 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:53.099401    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/fa085dd8-ca1e-4ec5-acb2-9b9dc878bf8a-cnibin\") pod \"multus-additional-cni-plugins-jwthj\" (UID: \"fa085dd8-ca1e-4ec5-acb2-9b9dc878bf8a\") " pod="openshift-multus/multus-additional-cni-plugins-jwthj"
Oct 31 01:03:53.099456 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:53.099418    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/08490837-b3b6-4c2b-a498-568af0d7086d-etc-systemd\") pod \"tuned-45bvr\" (UID: \"08490837-b3b6-4c2b-a498-568af0d7086d\") " pod="openshift-cluster-node-tuning-operator/tuned-45bvr"
Oct 31 01:03:53.099456 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:53.099440    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/58b0b75a-1568-4703-ac55-fa87f29225ad-host-run-multus-certs\") pod \"multus-hxh6j\" (UID: \"58b0b75a-1568-4703-ac55-fa87f29225ad\") " pod="openshift-multus/multus-hxh6j"
Oct 31 01:03:53.099632 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:53.099462    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/08490837-b3b6-4c2b-a498-568af0d7086d-etc-modprobe-d\") pod \"tuned-45bvr\" (UID: \"08490837-b3b6-4c2b-a498-568af0d7086d\") " pod="openshift-cluster-node-tuning-operator/tuned-45bvr"
Oct 31 01:03:53.099632 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:53.099463    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d55eb70a-f1a8-4346-b8df-ccd2c598dd6b-run-openvswitch\") pod \"ovnkube-node-pghw2\" (UID: \"d55eb70a-f1a8-4346-b8df-ccd2c598dd6b\") " pod="openshift-ovn-kubernetes/ovnkube-node-pghw2"
Oct 31 01:03:53.099632 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:53.099484    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/dd8134f9-b2a6-4542-82ec-c10edc3fcd72-host\") pod \"node-ca-rlxt2\" (UID: \"dd8134f9-b2a6-4542-82ec-c10edc3fcd72\") " pod="openshift-image-registry/node-ca-rlxt2"
Oct 31 01:03:53.099632 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:53.099494    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/fa085dd8-ca1e-4ec5-acb2-9b9dc878bf8a-cnibin\") pod \"multus-additional-cni-plugins-jwthj\" (UID: \"fa085dd8-ca1e-4ec5-acb2-9b9dc878bf8a\") " pod="openshift-multus/multus-additional-cni-plugins-jwthj"
Oct 31 01:03:53.099632 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:53.099504    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/d55eb70a-f1a8-4346-b8df-ccd2c598dd6b-host-cni-netd\") pod \"ovnkube-node-pghw2\" (UID: \"d55eb70a-f1a8-4346-b8df-ccd2c598dd6b\") " pod="openshift-ovn-kubernetes/ovnkube-node-pghw2"
Oct 31 01:03:53.099632 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:53.099509    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/08490837-b3b6-4c2b-a498-568af0d7086d-etc-systemd\") pod \"tuned-45bvr\" (UID: \"08490837-b3b6-4c2b-a498-568af0d7086d\") " pod="openshift-cluster-node-tuning-operator/tuned-45bvr"
Oct 31 01:03:53.099632 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:53.099507    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/58b0b75a-1568-4703-ac55-fa87f29225ad-host-run-multus-certs\") pod \"multus-hxh6j\" (UID: \"58b0b75a-1568-4703-ac55-fa87f29225ad\") " pod="openshift-multus/multus-hxh6j"
Oct 31 01:03:53.099632 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:53.099525    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/58b0b75a-1568-4703-ac55-fa87f29225ad-host-var-lib-cni-multus\") pod \"multus-hxh6j\" (UID: \"58b0b75a-1568-4703-ac55-fa87f29225ad\") " pod="openshift-multus/multus-hxh6j"
Oct 31 01:03:53.099632 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:53.099547    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/dd8134f9-b2a6-4542-82ec-c10edc3fcd72-host\") pod \"node-ca-rlxt2\" (UID: \"dd8134f9-b2a6-4542-82ec-c10edc3fcd72\") " pod="openshift-image-registry/node-ca-rlxt2"
Oct 31 01:03:53.099632 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:53.099551    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/d55eb70a-f1a8-4346-b8df-ccd2c598dd6b-host-cni-netd\") pod \"ovnkube-node-pghw2\" (UID: \"d55eb70a-f1a8-4346-b8df-ccd2c598dd6b\") " pod="openshift-ovn-kubernetes/ovnkube-node-pghw2"
Oct 31 01:03:53.099632 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:53.099548    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d55eb70a-f1a8-4346-b8df-ccd2c598dd6b-host-slash\") pod \"ovnkube-node-pghw2\" (UID: \"d55eb70a-f1a8-4346-b8df-ccd2c598dd6b\") " pod="openshift-ovn-kubernetes/ovnkube-node-pghw2"
Oct 31 01:03:53.099632 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:53.099563    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/08490837-b3b6-4c2b-a498-568af0d7086d-etc-modprobe-d\") pod \"tuned-45bvr\" (UID: \"08490837-b3b6-4c2b-a498-568af0d7086d\") " pod="openshift-cluster-node-tuning-operator/tuned-45bvr"
Oct 31 01:03:53.099632 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:53.099585    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/58b0b75a-1568-4703-ac55-fa87f29225ad-etc-kubernetes\") pod \"multus-hxh6j\" (UID: \"58b0b75a-1568-4703-ac55-fa87f29225ad\") " pod="openshift-multus/multus-hxh6j"
Oct 31 01:03:53.099632 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:53.099590    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/7dd98c21-e0b1-4289-8fe8-036bf413d444-crio-socket\") pod \"insights-runtime-extractor-q56x5\" (UID: \"7dd98c21-e0b1-4289-8fe8-036bf413d444\") " pod="openshift-insights/insights-runtime-extractor-q56x5"
Oct 31 01:03:53.099632 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:53.099602    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/318b3bc6-6e59-4211-af63-b116526a325d-socket-dir\") pod \"aws-ebs-csi-driver-node-snxz4\" (UID: \"318b3bc6-6e59-4211-af63-b116526a325d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-snxz4"
Oct 31 01:03:53.099632 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:53.099585    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d55eb70a-f1a8-4346-b8df-ccd2c598dd6b-host-slash\") pod \"ovnkube-node-pghw2\" (UID: \"d55eb70a-f1a8-4346-b8df-ccd2c598dd6b\") " pod="openshift-ovn-kubernetes/ovnkube-node-pghw2"
Oct 31 01:03:53.099632 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:53.099607    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/58b0b75a-1568-4703-ac55-fa87f29225ad-host-var-lib-cni-multus\") pod \"multus-hxh6j\" (UID: \"58b0b75a-1568-4703-ac55-fa87f29225ad\") " pod="openshift-multus/multus-hxh6j"
Oct 31 01:03:53.099632 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:53.099616    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/aa4836a5-c71a-485d-82fc-f8b53dae7717-dbus\") pod \"global-pull-secret-syncer-zs5g7\" (UID: \"aa4836a5-c71a-485d-82fc-f8b53dae7717\") " pod="kube-system/global-pull-secret-syncer-zs5g7"
Oct 31 01:03:53.099632 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:53.099639    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/318b3bc6-6e59-4211-af63-b116526a325d-kubelet-dir\") pod \"aws-ebs-csi-driver-node-snxz4\" (UID: \"318b3bc6-6e59-4211-af63-b116526a325d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-snxz4"
Oct 31 01:03:53.100451 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:53.099644    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/58b0b75a-1568-4703-ac55-fa87f29225ad-etc-kubernetes\") pod \"multus-hxh6j\" (UID: \"58b0b75a-1568-4703-ac55-fa87f29225ad\") " pod="openshift-multus/multus-hxh6j"
Oct 31 01:03:53.100451 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:53.099652    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/58b0b75a-1568-4703-ac55-fa87f29225ad-host-var-lib-kubelet\") pod \"multus-hxh6j\" (UID: \"58b0b75a-1568-4703-ac55-fa87f29225ad\") " pod="openshift-multus/multus-hxh6j"
Oct 31 01:03:53.100451 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:53.099669    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/85bb69db-c69b-467e-adeb-950bf2947c94-metrics-tls\") pod \"dns-default-t7r6c\" (UID: \"85bb69db-c69b-467e-adeb-950bf2947c94\") " pod="openshift-dns/dns-default-t7r6c"
Oct 31 01:03:53.100451 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:53.099682    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/58b0b75a-1568-4703-ac55-fa87f29225ad-multus-cni-dir\") pod \"multus-hxh6j\" (UID: \"58b0b75a-1568-4703-ac55-fa87f29225ad\") " pod="openshift-multus/multus-hxh6j"
Oct 31 01:03:53.100451 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:53.099690    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/aa4836a5-c71a-485d-82fc-f8b53dae7717-dbus\") pod \"global-pull-secret-syncer-zs5g7\" (UID: \"aa4836a5-c71a-485d-82fc-f8b53dae7717\") " pod="kube-system/global-pull-secret-syncer-zs5g7"
Oct 31 01:03:53.100451 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:53.099698    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/318b3bc6-6e59-4211-af63-b116526a325d-device-dir\") pod \"aws-ebs-csi-driver-node-snxz4\" (UID: \"318b3bc6-6e59-4211-af63-b116526a325d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-snxz4"
Oct 31 01:03:53.100451 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:53.099690    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/318b3bc6-6e59-4211-af63-b116526a325d-kubelet-dir\") pod \"aws-ebs-csi-driver-node-snxz4\" (UID: \"318b3bc6-6e59-4211-af63-b116526a325d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-snxz4"
Oct 31 01:03:53.100451 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:53.099693    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/318b3bc6-6e59-4211-af63-b116526a325d-socket-dir\") pod \"aws-ebs-csi-driver-node-snxz4\" (UID: \"318b3bc6-6e59-4211-af63-b116526a325d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-snxz4"
Oct 31 01:03:53.100451 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:53.099719    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/318b3bc6-6e59-4211-af63-b116526a325d-registration-dir\") pod \"aws-ebs-csi-driver-node-snxz4\" (UID: \"318b3bc6-6e59-4211-af63-b116526a325d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-snxz4"
Oct 31 01:03:53.100451 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:53.099747    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/58b0b75a-1568-4703-ac55-fa87f29225ad-multus-cni-dir\") pod \"multus-hxh6j\" (UID: \"58b0b75a-1568-4703-ac55-fa87f29225ad\") " pod="openshift-multus/multus-hxh6j"
Oct 31 01:03:53.100451 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:53.099751    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/aa4836a5-c71a-485d-82fc-f8b53dae7717-kubelet-config\") pod \"global-pull-secret-syncer-zs5g7\" (UID: \"aa4836a5-c71a-485d-82fc-f8b53dae7717\") " pod="kube-system/global-pull-secret-syncer-zs5g7"
Oct 31 01:03:53.100451 ip-10-0-133-197 kubenswrapper[4964]: E1031 01:03:53.099765    4964 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 31 01:03:53.100451 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:53.099774    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d55eb70a-f1a8-4346-b8df-ccd2c598dd6b-host-run-netns\") pod \"ovnkube-node-pghw2\" (UID: \"d55eb70a-f1a8-4346-b8df-ccd2c598dd6b\") " pod="openshift-ovn-kubernetes/ovnkube-node-pghw2"
Oct 31 01:03:53.100451 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:53.099790    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/aa4836a5-c71a-485d-82fc-f8b53dae7717-kubelet-config\") pod \"global-pull-secret-syncer-zs5g7\" (UID: \"aa4836a5-c71a-485d-82fc-f8b53dae7717\") " pod="kube-system/global-pull-secret-syncer-zs5g7"
Oct 31 01:03:53.100451 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:53.099791    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/318b3bc6-6e59-4211-af63-b116526a325d-device-dir\") pod \"aws-ebs-csi-driver-node-snxz4\" (UID: \"318b3bc6-6e59-4211-af63-b116526a325d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-snxz4"
Oct 31 01:03:53.100451 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:53.099798    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/08490837-b3b6-4c2b-a498-568af0d7086d-run\") pod \"tuned-45bvr\" (UID: \"08490837-b3b6-4c2b-a498-568af0d7086d\") " pod="openshift-cluster-node-tuning-operator/tuned-45bvr"
Oct 31 01:03:53.100451 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:53.099751    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/318b3bc6-6e59-4211-af63-b116526a325d-registration-dir\") pod \"aws-ebs-csi-driver-node-snxz4\" (UID: \"318b3bc6-6e59-4211-af63-b116526a325d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-snxz4"
Oct 31 01:03:53.100451 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:53.099822    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/08490837-b3b6-4c2b-a498-568af0d7086d-host\") pod \"tuned-45bvr\" (UID: \"08490837-b3b6-4c2b-a498-568af0d7086d\") " pod="openshift-cluster-node-tuning-operator/tuned-45bvr"
Oct 31 01:03:53.100451 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:53.099822    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d55eb70a-f1a8-4346-b8df-ccd2c598dd6b-host-run-netns\") pod \"ovnkube-node-pghw2\" (UID: \"d55eb70a-f1a8-4346-b8df-ccd2c598dd6b\") " pod="openshift-ovn-kubernetes/ovnkube-node-pghw2"
Oct 31 01:03:53.100451 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:53.099831    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/08490837-b3b6-4c2b-a498-568af0d7086d-run\") pod \"tuned-45bvr\" (UID: \"08490837-b3b6-4c2b-a498-568af0d7086d\") " pod="openshift-cluster-node-tuning-operator/tuned-45bvr"
Oct 31 01:03:53.100451 ip-10-0-133-197 kubenswrapper[4964]: E1031 01:03:53.099836    4964 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/85bb69db-c69b-467e-adeb-950bf2947c94-metrics-tls podName:85bb69db-c69b-467e-adeb-950bf2947c94 nodeName:}" failed. No retries permitted until 2026-10-31 01:03:53.599819368 +0000 UTC m=+2.076385740 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/85bb69db-c69b-467e-adeb-950bf2947c94-metrics-tls") pod "dns-default-t7r6c" (UID: "85bb69db-c69b-467e-adeb-950bf2947c94") : secret "dns-default-metrics-tls" not found
Oct 31 01:03:53.100451 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:53.099765    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/58b0b75a-1568-4703-ac55-fa87f29225ad-host-var-lib-kubelet\") pod \"multus-hxh6j\" (UID: \"58b0b75a-1568-4703-ac55-fa87f29225ad\") " pod="openshift-multus/multus-hxh6j"
Oct 31 01:03:53.100451 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:53.099873    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/08490837-b3b6-4c2b-a498-568af0d7086d-host\") pod \"tuned-45bvr\" (UID: \"08490837-b3b6-4c2b-a498-568af0d7086d\") " pod="openshift-cluster-node-tuning-operator/tuned-45bvr"
Oct 31 01:03:53.100451 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:53.099879    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/d55eb70a-f1a8-4346-b8df-ccd2c598dd6b-systemd-units\") pod \"ovnkube-node-pghw2\" (UID: \"d55eb70a-f1a8-4346-b8df-ccd2c598dd6b\") " pod="openshift-ovn-kubernetes/ovnkube-node-pghw2"
Oct 31 01:03:53.100451 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:53.099902    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8952fc79-c5a6-42ba-a296-8cb7a776f718-metrics-certs\") pod \"network-metrics-daemon-wdsmn\" (UID: \"8952fc79-c5a6-42ba-a296-8cb7a776f718\") " pod="openshift-multus/network-metrics-daemon-wdsmn"
Oct 31 01:03:53.100451 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:53.099918    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/08490837-b3b6-4c2b-a498-568af0d7086d-lib-modules\") pod \"tuned-45bvr\" (UID: \"08490837-b3b6-4c2b-a498-568af0d7086d\") " pod="openshift-cluster-node-tuning-operator/tuned-45bvr"
Oct 31 01:03:53.100451 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:53.099924    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/d55eb70a-f1a8-4346-b8df-ccd2c598dd6b-systemd-units\") pod \"ovnkube-node-pghw2\" (UID: \"d55eb70a-f1a8-4346-b8df-ccd2c598dd6b\") " pod="openshift-ovn-kubernetes/ovnkube-node-pghw2"
Oct 31 01:03:53.100451 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:53.099932    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/d55eb70a-f1a8-4346-b8df-ccd2c598dd6b-node-log\") pod \"ovnkube-node-pghw2\" (UID: \"d55eb70a-f1a8-4346-b8df-ccd2c598dd6b\") " pod="openshift-ovn-kubernetes/ovnkube-node-pghw2"
Oct 31 01:03:53.100451 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:53.099952    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/d55eb70a-f1a8-4346-b8df-ccd2c598dd6b-node-log\") pod \"ovnkube-node-pghw2\" (UID: \"d55eb70a-f1a8-4346-b8df-ccd2c598dd6b\") " pod="openshift-ovn-kubernetes/ovnkube-node-pghw2"
Oct 31 01:03:53.100451 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:53.099964    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/b66619db-fd81-4435-be2a-a5ea6111e204-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-jccjk\" (UID: \"b66619db-fd81-4435-be2a-a5ea6111e204\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-jccjk"
Oct 31 01:03:53.100451 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:53.099992    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fa085dd8-ca1e-4ec5-acb2-9b9dc878bf8a-system-cni-dir\") pod \"multus-additional-cni-plugins-jwthj\" (UID: \"fa085dd8-ca1e-4ec5-acb2-9b9dc878bf8a\") " pod="openshift-multus/multus-additional-cni-plugins-jwthj"
Oct 31 01:03:53.100451 ip-10-0-133-197 kubenswrapper[4964]: E1031 01:03:53.100007    4964 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Oct 31 01:03:53.100451 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:53.100014    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/d55eb70a-f1a8-4346-b8df-ccd2c598dd6b-run-systemd\") pod \"ovnkube-node-pghw2\" (UID: \"d55eb70a-f1a8-4346-b8df-ccd2c598dd6b\") " pod="openshift-ovn-kubernetes/ovnkube-node-pghw2"
Oct 31 01:03:53.100451 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:53.100034    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/08490837-b3b6-4c2b-a498-568af0d7086d-lib-modules\") pod \"tuned-45bvr\" (UID: \"08490837-b3b6-4c2b-a498-568af0d7086d\") " pod="openshift-cluster-node-tuning-operator/tuned-45bvr"
Oct 31 01:03:53.100451 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:53.100037    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d55eb70a-f1a8-4346-b8df-ccd2c598dd6b-host-cni-bin\") pod \"ovnkube-node-pghw2\" (UID: \"d55eb70a-f1a8-4346-b8df-ccd2c598dd6b\") " pod="openshift-ovn-kubernetes/ovnkube-node-pghw2"
Oct 31 01:03:53.100451 ip-10-0-133-197 kubenswrapper[4964]: E1031 01:03:53.100056    4964 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8952fc79-c5a6-42ba-a296-8cb7a776f718-metrics-certs podName:8952fc79-c5a6-42ba-a296-8cb7a776f718 nodeName:}" failed. No retries permitted until 2026-10-31 01:03:53.600041667 +0000 UTC m=+2.076608027 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8952fc79-c5a6-42ba-a296-8cb7a776f718-metrics-certs") pod "network-metrics-daemon-wdsmn" (UID: "8952fc79-c5a6-42ba-a296-8cb7a776f718") : secret "metrics-daemon-secret" not found
Oct 31 01:03:53.100451 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:53.100068    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d55eb70a-f1a8-4346-b8df-ccd2c598dd6b-host-cni-bin\") pod \"ovnkube-node-pghw2\" (UID: \"d55eb70a-f1a8-4346-b8df-ccd2c598dd6b\") " pod="openshift-ovn-kubernetes/ovnkube-node-pghw2"
Oct 31 01:03:53.100451 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:53.100076    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fa085dd8-ca1e-4ec5-acb2-9b9dc878bf8a-system-cni-dir\") pod \"multus-additional-cni-plugins-jwthj\" (UID: \"fa085dd8-ca1e-4ec5-acb2-9b9dc878bf8a\") " pod="openshift-multus/multus-additional-cni-plugins-jwthj"
Oct 31 01:03:53.100451 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:53.100085    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7dd98c21-e0b1-4289-8fe8-036bf413d444-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-q56x5\" (UID: \"7dd98c21-e0b1-4289-8fe8-036bf413d444\") " pod="openshift-insights/insights-runtime-extractor-q56x5"
Oct 31 01:03:53.100451 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:53.100080    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/d55eb70a-f1a8-4346-b8df-ccd2c598dd6b-run-systemd\") pod \"ovnkube-node-pghw2\" (UID: \"d55eb70a-f1a8-4346-b8df-ccd2c598dd6b\") " pod="openshift-ovn-kubernetes/ovnkube-node-pghw2"
Oct 31 01:03:53.100451 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:53.100138    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/d55eb70a-f1a8-4346-b8df-ccd2c598dd6b-run-ovn\") pod \"ovnkube-node-pghw2\" (UID: \"d55eb70a-f1a8-4346-b8df-ccd2c598dd6b\") " pod="openshift-ovn-kubernetes/ovnkube-node-pghw2"
Oct 31 01:03:53.100451 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:53.100109    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/d55eb70a-f1a8-4346-b8df-ccd2c598dd6b-run-ovn\") pod \"ovnkube-node-pghw2\" (UID: \"d55eb70a-f1a8-4346-b8df-ccd2c598dd6b\") " pod="openshift-ovn-kubernetes/ovnkube-node-pghw2"
Oct 31 01:03:53.100451 ip-10-0-133-197 kubenswrapper[4964]: E1031 01:03:53.100174    4964 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 31 01:03:53.100451 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:53.100204    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/d55eb70a-f1a8-4346-b8df-ccd2c598dd6b-log-socket\") pod \"ovnkube-node-pghw2\" (UID: \"d55eb70a-f1a8-4346-b8df-ccd2c598dd6b\") " pod="openshift-ovn-kubernetes/ovnkube-node-pghw2"
Oct 31 01:03:53.100451 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:53.100174    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/d55eb70a-f1a8-4346-b8df-ccd2c598dd6b-log-socket\") pod \"ovnkube-node-pghw2\" (UID: \"d55eb70a-f1a8-4346-b8df-ccd2c598dd6b\") " pod="openshift-ovn-kubernetes/ovnkube-node-pghw2"
Oct 31 01:03:53.100451 ip-10-0-133-197 kubenswrapper[4964]: E1031 01:03:53.100222    4964 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7dd98c21-e0b1-4289-8fe8-036bf413d444-insights-runtime-extractor-tls podName:7dd98c21-e0b1-4289-8fe8-036bf413d444 nodeName:}" failed. No retries permitted until 2026-10-31 01:03:53.600207013 +0000 UTC m=+2.076773374 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/7dd98c21-e0b1-4289-8fe8-036bf413d444-insights-runtime-extractor-tls") pod "insights-runtime-extractor-q56x5" (UID: "7dd98c21-e0b1-4289-8fe8-036bf413d444") : secret "insights-runtime-extractor-tls" not found
Oct 31 01:03:53.100451 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:53.100253    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d55eb70a-f1a8-4346-b8df-ccd2c598dd6b-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-pghw2\" (UID: \"d55eb70a-f1a8-4346-b8df-ccd2c598dd6b\") " pod="openshift-ovn-kubernetes/ovnkube-node-pghw2"
Oct 31 01:03:53.100451 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:53.100277    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/318b3bc6-6e59-4211-af63-b116526a325d-sys-fs\") pod \"aws-ebs-csi-driver-node-snxz4\" (UID: \"318b3bc6-6e59-4211-af63-b116526a325d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-snxz4"
Oct 31 01:03:53.100451 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:53.100289    4964 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Oct 31 01:03:53.100451 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:53.100300    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7dd98c21-e0b1-4289-8fe8-036bf413d444-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-q56x5\" (UID: \"7dd98c21-e0b1-4289-8fe8-036bf413d444\") " pod="openshift-insights/insights-runtime-extractor-q56x5"
Oct 31 01:03:53.100451 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:53.100344    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d55eb70a-f1a8-4346-b8df-ccd2c598dd6b-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-pghw2\" (UID: \"d55eb70a-f1a8-4346-b8df-ccd2c598dd6b\") " pod="openshift-ovn-kubernetes/ovnkube-node-pghw2"
Oct 31 01:03:53.100451 ip-10-0-133-197 kubenswrapper[4964]: E1031 01:03:53.100370    4964 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Oct 31 01:03:53.100451 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:53.100376    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/318b3bc6-6e59-4211-af63-b116526a325d-sys-fs\") pod \"aws-ebs-csi-driver-node-snxz4\" (UID: \"318b3bc6-6e59-4211-af63-b116526a325d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-snxz4"
Oct 31 01:03:53.100451 ip-10-0-133-197 kubenswrapper[4964]: E1031 01:03:53.100429    4964 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/7dd98c21-e0b1-4289-8fe8-036bf413d444-kube-rbac-proxy-cm podName:7dd98c21-e0b1-4289-8fe8-036bf413d444 nodeName:}" failed. No retries permitted until 2026-10-31 01:03:53.600415283 +0000 UTC m=+2.076981655 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/7dd98c21-e0b1-4289-8fe8-036bf413d444-kube-rbac-proxy-cm") pod "insights-runtime-extractor-q56x5" (UID: "7dd98c21-e0b1-4289-8fe8-036bf413d444") : configmap "kube-rbac-proxy" not found
Oct 31 01:03:53.100451 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:53.100445    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/08490837-b3b6-4c2b-a498-568af0d7086d-var-lib-kubelet\") pod \"tuned-45bvr\" (UID: \"08490837-b3b6-4c2b-a498-568af0d7086d\") " pod="openshift-cluster-node-tuning-operator/tuned-45bvr"
Oct 31 01:03:53.100451 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:53.100469    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/58b0b75a-1568-4703-ac55-fa87f29225ad-host-var-lib-cni-bin\") pod \"multus-hxh6j\" (UID: \"58b0b75a-1568-4703-ac55-fa87f29225ad\") " pod="openshift-multus/multus-hxh6j"
Oct 31 01:03:53.102424 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:53.100491    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e49453a8-c7d9-4248-a8ef-4a819b76806c-metrics-certs\") pod \"router-default-5964fff8f9-8k6pp\" (UID: \"e49453a8-c7d9-4248-a8ef-4a819b76806c\") " pod="openshift-ingress/router-default-5964fff8f9-8k6pp"
Oct 31 01:03:53.102424 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:53.100514    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d55eb70a-f1a8-4346-b8df-ccd2c598dd6b-etc-openvswitch\") pod \"ovnkube-node-pghw2\" (UID: \"d55eb70a-f1a8-4346-b8df-ccd2c598dd6b\") " pod="openshift-ovn-kubernetes/ovnkube-node-pghw2"
Oct 31 01:03:53.102424 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:53.100529    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/58b0b75a-1568-4703-ac55-fa87f29225ad-host-var-lib-cni-bin\") pod \"multus-hxh6j\" (UID: \"58b0b75a-1568-4703-ac55-fa87f29225ad\") " pod="openshift-multus/multus-hxh6j"
Oct 31 01:03:53.102424 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:53.100527    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/08490837-b3b6-4c2b-a498-568af0d7086d-var-lib-kubelet\") pod \"tuned-45bvr\" (UID: \"08490837-b3b6-4c2b-a498-568af0d7086d\") " pod="openshift-cluster-node-tuning-operator/tuned-45bvr"
Oct 31 01:03:53.102424 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:53.100539    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/58b0b75a-1568-4703-ac55-fa87f29225ad-hostroot\") pod \"multus-hxh6j\" (UID: \"58b0b75a-1568-4703-ac55-fa87f29225ad\") " pod="openshift-multus/multus-hxh6j"
Oct 31 01:03:53.102424 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:53.100569    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/58b0b75a-1568-4703-ac55-fa87f29225ad-system-cni-dir\") pod \"multus-hxh6j\" (UID: \"58b0b75a-1568-4703-ac55-fa87f29225ad\") " pod="openshift-multus/multus-hxh6j"
Oct 31 01:03:53.102424 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:53.100570    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d55eb70a-f1a8-4346-b8df-ccd2c598dd6b-etc-openvswitch\") pod \"ovnkube-node-pghw2\" (UID: \"d55eb70a-f1a8-4346-b8df-ccd2c598dd6b\") " pod="openshift-ovn-kubernetes/ovnkube-node-pghw2"
Oct 31 01:03:53.102424 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:53.100576    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/58b0b75a-1568-4703-ac55-fa87f29225ad-hostroot\") pod \"multus-hxh6j\" (UID: \"58b0b75a-1568-4703-ac55-fa87f29225ad\") " pod="openshift-multus/multus-hxh6j"
Oct 31 01:03:53.102424 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:53.100584    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/58b0b75a-1568-4703-ac55-fa87f29225ad-multus-socket-dir-parent\") pod \"multus-hxh6j\" (UID: \"58b0b75a-1568-4703-ac55-fa87f29225ad\") " pod="openshift-multus/multus-hxh6j"
Oct 31 01:03:53.102424 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:53.100601    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/aa112079-b749-404c-9081-113220707fcd-registry-tls\") pod \"image-registry-75575b5db8-wtq4d\" (UID: \"aa112079-b749-404c-9081-113220707fcd\") " pod="openshift-image-registry/image-registry-75575b5db8-wtq4d"
Oct 31 01:03:53.102424 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:53.100618    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/57c2f196-0d8a-4bce-a5bc-3b6b5c0d808c-hosts-file\") pod \"node-resolver-j46hv\" (UID: \"57c2f196-0d8a-4bce-a5bc-3b6b5c0d808c\") " pod="openshift-dns/node-resolver-j46hv"
Oct 31 01:03:53.102424 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:53.100619    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/58b0b75a-1568-4703-ac55-fa87f29225ad-system-cni-dir\") pod \"multus-hxh6j\" (UID: \"58b0b75a-1568-4703-ac55-fa87f29225ad\") " pod="openshift-multus/multus-hxh6j"
Oct 31 01:03:53.102424 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:53.100633    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/58b0b75a-1568-4703-ac55-fa87f29225ad-multus-socket-dir-parent\") pod \"multus-hxh6j\" (UID: \"58b0b75a-1568-4703-ac55-fa87f29225ad\") " pod="openshift-multus/multus-hxh6j"
Oct 31 01:03:53.102424 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:53.100638    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/58b0b75a-1568-4703-ac55-fa87f29225ad-host-run-k8s-cni-cncf-io\") pod \"multus-hxh6j\" (UID: \"58b0b75a-1568-4703-ac55-fa87f29225ad\") " pod="openshift-multus/multus-hxh6j"
Oct 31 01:03:53.102424 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:53.100666    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/58b0b75a-1568-4703-ac55-fa87f29225ad-host-run-k8s-cni-cncf-io\") pod \"multus-hxh6j\" (UID: \"58b0b75a-1568-4703-ac55-fa87f29225ad\") " pod="openshift-multus/multus-hxh6j"
Oct 31 01:03:53.102424 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:53.100667    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/58b0b75a-1568-4703-ac55-fa87f29225ad-multus-conf-dir\") pod \"multus-hxh6j\" (UID: \"58b0b75a-1568-4703-ac55-fa87f29225ad\") " pod="openshift-multus/multus-hxh6j"
Oct 31 01:03:53.102424 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:53.100692    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/58b0b75a-1568-4703-ac55-fa87f29225ad-cnibin\") pod \"multus-hxh6j\" (UID: \"58b0b75a-1568-4703-ac55-fa87f29225ad\") " pod="openshift-multus/multus-hxh6j"
Oct 31 01:03:53.102424 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:53.100699    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/58b0b75a-1568-4703-ac55-fa87f29225ad-multus-conf-dir\") pod \"multus-hxh6j\" (UID: \"58b0b75a-1568-4703-ac55-fa87f29225ad\") " pod="openshift-multus/multus-hxh6j"
Oct 31 01:03:53.102424 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:53.100727    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/08490837-b3b6-4c2b-a498-568af0d7086d-etc-sysctl-d\") pod \"tuned-45bvr\" (UID: \"08490837-b3b6-4c2b-a498-568af0d7086d\") " pod="openshift-cluster-node-tuning-operator/tuned-45bvr"
Oct 31 01:03:53.102424 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:53.100735    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/57c2f196-0d8a-4bce-a5bc-3b6b5c0d808c-hosts-file\") pod \"node-resolver-j46hv\" (UID: \"57c2f196-0d8a-4bce-a5bc-3b6b5c0d808c\") " pod="openshift-dns/node-resolver-j46hv"
Oct 31 01:03:53.102424 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:53.100744    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/58b0b75a-1568-4703-ac55-fa87f29225ad-cnibin\") pod \"multus-hxh6j\" (UID: \"58b0b75a-1568-4703-ac55-fa87f29225ad\") " pod="openshift-multus/multus-hxh6j"
Oct 31 01:03:53.102424 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:53.100779    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d55eb70a-f1a8-4346-b8df-ccd2c598dd6b-host-run-ovn-kubernetes\") pod \"ovnkube-node-pghw2\" (UID: \"d55eb70a-f1a8-4346-b8df-ccd2c598dd6b\") " pod="openshift-ovn-kubernetes/ovnkube-node-pghw2"
Oct 31 01:03:53.102424 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:53.100804    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/9dfae6d6-b9a6-4d85-b594-9d947a9e6383-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-wm4qt\" (UID: \"9dfae6d6-b9a6-4d85-b594-9d947a9e6383\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-wm4qt"
Oct 31 01:03:53.102424 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:53.100828    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d55eb70a-f1a8-4346-b8df-ccd2c598dd6b-host-run-ovn-kubernetes\") pod \"ovnkube-node-pghw2\" (UID: \"d55eb70a-f1a8-4346-b8df-ccd2c598dd6b\") " pod="openshift-ovn-kubernetes/ovnkube-node-pghw2"
Oct 31 01:03:53.102424 ip-10-0-133-197 kubenswrapper[4964]: E1031 01:03:53.100876    4964 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Oct 31 01:03:53.102424 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:53.100888    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e49453a8-c7d9-4248-a8ef-4a819b76806c-service-ca-bundle\") pod \"router-default-5964fff8f9-8k6pp\" (UID: \"e49453a8-c7d9-4248-a8ef-4a819b76806c\") " pod="openshift-ingress/router-default-5964fff8f9-8k6pp"
Oct 31 01:03:53.102424 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:53.100896    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/08490837-b3b6-4c2b-a498-568af0d7086d-etc-sysctl-d\") pod \"tuned-45bvr\" (UID: \"08490837-b3b6-4c2b-a498-568af0d7086d\") " pod="openshift-cluster-node-tuning-operator/tuned-45bvr"
Oct 31 01:03:53.102424 ip-10-0-133-197 kubenswrapper[4964]: E1031 01:03:53.100913    4964 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9dfae6d6-b9a6-4d85-b594-9d947a9e6383-networking-console-plugin-cert podName:9dfae6d6-b9a6-4d85-b594-9d947a9e6383 nodeName:}" failed. No retries permitted until 2026-10-31 01:03:53.600900555 +0000 UTC m=+2.077466963 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/9dfae6d6-b9a6-4d85-b594-9d947a9e6383-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-wm4qt" (UID: "9dfae6d6-b9a6-4d85-b594-9d947a9e6383") : secret "networking-console-plugin-cert" not found
Oct 31 01:03:53.102424 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:53.100929    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ff4dcba2-df8d-4324-9e0c-abb23a5e6982-registry-tls\") pod \"image-registry-56d8464d7c-vpwjc\" (UID: \"ff4dcba2-df8d-4324-9e0c-abb23a5e6982\") " pod="openshift-image-registry/image-registry-56d8464d7c-vpwjc"
Oct 31 01:03:53.102424 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:53.100952    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9cd1266e-c630-4259-b1f7-1b88dbde7385-host-slash\") pod \"iptables-alerter-ssw8w\" (UID: \"9cd1266e-c630-4259-b1f7-1b88dbde7385\") " pod="openshift-network-operator/iptables-alerter-ssw8w"
Oct 31 01:03:53.102424 ip-10-0-133-197 kubenswrapper[4964]: E1031 01:03:53.100965    4964 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e49453a8-c7d9-4248-a8ef-4a819b76806c-service-ca-bundle podName:e49453a8-c7d9-4248-a8ef-4a819b76806c nodeName:}" failed. No retries permitted until 2026-10-31 01:03:53.600955402 +0000 UTC m=+2.077521760 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/e49453a8-c7d9-4248-a8ef-4a819b76806c-service-ca-bundle") pod "router-default-5964fff8f9-8k6pp" (UID: "e49453a8-c7d9-4248-a8ef-4a819b76806c") : configmap references non-existent config key: service-ca.crt
Oct 31 01:03:53.102424 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:53.100981    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/08490837-b3b6-4c2b-a498-568af0d7086d-etc-sysctl-conf\") pod \"tuned-45bvr\" (UID: \"08490837-b3b6-4c2b-a498-568af0d7086d\") " pod="openshift-cluster-node-tuning-operator/tuned-45bvr"
Oct 31 01:03:53.102424 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:53.100980    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9cd1266e-c630-4259-b1f7-1b88dbde7385-host-slash\") pod \"iptables-alerter-ssw8w\" (UID: \"9cd1266e-c630-4259-b1f7-1b88dbde7385\") " pod="openshift-network-operator/iptables-alerter-ssw8w"
Oct 31 01:03:53.102424 ip-10-0-133-197 kubenswrapper[4964]: E1031 01:03:53.101077    4964 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Oct 31 01:03:53.102424 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:53.101005    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/f273b9e5-0419-4587-a9c7-a0c8bec02acf-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-srcwl\" (UID: \"f273b9e5-0419-4587-a9c7-a0c8bec02acf\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-srcwl"
Oct 31 01:03:53.102424 ip-10-0-133-197 kubenswrapper[4964]: E1031 01:03:53.101113    4964 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f273b9e5-0419-4587-a9c7-a0c8bec02acf-samples-operator-tls podName:f273b9e5-0419-4587-a9c7-a0c8bec02acf nodeName:}" failed. No retries permitted until 2026-10-31 01:03:53.601101791 +0000 UTC m=+2.077668164 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/f273b9e5-0419-4587-a9c7-a0c8bec02acf-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-srcwl" (UID: "f273b9e5-0419-4587-a9c7-a0c8bec02acf") : secret "samples-operator-tls" not found
Oct 31 01:03:53.102424 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:53.101131    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/08490837-b3b6-4c2b-a498-568af0d7086d-etc-sysconfig\") pod \"tuned-45bvr\" (UID: \"08490837-b3b6-4c2b-a498-568af0d7086d\") " pod="openshift-cluster-node-tuning-operator/tuned-45bvr"
Oct 31 01:03:53.102424 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:53.101156    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/58b0b75a-1568-4703-ac55-fa87f29225ad-host-run-netns\") pod \"multus-hxh6j\" (UID: \"58b0b75a-1568-4703-ac55-fa87f29225ad\") " pod="openshift-multus/multus-hxh6j"
Oct 31 01:03:53.102424 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:53.101173    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/08490837-b3b6-4c2b-a498-568af0d7086d-etc-sysctl-conf\") pod \"tuned-45bvr\" (UID: \"08490837-b3b6-4c2b-a498-568af0d7086d\") " pod="openshift-cluster-node-tuning-operator/tuned-45bvr"
Oct 31 01:03:53.102424 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:53.101183    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/318b3bc6-6e59-4211-af63-b116526a325d-etc-selinux\") pod \"aws-ebs-csi-driver-node-snxz4\" (UID: \"318b3bc6-6e59-4211-af63-b116526a325d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-snxz4"
Oct 31 01:03:53.102424 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:53.101192    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/08490837-b3b6-4c2b-a498-568af0d7086d-etc-sysconfig\") pod \"tuned-45bvr\" (UID: \"08490837-b3b6-4c2b-a498-568af0d7086d\") " pod="openshift-cluster-node-tuning-operator/tuned-45bvr"
Oct 31 01:03:53.102424 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:53.101208    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/fa085dd8-ca1e-4ec5-acb2-9b9dc878bf8a-tuning-conf-dir\") pod \"multus-additional-cni-plugins-jwthj\" (UID: \"fa085dd8-ca1e-4ec5-acb2-9b9dc878bf8a\") " pod="openshift-multus/multus-additional-cni-plugins-jwthj"
Oct 31 01:03:53.102424 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:53.101212    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/58b0b75a-1568-4703-ac55-fa87f29225ad-host-run-netns\") pod \"multus-hxh6j\" (UID: \"58b0b75a-1568-4703-ac55-fa87f29225ad\") " pod="openshift-multus/multus-hxh6j"
Oct 31 01:03:53.102424 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:53.101247    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/318b3bc6-6e59-4211-af63-b116526a325d-etc-selinux\") pod \"aws-ebs-csi-driver-node-snxz4\" (UID: \"318b3bc6-6e59-4211-af63-b116526a325d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-snxz4"
Oct 31 01:03:53.102424 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:53.101251    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/08490837-b3b6-4c2b-a498-568af0d7086d-etc-kubernetes\") pod \"tuned-45bvr\" (UID: \"08490837-b3b6-4c2b-a498-568af0d7086d\") " pod="openshift-cluster-node-tuning-operator/tuned-45bvr"
Oct 31 01:03:53.102424 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:53.101274    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/08490837-b3b6-4c2b-a498-568af0d7086d-sys\") pod \"tuned-45bvr\" (UID: \"08490837-b3b6-4c2b-a498-568af0d7086d\") " pod="openshift-cluster-node-tuning-operator/tuned-45bvr"
Oct 31 01:03:53.102424 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:53.101287    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/fa085dd8-ca1e-4ec5-acb2-9b9dc878bf8a-tuning-conf-dir\") pod \"multus-additional-cni-plugins-jwthj\" (UID: \"fa085dd8-ca1e-4ec5-acb2-9b9dc878bf8a\") " pod="openshift-multus/multus-additional-cni-plugins-jwthj"
Oct 31 01:03:53.102424 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:53.101339    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/08490837-b3b6-4c2b-a498-568af0d7086d-etc-kubernetes\") pod \"tuned-45bvr\" (UID: \"08490837-b3b6-4c2b-a498-568af0d7086d\") " pod="openshift-cluster-node-tuning-operator/tuned-45bvr"
Oct 31 01:03:53.102424 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:53.101344    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/d55eb70a-f1a8-4346-b8df-ccd2c598dd6b-host-kubelet\") pod \"ovnkube-node-pghw2\" (UID: \"d55eb70a-f1a8-4346-b8df-ccd2c598dd6b\") " pod="openshift-ovn-kubernetes/ovnkube-node-pghw2"
Oct 31 01:03:53.102424 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:53.101363    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/08490837-b3b6-4c2b-a498-568af0d7086d-sys\") pod \"tuned-45bvr\" (UID: \"08490837-b3b6-4c2b-a498-568af0d7086d\") " pod="openshift-cluster-node-tuning-operator/tuned-45bvr"
Oct 31 01:03:53.102424 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:53.101368    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d55eb70a-f1a8-4346-b8df-ccd2c598dd6b-var-lib-openvswitch\") pod \"ovnkube-node-pghw2\" (UID: \"d55eb70a-f1a8-4346-b8df-ccd2c598dd6b\") " pod="openshift-ovn-kubernetes/ovnkube-node-pghw2"
Oct 31 01:03:53.102424 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:53.101388    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/d55eb70a-f1a8-4346-b8df-ccd2c598dd6b-host-kubelet\") pod \"ovnkube-node-pghw2\" (UID: \"d55eb70a-f1a8-4346-b8df-ccd2c598dd6b\") " pod="openshift-ovn-kubernetes/ovnkube-node-pghw2"
Oct 31 01:03:53.102424 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:53.101449    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d55eb70a-f1a8-4346-b8df-ccd2c598dd6b-var-lib-openvswitch\") pod \"ovnkube-node-pghw2\" (UID: \"d55eb70a-f1a8-4346-b8df-ccd2c598dd6b\") " pod="openshift-ovn-kubernetes/ovnkube-node-pghw2"
Oct 31 01:03:53.104016 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:53.103625    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e49453a8-c7d9-4248-a8ef-4a819b76806c-metrics-certs\") pod \"router-default-5964fff8f9-8k6pp\" (UID: \"e49453a8-c7d9-4248-a8ef-4a819b76806c\") " pod="openshift-ingress/router-default-5964fff8f9-8k6pp"
Oct 31 01:03:53.104016 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:53.103665    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/b66619db-fd81-4435-be2a-a5ea6111e204-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-jccjk\" (UID: \"b66619db-fd81-4435-be2a-a5ea6111e204\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-jccjk"
Oct 31 01:03:53.104016 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:53.103680    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ff4dcba2-df8d-4324-9e0c-abb23a5e6982-registry-tls\") pod \"image-registry-56d8464d7c-vpwjc\" (UID: \"ff4dcba2-df8d-4324-9e0c-abb23a5e6982\") " pod="openshift-image-registry/image-registry-56d8464d7c-vpwjc"
Oct 31 01:03:53.104016 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:53.103728    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/aa112079-b749-404c-9081-113220707fcd-registry-tls\") pod \"image-registry-75575b5db8-wtq4d\" (UID: \"aa112079-b749-404c-9081-113220707fcd\") " pod="openshift-image-registry/image-registry-75575b5db8-wtq4d"
Oct 31 01:03:53.132857 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:53.132837    4964 scope.go:122] "RemoveContainer" containerID="46df6877e0e8bb148e755797ac3b08c39be418ccb7326153e8ea50c8505fe54b"
Oct 31 01:03:53.137810 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:53.137760    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-jccjk"
Oct 31 01:03:53.137978 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:53.137942    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-56d8464d7c-vpwjc"
Oct 31 01:03:53.143448 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:53.143431    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-75575b5db8-wtq4d"
Oct 31 01:03:53.148766 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:53.148751    4964 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 31 01:03:53.285688 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:53.285608    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:03:53.482962 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:53.482935    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:03:53.606398 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:53.606365    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/9dfae6d6-b9a6-4d85-b594-9d947a9e6383-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-wm4qt\" (UID: \"9dfae6d6-b9a6-4d85-b594-9d947a9e6383\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-wm4qt"
Oct 31 01:03:53.606541 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:53.606407    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e49453a8-c7d9-4248-a8ef-4a819b76806c-service-ca-bundle\") pod \"router-default-5964fff8f9-8k6pp\" (UID: \"e49453a8-c7d9-4248-a8ef-4a819b76806c\") " pod="openshift-ingress/router-default-5964fff8f9-8k6pp"
Oct 31 01:03:53.606541 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:53.606440    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/f273b9e5-0419-4587-a9c7-a0c8bec02acf-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-srcwl\" (UID: \"f273b9e5-0419-4587-a9c7-a0c8bec02acf\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-srcwl"
Oct 31 01:03:53.606541 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:53.606512    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/85bb69db-c69b-467e-adeb-950bf2947c94-metrics-tls\") pod \"dns-default-t7r6c\" (UID: \"85bb69db-c69b-467e-adeb-950bf2947c94\") " pod="openshift-dns/dns-default-t7r6c"
Oct 31 01:03:53.606541 ip-10-0-133-197 kubenswrapper[4964]: E1031 01:03:53.606520    4964 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Oct 31 01:03:53.606760 ip-10-0-133-197 kubenswrapper[4964]: E1031 01:03:53.606542    4964 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e49453a8-c7d9-4248-a8ef-4a819b76806c-service-ca-bundle podName:e49453a8-c7d9-4248-a8ef-4a819b76806c nodeName:}" failed. No retries permitted until 2026-10-31 01:03:54.606524529 +0000 UTC m=+3.083090889 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/e49453a8-c7d9-4248-a8ef-4a819b76806c-service-ca-bundle") pod "router-default-5964fff8f9-8k6pp" (UID: "e49453a8-c7d9-4248-a8ef-4a819b76806c") : configmap references non-existent config key: service-ca.crt
Oct 31 01:03:53.606760 ip-10-0-133-197 kubenswrapper[4964]: E1031 01:03:53.606575    4964 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9dfae6d6-b9a6-4d85-b594-9d947a9e6383-networking-console-plugin-cert podName:9dfae6d6-b9a6-4d85-b594-9d947a9e6383 nodeName:}" failed. No retries permitted until 2026-10-31 01:03:54.606559967 +0000 UTC m=+3.083126339 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/9dfae6d6-b9a6-4d85-b594-9d947a9e6383-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-wm4qt" (UID: "9dfae6d6-b9a6-4d85-b594-9d947a9e6383") : secret "networking-console-plugin-cert" not found
Oct 31 01:03:53.606760 ip-10-0-133-197 kubenswrapper[4964]: E1031 01:03:53.606588    4964 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 31 01:03:53.606760 ip-10-0-133-197 kubenswrapper[4964]: E1031 01:03:53.606607    4964 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Oct 31 01:03:53.606760 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:53.606626    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8952fc79-c5a6-42ba-a296-8cb7a776f718-metrics-certs\") pod \"network-metrics-daemon-wdsmn\" (UID: \"8952fc79-c5a6-42ba-a296-8cb7a776f718\") " pod="openshift-multus/network-metrics-daemon-wdsmn"
Oct 31 01:03:53.606760 ip-10-0-133-197 kubenswrapper[4964]: E1031 01:03:53.606633    4964 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/85bb69db-c69b-467e-adeb-950bf2947c94-metrics-tls podName:85bb69db-c69b-467e-adeb-950bf2947c94 nodeName:}" failed. No retries permitted until 2026-10-31 01:03:54.606620857 +0000 UTC m=+3.083187220 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/85bb69db-c69b-467e-adeb-950bf2947c94-metrics-tls") pod "dns-default-t7r6c" (UID: "85bb69db-c69b-467e-adeb-950bf2947c94") : secret "dns-default-metrics-tls" not found
Oct 31 01:03:53.606760 ip-10-0-133-197 kubenswrapper[4964]: E1031 01:03:53.606656    4964 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f273b9e5-0419-4587-a9c7-a0c8bec02acf-samples-operator-tls podName:f273b9e5-0419-4587-a9c7-a0c8bec02acf nodeName:}" failed. No retries permitted until 2026-10-31 01:03:54.606644638 +0000 UTC m=+3.083211001 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/f273b9e5-0419-4587-a9c7-a0c8bec02acf-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-srcwl" (UID: "f273b9e5-0419-4587-a9c7-a0c8bec02acf") : secret "samples-operator-tls" not found
Oct 31 01:03:53.606760 ip-10-0-133-197 kubenswrapper[4964]: E1031 01:03:53.606699    4964 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Oct 31 01:03:53.606760 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:53.606702    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7dd98c21-e0b1-4289-8fe8-036bf413d444-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-q56x5\" (UID: \"7dd98c21-e0b1-4289-8fe8-036bf413d444\") " pod="openshift-insights/insights-runtime-extractor-q56x5"
Oct 31 01:03:53.606760 ip-10-0-133-197 kubenswrapper[4964]: E1031 01:03:53.606728    4964 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8952fc79-c5a6-42ba-a296-8cb7a776f718-metrics-certs podName:8952fc79-c5a6-42ba-a296-8cb7a776f718 nodeName:}" failed. No retries permitted until 2026-10-31 01:03:54.606719206 +0000 UTC m=+3.083285578 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8952fc79-c5a6-42ba-a296-8cb7a776f718-metrics-certs") pod "network-metrics-daemon-wdsmn" (UID: "8952fc79-c5a6-42ba-a296-8cb7a776f718") : secret "metrics-daemon-secret" not found
Oct 31 01:03:53.606760 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:53.606747    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7dd98c21-e0b1-4289-8fe8-036bf413d444-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-q56x5\" (UID: \"7dd98c21-e0b1-4289-8fe8-036bf413d444\") " pod="openshift-insights/insights-runtime-extractor-q56x5"
Oct 31 01:03:53.607195 ip-10-0-133-197 kubenswrapper[4964]: E1031 01:03:53.606782    4964 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 31 01:03:53.607195 ip-10-0-133-197 kubenswrapper[4964]: E1031 01:03:53.606815    4964 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Oct 31 01:03:53.607195 ip-10-0-133-197 kubenswrapper[4964]: E1031 01:03:53.606818    4964 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7dd98c21-e0b1-4289-8fe8-036bf413d444-insights-runtime-extractor-tls podName:7dd98c21-e0b1-4289-8fe8-036bf413d444 nodeName:}" failed. No retries permitted until 2026-10-31 01:03:54.606807788 +0000 UTC m=+3.083374170 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/7dd98c21-e0b1-4289-8fe8-036bf413d444-insights-runtime-extractor-tls") pod "insights-runtime-extractor-q56x5" (UID: "7dd98c21-e0b1-4289-8fe8-036bf413d444") : secret "insights-runtime-extractor-tls" not found
Oct 31 01:03:53.607195 ip-10-0-133-197 kubenswrapper[4964]: E1031 01:03:53.606852    4964 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/7dd98c21-e0b1-4289-8fe8-036bf413d444-kube-rbac-proxy-cm podName:7dd98c21-e0b1-4289-8fe8-036bf413d444 nodeName:}" failed. No retries permitted until 2026-10-31 01:03:54.606843347 +0000 UTC m=+3.083409709 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/7dd98c21-e0b1-4289-8fe8-036bf413d444-kube-rbac-proxy-cm") pod "insights-runtime-extractor-q56x5" (UID: "7dd98c21-e0b1-4289-8fe8-036bf413d444") : configmap "kube-rbac-proxy" not found
Oct 31 01:03:53.607378 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:53.607362    4964 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-rlxt2_dd8134f9-b2a6-4542-82ec-c10edc3fcd72/node-ca/0.log"
Oct 31 01:03:53.633068 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:53.633043    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-56d8464d7c-vpwjc"]
Oct 31 01:03:53.637496 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:53.637467    4964 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff4dcba2_df8d_4324_9e0c_abb23a5e6982.slice/crio-9cd59c49b732bc348537c81f271393f2b7370197baf9479034c23d8b9260d572 WatchSource:0}: Error finding container 9cd59c49b732bc348537c81f271393f2b7370197baf9479034c23d8b9260d572: Status 404 returned error can't find the container with id 9cd59c49b732bc348537c81f271393f2b7370197baf9479034c23d8b9260d572
Oct 31 01:03:53.659845 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:53.659823    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-jccjk"]
Oct 31 01:03:53.663092 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:53.663077    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-75575b5db8-wtq4d"]
Oct 31 01:03:53.672282 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:53.672247    4964 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb66619db_fd81_4435_be2a_a5ea6111e204.slice/crio-ea2980470a3707fc71449b395e45b1a9af78a93de660882c6d36d3e93669da6c WatchSource:0}: Error finding container ea2980470a3707fc71449b395e45b1a9af78a93de660882c6d36d3e93669da6c: Status 404 returned error can't find the container with id ea2980470a3707fc71449b395e45b1a9af78a93de660882c6d36d3e93669da6c
Oct 31 01:03:53.685210 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:53.685178    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:03:53.690513 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:03:53.690493    4964 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa112079_b749_404c_9081_113220707fcd.slice/crio-6061e137a9ba52d29809eef437d64849764736661809b0de896d4eff81760b69 WatchSource:0}: Error finding container 6061e137a9ba52d29809eef437d64849764736661809b0de896d4eff81760b69: Status 404 returned error can't find the container with id 6061e137a9ba52d29809eef437d64849764736661809b0de896d4eff81760b69
Oct 31 01:03:53.849544 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:53.849511    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-56d8464d7c-vpwjc" event={"ID":"ff4dcba2-df8d-4324-9e0c-abb23a5e6982","Type":"ContainerStarted","Data":"f52fc06311bb5c0fd0ea014c6b80a1c1e84c54d8e0c75f46a4db73d0b18c3fd7"}
Oct 31 01:03:53.849681 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:53.849595    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-56d8464d7c-vpwjc" event={"ID":"ff4dcba2-df8d-4324-9e0c-abb23a5e6982","Type":"ContainerStarted","Data":"9cd59c49b732bc348537c81f271393f2b7370197baf9479034c23d8b9260d572"}
Oct 31 01:03:53.851248 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:53.851215    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-75575b5db8-wtq4d" event={"ID":"aa112079-b749-404c-9081-113220707fcd","Type":"ContainerStarted","Data":"633d26971872c25b8ea849d33f1ef31d6c1ae5fb02e89c28a0c19568874f0f1c"}
Oct 31 01:03:53.851248 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:53.851250    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-75575b5db8-wtq4d" event={"ID":"aa112079-b749-404c-9081-113220707fcd","Type":"ContainerStarted","Data":"6061e137a9ba52d29809eef437d64849764736661809b0de896d4eff81760b69"}
Oct 31 01:03:53.852635 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:53.852617    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-jccjk" event={"ID":"b66619db-fd81-4435-be2a-a5ea6111e204","Type":"ContainerStarted","Data":"ea2980470a3707fc71449b395e45b1a9af78a93de660882c6d36d3e93669da6c"}
Oct 31 01:03:53.854141 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:53.854125    4964 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-9257n_f0c0ada7-7688-4dc2-97fc-813e4d0c4ef3/console-operator/2.log"
Oct 31 01:03:53.854575 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:53.854557    4964 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-9257n_f0c0ada7-7688-4dc2-97fc-813e4d0c4ef3/console-operator/1.log"
Oct 31 01:03:53.854636 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:53.854599    4964 generic.go:356] "Generic (PLEG): container finished" podID="f0c0ada7-7688-4dc2-97fc-813e4d0c4ef3" containerID="b70ef6e4580b026c0566fcbee7f62b195af7e015b7934645c9d9e3a310d27128" exitCode=255
Oct 31 01:03:53.854694 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:53.854675    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-9257n" event={"ID":"f0c0ada7-7688-4dc2-97fc-813e4d0c4ef3","Type":"ContainerDied","Data":"b70ef6e4580b026c0566fcbee7f62b195af7e015b7934645c9d9e3a310d27128"}
Oct 31 01:03:53.854756 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:53.854744    4964 scope.go:122] "RemoveContainer" containerID="46df6877e0e8bb148e755797ac3b08c39be418ccb7326153e8ea50c8505fe54b"
Oct 31 01:03:53.855187 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:53.855152    4964 scope.go:122] "RemoveContainer" containerID="b70ef6e4580b026c0566fcbee7f62b195af7e015b7934645c9d9e3a310d27128"
Oct 31 01:03:53.855357 ip-10-0-133-197 kubenswrapper[4964]: E1031 01:03:53.855336    4964 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-9257n_openshift-console-operator(f0c0ada7-7688-4dc2-97fc-813e4d0c4ef3)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-9257n" podUID="f0c0ada7-7688-4dc2-97fc-813e4d0c4ef3"
Oct 31 01:03:54.051968 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:54.051899    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:03:54.206623 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:54.206590    4964 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-7dpwv_147bb9af-dfc7-4785-bb5e-906f25d65dd1/serve-healthcheck-canary/0.log"
Oct 31 01:03:54.442755 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:54.442688    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-9257n"
Oct 31 01:03:54.614710 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:54.614685    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/9dfae6d6-b9a6-4d85-b594-9d947a9e6383-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-wm4qt\" (UID: \"9dfae6d6-b9a6-4d85-b594-9d947a9e6383\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-wm4qt"
Oct 31 01:03:54.614710 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:54.614714    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e49453a8-c7d9-4248-a8ef-4a819b76806c-service-ca-bundle\") pod \"router-default-5964fff8f9-8k6pp\" (UID: \"e49453a8-c7d9-4248-a8ef-4a819b76806c\") " pod="openshift-ingress/router-default-5964fff8f9-8k6pp"
Oct 31 01:03:54.614890 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:54.614733    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/f273b9e5-0419-4587-a9c7-a0c8bec02acf-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-srcwl\" (UID: \"f273b9e5-0419-4587-a9c7-a0c8bec02acf\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-srcwl"
Oct 31 01:03:54.614890 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:54.614773    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/85bb69db-c69b-467e-adeb-950bf2947c94-metrics-tls\") pod \"dns-default-t7r6c\" (UID: \"85bb69db-c69b-467e-adeb-950bf2947c94\") " pod="openshift-dns/dns-default-t7r6c"
Oct 31 01:03:54.614890 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:54.614797    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8952fc79-c5a6-42ba-a296-8cb7a776f718-metrics-certs\") pod \"network-metrics-daemon-wdsmn\" (UID: \"8952fc79-c5a6-42ba-a296-8cb7a776f718\") " pod="openshift-multus/network-metrics-daemon-wdsmn"
Oct 31 01:03:54.614890 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:54.614815    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7dd98c21-e0b1-4289-8fe8-036bf413d444-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-q56x5\" (UID: \"7dd98c21-e0b1-4289-8fe8-036bf413d444\") " pod="openshift-insights/insights-runtime-extractor-q56x5"
Oct 31 01:03:54.614890 ip-10-0-133-197 kubenswrapper[4964]: E1031 01:03:54.614834    4964 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Oct 31 01:03:54.614890 ip-10-0-133-197 kubenswrapper[4964]: E1031 01:03:54.614857    4964 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e49453a8-c7d9-4248-a8ef-4a819b76806c-service-ca-bundle podName:e49453a8-c7d9-4248-a8ef-4a819b76806c nodeName:}" failed. No retries permitted until 2026-10-31 01:03:56.61484045 +0000 UTC m=+5.091406828 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/e49453a8-c7d9-4248-a8ef-4a819b76806c-service-ca-bundle") pod "router-default-5964fff8f9-8k6pp" (UID: "e49453a8-c7d9-4248-a8ef-4a819b76806c") : configmap references non-existent config key: service-ca.crt
Oct 31 01:03:54.614890 ip-10-0-133-197 kubenswrapper[4964]: E1031 01:03:54.614879    4964 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9dfae6d6-b9a6-4d85-b594-9d947a9e6383-networking-console-plugin-cert podName:9dfae6d6-b9a6-4d85-b594-9d947a9e6383 nodeName:}" failed. No retries permitted until 2026-10-31 01:03:56.614866624 +0000 UTC m=+5.091432982 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/9dfae6d6-b9a6-4d85-b594-9d947a9e6383-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-wm4qt" (UID: "9dfae6d6-b9a6-4d85-b594-9d947a9e6383") : secret "networking-console-plugin-cert" not found
Oct 31 01:03:54.614890 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:54.614835    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7dd98c21-e0b1-4289-8fe8-036bf413d444-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-q56x5\" (UID: \"7dd98c21-e0b1-4289-8fe8-036bf413d444\") " pod="openshift-insights/insights-runtime-extractor-q56x5"
Oct 31 01:03:54.615177 ip-10-0-133-197 kubenswrapper[4964]: E1031 01:03:54.614913    4964 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 31 01:03:54.615177 ip-10-0-133-197 kubenswrapper[4964]: E1031 01:03:54.614920    4964 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Oct 31 01:03:54.615177 ip-10-0-133-197 kubenswrapper[4964]: E1031 01:03:54.614939    4964 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Oct 31 01:03:54.615177 ip-10-0-133-197 kubenswrapper[4964]: E1031 01:03:54.614913    4964 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Oct 31 01:03:54.615177 ip-10-0-133-197 kubenswrapper[4964]: E1031 01:03:54.614960    4964 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7dd98c21-e0b1-4289-8fe8-036bf413d444-insights-runtime-extractor-tls podName:7dd98c21-e0b1-4289-8fe8-036bf413d444 nodeName:}" failed. No retries permitted until 2026-10-31 01:03:56.61494616 +0000 UTC m=+5.091512537 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/7dd98c21-e0b1-4289-8fe8-036bf413d444-insights-runtime-extractor-tls") pod "insights-runtime-extractor-q56x5" (UID: "7dd98c21-e0b1-4289-8fe8-036bf413d444") : secret "insights-runtime-extractor-tls" not found
Oct 31 01:03:54.615177 ip-10-0-133-197 kubenswrapper[4964]: E1031 01:03:54.614913    4964 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 31 01:03:54.615177 ip-10-0-133-197 kubenswrapper[4964]: E1031 01:03:54.614979    4964 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/7dd98c21-e0b1-4289-8fe8-036bf413d444-kube-rbac-proxy-cm podName:7dd98c21-e0b1-4289-8fe8-036bf413d444 nodeName:}" failed. No retries permitted until 2026-10-31 01:03:56.61496685 +0000 UTC m=+5.091533224 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/7dd98c21-e0b1-4289-8fe8-036bf413d444-kube-rbac-proxy-cm") pod "insights-runtime-extractor-q56x5" (UID: "7dd98c21-e0b1-4289-8fe8-036bf413d444") : configmap "kube-rbac-proxy" not found
Oct 31 01:03:54.615177 ip-10-0-133-197 kubenswrapper[4964]: E1031 01:03:54.614994    4964 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f273b9e5-0419-4587-a9c7-a0c8bec02acf-samples-operator-tls podName:f273b9e5-0419-4587-a9c7-a0c8bec02acf nodeName:}" failed. No retries permitted until 2026-10-31 01:03:56.614987789 +0000 UTC m=+5.091554159 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/f273b9e5-0419-4587-a9c7-a0c8bec02acf-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-srcwl" (UID: "f273b9e5-0419-4587-a9c7-a0c8bec02acf") : secret "samples-operator-tls" not found
Oct 31 01:03:54.615177 ip-10-0-133-197 kubenswrapper[4964]: E1031 01:03:54.615003    4964 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8952fc79-c5a6-42ba-a296-8cb7a776f718-metrics-certs podName:8952fc79-c5a6-42ba-a296-8cb7a776f718 nodeName:}" failed. No retries permitted until 2026-10-31 01:03:56.614998363 +0000 UTC m=+5.091564721 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8952fc79-c5a6-42ba-a296-8cb7a776f718-metrics-certs") pod "network-metrics-daemon-wdsmn" (UID: "8952fc79-c5a6-42ba-a296-8cb7a776f718") : secret "metrics-daemon-secret" not found
Oct 31 01:03:54.615177 ip-10-0-133-197 kubenswrapper[4964]: E1031 01:03:54.615012    4964 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/85bb69db-c69b-467e-adeb-950bf2947c94-metrics-tls podName:85bb69db-c69b-467e-adeb-950bf2947c94 nodeName:}" failed. No retries permitted until 2026-10-31 01:03:56.615007224 +0000 UTC m=+5.091573581 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/85bb69db-c69b-467e-adeb-950bf2947c94-metrics-tls") pod "dns-default-t7r6c" (UID: "85bb69db-c69b-467e-adeb-950bf2947c94") : secret "dns-default-metrics-tls" not found
Oct 31 01:03:54.637560 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:54.637527    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:03:54.808143 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:54.808110    4964 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-7d4bc45dd6-n5q9v_c2be658b-ec92-4782-a78b-d2dccf2249cb/kube-storage-version-migrator-operator/0.log"
Oct 31 01:03:54.831292 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:54.831268    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-pghw2"
Oct 31 01:03:54.859475 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:54.859446    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-s2r9l" event={"ID":"b5975856-8eb3-4601-a7e1-d1b1b3a957e8","Type":"ContainerStarted","Data":"ad8d638771cd15409b7ce57c107d0adbb027893a0887748c230d4aa42e9d5ac1"}
Oct 31 01:03:54.860973 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:54.860949    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-cqpmt" event={"ID":"cb657366-6696-4c3e-a3a3-603ce00cf11a","Type":"ContainerStarted","Data":"8e444f30a1e0f09ad75e6207680880ee6c827bd89e821a575531e0cb15d91516"}
Oct 31 01:03:54.862458 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:54.862442    4964 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-9257n_f0c0ada7-7688-4dc2-97fc-813e4d0c4ef3/console-operator/2.log"
Oct 31 01:03:54.862532 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:54.862514    4964 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 31 01:03:54.862596 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:54.862551    4964 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 31 01:03:54.862727 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:54.862712    4964 scope.go:122] "RemoveContainer" containerID="b70ef6e4580b026c0566fcbee7f62b195af7e015b7934645c9d9e3a310d27128"
Oct 31 01:03:54.862926 ip-10-0-133-197 kubenswrapper[4964]: E1031 01:03:54.862905    4964 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-9257n_openshift-console-operator(f0c0ada7-7688-4dc2-97fc-813e4d0c4ef3)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-9257n" podUID="f0c0ada7-7688-4dc2-97fc-813e4d0c4ef3"
Oct 31 01:03:55.243450 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:55.243416    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:03:55.446787 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:55.446754    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:03:55.846783 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:55.846749    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:03:56.042998 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:56.042971    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:03:56.231845 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:56.231807    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:03:56.562997 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:56.562928    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-78szg"
Oct 31 01:03:56.633175 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:56.633145    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/85bb69db-c69b-467e-adeb-950bf2947c94-metrics-tls\") pod \"dns-default-t7r6c\" (UID: \"85bb69db-c69b-467e-adeb-950bf2947c94\") " pod="openshift-dns/dns-default-t7r6c"
Oct 31 01:03:56.633375 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:56.633185    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8952fc79-c5a6-42ba-a296-8cb7a776f718-metrics-certs\") pod \"network-metrics-daemon-wdsmn\" (UID: \"8952fc79-c5a6-42ba-a296-8cb7a776f718\") " pod="openshift-multus/network-metrics-daemon-wdsmn"
Oct 31 01:03:56.633375 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:56.633209    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7dd98c21-e0b1-4289-8fe8-036bf413d444-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-q56x5\" (UID: \"7dd98c21-e0b1-4289-8fe8-036bf413d444\") " pod="openshift-insights/insights-runtime-extractor-q56x5"
Oct 31 01:03:56.633375 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:56.633229    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7dd98c21-e0b1-4289-8fe8-036bf413d444-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-q56x5\" (UID: \"7dd98c21-e0b1-4289-8fe8-036bf413d444\") " pod="openshift-insights/insights-runtime-extractor-q56x5"
Oct 31 01:03:56.633375 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:56.633260    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/9dfae6d6-b9a6-4d85-b594-9d947a9e6383-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-wm4qt\" (UID: \"9dfae6d6-b9a6-4d85-b594-9d947a9e6383\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-wm4qt"
Oct 31 01:03:56.633375 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:56.633275    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e49453a8-c7d9-4248-a8ef-4a819b76806c-service-ca-bundle\") pod \"router-default-5964fff8f9-8k6pp\" (UID: \"e49453a8-c7d9-4248-a8ef-4a819b76806c\") " pod="openshift-ingress/router-default-5964fff8f9-8k6pp"
Oct 31 01:03:56.633375 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:56.633301    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/f273b9e5-0419-4587-a9c7-a0c8bec02acf-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-srcwl\" (UID: \"f273b9e5-0419-4587-a9c7-a0c8bec02acf\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-srcwl"
Oct 31 01:03:56.633375 ip-10-0-133-197 kubenswrapper[4964]: E1031 01:03:56.633310    4964 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 31 01:03:56.633375 ip-10-0-133-197 kubenswrapper[4964]: E1031 01:03:56.633310    4964 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 31 01:03:56.633375 ip-10-0-133-197 kubenswrapper[4964]: E1031 01:03:56.633375    4964 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Oct 31 01:03:56.633674 ip-10-0-133-197 kubenswrapper[4964]: E1031 01:03:56.633309    4964 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Oct 31 01:03:56.633674 ip-10-0-133-197 kubenswrapper[4964]: E1031 01:03:56.633394    4964 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/85bb69db-c69b-467e-adeb-950bf2947c94-metrics-tls podName:85bb69db-c69b-467e-adeb-950bf2947c94 nodeName:}" failed. No retries permitted until 2026-10-31 01:04:00.633375684 +0000 UTC m=+9.109942041 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/85bb69db-c69b-467e-adeb-950bf2947c94-metrics-tls") pod "dns-default-t7r6c" (UID: "85bb69db-c69b-467e-adeb-950bf2947c94") : secret "dns-default-metrics-tls" not found
Oct 31 01:03:56.633674 ip-10-0-133-197 kubenswrapper[4964]: E1031 01:03:56.633394    4964 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Oct 31 01:03:56.633674 ip-10-0-133-197 kubenswrapper[4964]: E1031 01:03:56.633409    4964 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Oct 31 01:03:56.633674 ip-10-0-133-197 kubenswrapper[4964]: E1031 01:03:56.633412    4964 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7dd98c21-e0b1-4289-8fe8-036bf413d444-insights-runtime-extractor-tls podName:7dd98c21-e0b1-4289-8fe8-036bf413d444 nodeName:}" failed. No retries permitted until 2026-10-31 01:04:00.633403893 +0000 UTC m=+9.109970250 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/7dd98c21-e0b1-4289-8fe8-036bf413d444-insights-runtime-extractor-tls") pod "insights-runtime-extractor-q56x5" (UID: "7dd98c21-e0b1-4289-8fe8-036bf413d444") : secret "insights-runtime-extractor-tls" not found
Oct 31 01:03:56.633674 ip-10-0-133-197 kubenswrapper[4964]: E1031 01:03:56.633438    4964 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9dfae6d6-b9a6-4d85-b594-9d947a9e6383-networking-console-plugin-cert podName:9dfae6d6-b9a6-4d85-b594-9d947a9e6383 nodeName:}" failed. No retries permitted until 2026-10-31 01:04:00.633423166 +0000 UTC m=+9.109989542 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/9dfae6d6-b9a6-4d85-b594-9d947a9e6383-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-wm4qt" (UID: "9dfae6d6-b9a6-4d85-b594-9d947a9e6383") : secret "networking-console-plugin-cert" not found
Oct 31 01:03:56.633674 ip-10-0-133-197 kubenswrapper[4964]: E1031 01:03:56.633456    4964 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f273b9e5-0419-4587-a9c7-a0c8bec02acf-samples-operator-tls podName:f273b9e5-0419-4587-a9c7-a0c8bec02acf nodeName:}" failed. No retries permitted until 2026-10-31 01:04:00.633446348 +0000 UTC m=+9.110012706 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/f273b9e5-0419-4587-a9c7-a0c8bec02acf-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-srcwl" (UID: "f273b9e5-0419-4587-a9c7-a0c8bec02acf") : secret "samples-operator-tls" not found
Oct 31 01:03:56.633674 ip-10-0-133-197 kubenswrapper[4964]: E1031 01:03:56.633468    4964 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8952fc79-c5a6-42ba-a296-8cb7a776f718-metrics-certs podName:8952fc79-c5a6-42ba-a296-8cb7a776f718 nodeName:}" failed. No retries permitted until 2026-10-31 01:04:00.633462648 +0000 UTC m=+9.110029005 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8952fc79-c5a6-42ba-a296-8cb7a776f718-metrics-certs") pod "network-metrics-daemon-wdsmn" (UID: "8952fc79-c5a6-42ba-a296-8cb7a776f718") : secret "metrics-daemon-secret" not found
Oct 31 01:03:56.633674 ip-10-0-133-197 kubenswrapper[4964]: E1031 01:03:56.633478    4964 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/7dd98c21-e0b1-4289-8fe8-036bf413d444-kube-rbac-proxy-cm podName:7dd98c21-e0b1-4289-8fe8-036bf413d444 nodeName:}" failed. No retries permitted until 2026-10-31 01:04:00.633472869 +0000 UTC m=+9.110039227 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/7dd98c21-e0b1-4289-8fe8-036bf413d444-kube-rbac-proxy-cm") pod "insights-runtime-extractor-q56x5" (UID: "7dd98c21-e0b1-4289-8fe8-036bf413d444") : configmap "kube-rbac-proxy" not found
Oct 31 01:03:56.633674 ip-10-0-133-197 kubenswrapper[4964]: E1031 01:03:56.633487    4964 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e49453a8-c7d9-4248-a8ef-4a819b76806c-service-ca-bundle podName:e49453a8-c7d9-4248-a8ef-4a819b76806c nodeName:}" failed. No retries permitted until 2026-10-31 01:04:00.633482455 +0000 UTC m=+9.110048813 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/e49453a8-c7d9-4248-a8ef-4a819b76806c-service-ca-bundle") pod "router-default-5964fff8f9-8k6pp" (UID: "e49453a8-c7d9-4248-a8ef-4a819b76806c") : configmap references non-existent config key: service-ca.crt
Oct 31 01:03:56.652588 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:56.652566    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:03:57.052378 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:57.052349    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:03:57.279424 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:57.279391    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:03:57.476540 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:57.476507    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:03:57.675921 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:57.675891    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:03:57.875056 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:57.874985    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-jccjk" event={"ID":"b66619db-fd81-4435-be2a-a5ea6111e204","Type":"ContainerStarted","Data":"a5deba601223950bd9ac27e42911995003e60b81a1d74b79550cf078c20d8d09"}
Oct 31 01:03:57.912487 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:57.912454    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:03:58.134118 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:58.134051    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:03:58.331553 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:58.331523    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:03:58.556553 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:58.556521    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:03:59.021064 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:59.021031    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-75575b5db8-wtq4d"
Oct 31 01:03:59.242239 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:59.242213    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:03:59.424263 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:59.424201    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:03:59.637128 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:59.637101    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:03:59.843643 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:03:59.843615    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:04:00.055678 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:04:00.055646    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:04:00.192588 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:04:00.192533    4964 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-7cd56b8dbc-9257n"
Oct 31 01:04:00.192874 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:04:00.192862    4964 scope.go:122] "RemoveContainer" containerID="b70ef6e4580b026c0566fcbee7f62b195af7e015b7934645c9d9e3a310d27128"
Oct 31 01:04:00.193035 ip-10-0-133-197 kubenswrapper[4964]: E1031 01:04:00.193019    4964 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-9257n_openshift-console-operator(f0c0ada7-7688-4dc2-97fc-813e4d0c4ef3)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-9257n" podUID="f0c0ada7-7688-4dc2-97fc-813e4d0c4ef3"
Oct 31 01:04:00.235935 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:04:00.235910    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-56d8464d7c-vpwjc"
Oct 31 01:04:00.435096 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:04:00.435069    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:04:00.672556 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:04:00.672527    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/85bb69db-c69b-467e-adeb-950bf2947c94-metrics-tls\") pod \"dns-default-t7r6c\" (UID: \"85bb69db-c69b-467e-adeb-950bf2947c94\") " pod="openshift-dns/dns-default-t7r6c"
Oct 31 01:04:00.672694 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:04:00.672572    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8952fc79-c5a6-42ba-a296-8cb7a776f718-metrics-certs\") pod \"network-metrics-daemon-wdsmn\" (UID: \"8952fc79-c5a6-42ba-a296-8cb7a776f718\") " pod="openshift-multus/network-metrics-daemon-wdsmn"
Oct 31 01:04:00.672694 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:04:00.672630    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7dd98c21-e0b1-4289-8fe8-036bf413d444-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-q56x5\" (UID: \"7dd98c21-e0b1-4289-8fe8-036bf413d444\") " pod="openshift-insights/insights-runtime-extractor-q56x5"
Oct 31 01:04:00.672694 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:04:00.672666    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7dd98c21-e0b1-4289-8fe8-036bf413d444-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-q56x5\" (UID: \"7dd98c21-e0b1-4289-8fe8-036bf413d444\") " pod="openshift-insights/insights-runtime-extractor-q56x5"
Oct 31 01:04:00.672857 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:04:00.672721    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/9dfae6d6-b9a6-4d85-b594-9d947a9e6383-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-wm4qt\" (UID: \"9dfae6d6-b9a6-4d85-b594-9d947a9e6383\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-wm4qt"
Oct 31 01:04:00.672857 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:04:00.672752    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e49453a8-c7d9-4248-a8ef-4a819b76806c-service-ca-bundle\") pod \"router-default-5964fff8f9-8k6pp\" (UID: \"e49453a8-c7d9-4248-a8ef-4a819b76806c\") " pod="openshift-ingress/router-default-5964fff8f9-8k6pp"
Oct 31 01:04:00.672857 ip-10-0-133-197 kubenswrapper[4964]: E1031 01:04:00.672761    4964 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 31 01:04:00.672857 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:04:00.672779    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/f273b9e5-0419-4587-a9c7-a0c8bec02acf-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-srcwl\" (UID: \"f273b9e5-0419-4587-a9c7-a0c8bec02acf\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-srcwl"
Oct 31 01:04:00.672857 ip-10-0-133-197 kubenswrapper[4964]: E1031 01:04:00.672820    4964 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7dd98c21-e0b1-4289-8fe8-036bf413d444-insights-runtime-extractor-tls podName:7dd98c21-e0b1-4289-8fe8-036bf413d444 nodeName:}" failed. No retries permitted until 2026-10-31 01:04:08.67280202 +0000 UTC m=+17.149368384 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/7dd98c21-e0b1-4289-8fe8-036bf413d444-insights-runtime-extractor-tls") pod "insights-runtime-extractor-q56x5" (UID: "7dd98c21-e0b1-4289-8fe8-036bf413d444") : secret "insights-runtime-extractor-tls" not found
Oct 31 01:04:00.672857 ip-10-0-133-197 kubenswrapper[4964]: E1031 01:04:00.672840    4964 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Oct 31 01:04:00.673129 ip-10-0-133-197 kubenswrapper[4964]: E1031 01:04:00.672859    4964 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Oct 31 01:04:00.673129 ip-10-0-133-197 kubenswrapper[4964]: E1031 01:04:00.672917    4964 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9dfae6d6-b9a6-4d85-b594-9d947a9e6383-networking-console-plugin-cert podName:9dfae6d6-b9a6-4d85-b594-9d947a9e6383 nodeName:}" failed. No retries permitted until 2026-10-31 01:04:08.672901237 +0000 UTC m=+17.149467597 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/9dfae6d6-b9a6-4d85-b594-9d947a9e6383-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-wm4qt" (UID: "9dfae6d6-b9a6-4d85-b594-9d947a9e6383") : secret "networking-console-plugin-cert" not found
Oct 31 01:04:00.673129 ip-10-0-133-197 kubenswrapper[4964]: E1031 01:04:00.672936    4964 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/7dd98c21-e0b1-4289-8fe8-036bf413d444-kube-rbac-proxy-cm podName:7dd98c21-e0b1-4289-8fe8-036bf413d444 nodeName:}" failed. No retries permitted until 2026-10-31 01:04:08.672926186 +0000 UTC m=+17.149492548 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/7dd98c21-e0b1-4289-8fe8-036bf413d444-kube-rbac-proxy-cm") pod "insights-runtime-extractor-q56x5" (UID: "7dd98c21-e0b1-4289-8fe8-036bf413d444") : configmap "kube-rbac-proxy" not found
Oct 31 01:04:00.673129 ip-10-0-133-197 kubenswrapper[4964]: E1031 01:04:00.672950    4964 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e49453a8-c7d9-4248-a8ef-4a819b76806c-service-ca-bundle podName:e49453a8-c7d9-4248-a8ef-4a819b76806c nodeName:}" failed. No retries permitted until 2026-10-31 01:04:08.672941995 +0000 UTC m=+17.149508355 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/e49453a8-c7d9-4248-a8ef-4a819b76806c-service-ca-bundle") pod "router-default-5964fff8f9-8k6pp" (UID: "e49453a8-c7d9-4248-a8ef-4a819b76806c") : configmap references non-existent config key: service-ca.crt
Oct 31 01:04:00.675041 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:04:00.675016    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/85bb69db-c69b-467e-adeb-950bf2947c94-metrics-tls\") pod \"dns-default-t7r6c\" (UID: \"85bb69db-c69b-467e-adeb-950bf2947c94\") " pod="openshift-dns/dns-default-t7r6c"
Oct 31 01:04:00.675158 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:04:00.675142    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/f273b9e5-0419-4587-a9c7-a0c8bec02acf-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-srcwl\" (UID: \"f273b9e5-0419-4587-a9c7-a0c8bec02acf\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-srcwl"
Oct 31 01:04:00.675217 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:04:00.675173    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8952fc79-c5a6-42ba-a296-8cb7a776f718-metrics-certs\") pod \"network-metrics-daemon-wdsmn\" (UID: \"8952fc79-c5a6-42ba-a296-8cb7a776f718\") " pod="openshift-multus/network-metrics-daemon-wdsmn"
Oct 31 01:04:00.931313 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:04:00.931260    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-srcwl"
Oct 31 01:04:00.939827 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:04:00.939809    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wdsmn"
Oct 31 01:04:00.944843 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:04:00.944822    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-t7r6c"
Oct 31 01:04:01.249447 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:04:01.249416    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:04:01.318060 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:04:01.318034    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-srcwl"]
Oct 31 01:04:01.355010 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:04:01.354988    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-wdsmn"]
Oct 31 01:04:01.364240 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:04:01.364218    4964 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8952fc79_c5a6_42ba_a296_8cb7a776f718.slice/crio-f5b808dac049659e1093df65822bb8425a9885c5e1efbdb78ec9be81dc70b51a WatchSource:0}: Error finding container f5b808dac049659e1093df65822bb8425a9885c5e1efbdb78ec9be81dc70b51a: Status 404 returned error can't find the container with id f5b808dac049659e1093df65822bb8425a9885c5e1efbdb78ec9be81dc70b51a
Oct 31 01:04:01.368528 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:04:01.368511    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-t7r6c"]
Oct 31 01:04:01.371945 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:04:01.371919    4964 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod85bb69db_c69b_467e_adeb_950bf2947c94.slice/crio-926e3b2a644d5d47133af4901e7a3d827dd6e62f26984ea0b63b0266c1ef1153 WatchSource:0}: Error finding container 926e3b2a644d5d47133af4901e7a3d827dd6e62f26984ea0b63b0266c1ef1153: Status 404 returned error can't find the container with id 926e3b2a644d5d47133af4901e7a3d827dd6e62f26984ea0b63b0266c1ef1153
Oct 31 01:04:01.493985 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:04:01.493953    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:04:01.540879 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:04:01.540823    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-8mqbd"
Oct 31 01:04:01.648913 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:04:01.648884    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-pghw2"
Oct 31 01:04:01.649127 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:04:01.649113    4964 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 31 01:04:01.681214 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:04:01.681191    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:04:01.867399 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:04:01.867310    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:04:01.888870 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:04:01.888837    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-srcwl" event={"ID":"f273b9e5-0419-4587-a9c7-a0c8bec02acf","Type":"ContainerStarted","Data":"dcb909f4db25f3d8d6b373c4a55bfb2d4627ef5a635a65536461f861fc47bfa2"}
Oct 31 01:04:01.890158 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:04:01.890132    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-t7r6c" event={"ID":"85bb69db-c69b-467e-adeb-950bf2947c94","Type":"ContainerStarted","Data":"926e3b2a644d5d47133af4901e7a3d827dd6e62f26984ea0b63b0266c1ef1153"}
Oct 31 01:04:01.891266 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:04:01.891246    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-wdsmn" event={"ID":"8952fc79-c5a6-42ba-a296-8cb7a776f718","Type":"ContainerStarted","Data":"f5b808dac049659e1093df65822bb8425a9885c5e1efbdb78ec9be81dc70b51a"}
Oct 31 01:04:02.055815 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:04:02.055785    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:04:02.446293 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:04:02.445667    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:04:02.644199 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:04:02.644160    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:04:03.904555 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:04:03.901714    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-srcwl" event={"ID":"f273b9e5-0419-4587-a9c7-a0c8bec02acf","Type":"ContainerStarted","Data":"d26ed0598a2e3c2b6228848650471d70348295152343f25e8f153676f4f108d0"}
Oct 31 01:04:03.904966 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:04:03.904596    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-srcwl" event={"ID":"f273b9e5-0419-4587-a9c7-a0c8bec02acf","Type":"ContainerStarted","Data":"1e8dfab7ab10ea86917595757aa748a7e90663d62a27ab11234614f652755352"}
Oct 31 01:04:03.907943 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:04:03.907916    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-t7r6c" event={"ID":"85bb69db-c69b-467e-adeb-950bf2947c94","Type":"ContainerStarted","Data":"f3d351d71e588c60de1261d0f073d774991ed92b136a29a838763635d39e6457"}
Oct 31 01:04:03.908060 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:04:03.907976    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-t7r6c" event={"ID":"85bb69db-c69b-467e-adeb-950bf2947c94","Type":"ContainerStarted","Data":"a29be4eca9c7b802e72bfe4a2195e9445f13dd617b5fffefdbd9ef9821cdf195"}
Oct 31 01:04:03.908367 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:04:03.908340    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-t7r6c"
Oct 31 01:04:03.910558 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:04:03.910539    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-wdsmn" event={"ID":"8952fc79-c5a6-42ba-a296-8cb7a776f718","Type":"ContainerStarted","Data":"e26cc37e27da68a53817aba992d8154cf490a4ec709dca81c527c074b4615804"}
Oct 31 01:04:03.910642 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:04:03.910573    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-wdsmn" event={"ID":"8952fc79-c5a6-42ba-a296-8cb7a776f718","Type":"ContainerStarted","Data":"6ad7e26ed0f311dabecc41339b2992715f9581043d709d839cec6aa4cc170211"}
Oct 31 01:04:05.769766 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:04:05.769740    4964 scope.go:122] "RemoveContainer" containerID="b70ef6e4580b026c0566fcbee7f62b195af7e015b7934645c9d9e3a310d27128"
Oct 31 01:04:05.919932 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:04:05.919904    4964 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-9257n_f0c0ada7-7688-4dc2-97fc-813e4d0c4ef3/console-operator/2.log"
Oct 31 01:04:05.920073 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:04:05.920019    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-9257n" event={"ID":"f0c0ada7-7688-4dc2-97fc-813e4d0c4ef3","Type":"ContainerStarted","Data":"1f997a4c53b4bbf50baa31e2501e6c4be22d0d52e4c816a844b1835b476b064a"}
Oct 31 01:04:05.920372 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:04:05.920354    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-9257n"
Oct 31 01:04:05.921196 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:04:05.921173    4964 patch_prober.go:29] interesting pod/console-operator-7cd56b8dbc-9257n container/console-operator namespace/openshift-console-operator: Readiness probe status=failure output="Get \"https://10.132.0.15:8443/readyz\": dial tcp 10.132.0.15:8443: connect: connection refused" start-of-body=
Oct 31 01:04:05.921307 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:04:05.921210    4964 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-console-operator/console-operator-7cd56b8dbc-9257n" podUID="f0c0ada7-7688-4dc2-97fc-813e4d0c4ef3" containerName="console-operator" probeResult="failure" output="Get \"https://10.132.0.15:8443/readyz\": dial tcp 10.132.0.15:8443: connect: connection refused"
Oct 31 01:04:06.923861 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:04:06.923828    4964 patch_prober.go:29] interesting pod/console-operator-7cd56b8dbc-9257n container/console-operator namespace/openshift-console-operator: Readiness probe status=failure output="Get \"https://10.132.0.15:8443/readyz\": dial tcp 10.132.0.15:8443: connect: connection refused" start-of-body=
Oct 31 01:04:06.924263 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:04:06.923873    4964 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-console-operator/console-operator-7cd56b8dbc-9257n" podUID="f0c0ada7-7688-4dc2-97fc-813e4d0c4ef3" containerName="console-operator" probeResult="failure" output="Get \"https://10.132.0.15:8443/readyz\": dial tcp 10.132.0.15:8443: connect: connection refused"
Oct 31 01:04:08.002062 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:04:08.002026    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:04:08.175961 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:04:08.175921    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:04:08.740726 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:04:08.740696    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7dd98c21-e0b1-4289-8fe8-036bf413d444-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-q56x5\" (UID: \"7dd98c21-e0b1-4289-8fe8-036bf413d444\") " pod="openshift-insights/insights-runtime-extractor-q56x5"
Oct 31 01:04:08.740726 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:04:08.740729    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7dd98c21-e0b1-4289-8fe8-036bf413d444-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-q56x5\" (UID: \"7dd98c21-e0b1-4289-8fe8-036bf413d444\") " pod="openshift-insights/insights-runtime-extractor-q56x5"
Oct 31 01:04:08.740957 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:04:08.740761    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/9dfae6d6-b9a6-4d85-b594-9d947a9e6383-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-wm4qt\" (UID: \"9dfae6d6-b9a6-4d85-b594-9d947a9e6383\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-wm4qt"
Oct 31 01:04:08.740957 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:04:08.740780    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e49453a8-c7d9-4248-a8ef-4a819b76806c-service-ca-bundle\") pod \"router-default-5964fff8f9-8k6pp\" (UID: \"e49453a8-c7d9-4248-a8ef-4a819b76806c\") " pod="openshift-ingress/router-default-5964fff8f9-8k6pp"
Oct 31 01:04:08.741393 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:04:08.741371    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7dd98c21-e0b1-4289-8fe8-036bf413d444-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-q56x5\" (UID: \"7dd98c21-e0b1-4289-8fe8-036bf413d444\") " pod="openshift-insights/insights-runtime-extractor-q56x5"
Oct 31 01:04:08.741507 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:04:08.741423    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e49453a8-c7d9-4248-a8ef-4a819b76806c-service-ca-bundle\") pod \"router-default-5964fff8f9-8k6pp\" (UID: \"e49453a8-c7d9-4248-a8ef-4a819b76806c\") " pod="openshift-ingress/router-default-5964fff8f9-8k6pp"
Oct 31 01:04:08.743130 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:04:08.743109    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7dd98c21-e0b1-4289-8fe8-036bf413d444-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-q56x5\" (UID: \"7dd98c21-e0b1-4289-8fe8-036bf413d444\") " pod="openshift-insights/insights-runtime-extractor-q56x5"
Oct 31 01:04:08.743255 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:04:08.743237    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/9dfae6d6-b9a6-4d85-b594-9d947a9e6383-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-wm4qt\" (UID: \"9dfae6d6-b9a6-4d85-b594-9d947a9e6383\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-wm4qt"
Oct 31 01:04:08.747107 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:04:08.747089    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-q56x5"
Oct 31 01:04:09.025883 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:04:09.025813    4964 patch_prober.go:29] interesting pod/image-registry-75575b5db8-wtq4d container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Oct 31 01:04:09.025883 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:04:09.025868    4964 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-75575b5db8-wtq4d" podUID="aa112079-b749-404c-9081-113220707fcd" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 01:04:09.036361 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:04:09.036339    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-5964fff8f9-8k6pp"
Oct 31 01:04:09.038100 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:04:09.038083    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-wm4qt"
Oct 31 01:04:09.116152 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:04:09.116125    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-q56x5"]
Oct 31 01:04:09.118927 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:04:09.118905    4964 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7dd98c21_e0b1_4289_8fe8_036bf413d444.slice/crio-b341e45e493c55f1210673afa7df96db14d135059c7da15c766d72b20eada42e WatchSource:0}: Error finding container b341e45e493c55f1210673afa7df96db14d135059c7da15c766d72b20eada42e: Status 404 returned error can't find the container with id b341e45e493c55f1210673afa7df96db14d135059c7da15c766d72b20eada42e
Oct 31 01:04:09.409970 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:04:09.409895    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-wm4qt"]
Oct 31 01:04:09.413046 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:04:09.413018    4964 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9dfae6d6_b9a6_4d85_b594_9d947a9e6383.slice/crio-b388b41344e2976ed6386cad2a1005419bd546ddfdf3afa9df1134e72349d3e4 WatchSource:0}: Error finding container b388b41344e2976ed6386cad2a1005419bd546ddfdf3afa9df1134e72349d3e4: Status 404 returned error can't find the container with id b388b41344e2976ed6386cad2a1005419bd546ddfdf3afa9df1134e72349d3e4
Oct 31 01:04:09.462146 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:04:09.462120    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-5964fff8f9-8k6pp"]
Oct 31 01:04:09.464852 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:04:09.464830    4964 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode49453a8_c7d9_4248_a8ef_4a819b76806c.slice/crio-b5213062f99f7c9fdcc2248a8119790f797e3186007ce57b691de9d3152c12de WatchSource:0}: Error finding container b5213062f99f7c9fdcc2248a8119790f797e3186007ce57b691de9d3152c12de: Status 404 returned error can't find the container with id b5213062f99f7c9fdcc2248a8119790f797e3186007ce57b691de9d3152c12de
Oct 31 01:04:09.561006 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:04:09.560981    4964 patch_prober.go:29] interesting pod/image-registry-75575b5db8-wtq4d container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Oct 31 01:04:09.561140 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:04:09.561024    4964 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-75575b5db8-wtq4d" podUID="aa112079-b749-404c-9081-113220707fcd" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 01:04:09.938984 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:04:09.938953    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-q56x5" event={"ID":"7dd98c21-e0b1-4289-8fe8-036bf413d444","Type":"ContainerStarted","Data":"5da5fe5c73b559da32b63170f5b152c5d7552905933b6ec026b361fb4ccbd7cd"}
Oct 31 01:04:09.939147 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:04:09.939012    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-q56x5" event={"ID":"7dd98c21-e0b1-4289-8fe8-036bf413d444","Type":"ContainerStarted","Data":"b341e45e493c55f1210673afa7df96db14d135059c7da15c766d72b20eada42e"}
Oct 31 01:04:09.940473 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:04:09.940438    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-5964fff8f9-8k6pp" event={"ID":"e49453a8-c7d9-4248-a8ef-4a819b76806c","Type":"ContainerStarted","Data":"3b0295fe0d0c3223ec88a0e9df36e6eaf944f4f8fd39bea3bbf77fc21c5a135a"}
Oct 31 01:04:09.940620 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:04:09.940479    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-5964fff8f9-8k6pp" event={"ID":"e49453a8-c7d9-4248-a8ef-4a819b76806c","Type":"ContainerStarted","Data":"b5213062f99f7c9fdcc2248a8119790f797e3186007ce57b691de9d3152c12de"}
Oct 31 01:04:09.941557 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:04:09.941537    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-86c478bb88-wm4qt" event={"ID":"9dfae6d6-b9a6-4d85-b594-9d947a9e6383","Type":"ContainerStarted","Data":"b388b41344e2976ed6386cad2a1005419bd546ddfdf3afa9df1134e72349d3e4"}
Oct 31 01:04:10.037329 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:04:10.037278    4964 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-5964fff8f9-8k6pp"
Oct 31 01:04:10.040961 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:04:10.040937    4964 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-5964fff8f9-8k6pp"
Oct 31 01:04:10.193648 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:04:10.193566    4964 patch_prober.go:29] interesting pod/console-operator-7cd56b8dbc-9257n container/console-operator namespace/openshift-console-operator: Liveness probe status=failure output="Get \"https://10.132.0.15:8443/healthz\": dial tcp 10.132.0.15:8443: connect: connection refused" start-of-body=
Oct 31 01:04:10.193648 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:04:10.193626    4964 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-console-operator/console-operator-7cd56b8dbc-9257n" podUID="f0c0ada7-7688-4dc2-97fc-813e4d0c4ef3" containerName="console-operator" probeResult="failure" output="Get \"https://10.132.0.15:8443/healthz\": dial tcp 10.132.0.15:8443: connect: connection refused"
Oct 31 01:04:10.240939 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:04:10.240898    4964 patch_prober.go:29] interesting pod/image-registry-56d8464d7c-vpwjc container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Oct 31 01:04:10.241090 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:04:10.240954    4964 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-56d8464d7c-vpwjc" podUID="ff4dcba2-df8d-4324-9e0c-abb23a5e6982" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 01:04:10.947447 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:04:10.947415    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-q56x5" event={"ID":"7dd98c21-e0b1-4289-8fe8-036bf413d444","Type":"ContainerStarted","Data":"d6aa8622381f09ce74b79de999a81e535cd946b94117b957819422fae7073772"}
Oct 31 01:04:10.948865 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:04:10.948843    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-86c478bb88-wm4qt" event={"ID":"9dfae6d6-b9a6-4d85-b594-9d947a9e6383","Type":"ContainerStarted","Data":"083e26414058088300c448262b6265e58d099d7d97f0ff9c77726b2157333080"}
Oct 31 01:04:10.949166 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:04:10.949149    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-5964fff8f9-8k6pp"
Oct 31 01:04:10.950290 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:04:10.950270    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-5964fff8f9-8k6pp"
Oct 31 01:04:11.379814 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:04:11.379780    4964 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-xwkdt"]
Oct 31 01:04:11.382291 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:04:11.382272    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-xwkdt"
Oct 31 01:04:11.420300 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:04:11.420263    4964 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-56d8464d7c-vpwjc"]
Oct 31 01:04:11.424963 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:04:11.424929    4964 patch_prober.go:29] interesting pod/image-registry-56d8464d7c-vpwjc container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Oct 31 01:04:11.425112 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:04:11.424976    4964 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-56d8464d7c-vpwjc" podUID="ff4dcba2-df8d-4324-9e0c-abb23a5e6982" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 01:04:11.440484 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:04:11.440463    4964 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\""
Oct 31 01:04:11.440484 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:04:11.440464    4964 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-2bf77\""
Oct 31 01:04:11.447037 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:04:11.447005    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-xwkdt"]
Oct 31 01:04:11.462735 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:04:11.462706    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/b6953f8d-148b-4079-b4fa-365d83084e44-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-xwkdt\" (UID: \"b6953f8d-148b-4079-b4fa-365d83084e44\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-xwkdt"
Oct 31 01:04:11.481458 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:04:11.481430    4964 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-6464b7c4f6-dfbph"]
Oct 31 01:04:11.484335 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:04:11.484295    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6464b7c4f6-dfbph"
Oct 31 01:04:11.486579 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:04:11.486541    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:04:11.549230 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:04:11.549127    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6464b7c4f6-dfbph"]
Oct 31 01:04:11.563834 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:04:11.563809    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/e5089828-1c26-424e-b51b-d62ca3490b07-registry-certificates\") pod \"image-registry-6464b7c4f6-dfbph\" (UID: \"e5089828-1c26-424e-b51b-d62ca3490b07\") " pod="openshift-image-registry/image-registry-6464b7c4f6-dfbph"
Oct 31 01:04:11.563975 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:04:11.563896    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/e5089828-1c26-424e-b51b-d62ca3490b07-ca-trust-extracted\") pod \"image-registry-6464b7c4f6-dfbph\" (UID: \"e5089828-1c26-424e-b51b-d62ca3490b07\") " pod="openshift-image-registry/image-registry-6464b7c4f6-dfbph"
Oct 31 01:04:11.563975 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:04:11.563924    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/e5089828-1c26-424e-b51b-d62ca3490b07-image-registry-private-configuration\") pod \"image-registry-6464b7c4f6-dfbph\" (UID: \"e5089828-1c26-424e-b51b-d62ca3490b07\") " pod="openshift-image-registry/image-registry-6464b7c4f6-dfbph"
Oct 31 01:04:11.564059 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:04:11.564003    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e5089828-1c26-424e-b51b-d62ca3490b07-registry-tls\") pod \"image-registry-6464b7c4f6-dfbph\" (UID: \"e5089828-1c26-424e-b51b-d62ca3490b07\") " pod="openshift-image-registry/image-registry-6464b7c4f6-dfbph"
Oct 31 01:04:11.564059 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:04:11.564028    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e5089828-1c26-424e-b51b-d62ca3490b07-bound-sa-token\") pod \"image-registry-6464b7c4f6-dfbph\" (UID: \"e5089828-1c26-424e-b51b-d62ca3490b07\") " pod="openshift-image-registry/image-registry-6464b7c4f6-dfbph"
Oct 31 01:04:11.564128 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:04:11.564055    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/b6953f8d-148b-4079-b4fa-365d83084e44-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-xwkdt\" (UID: \"b6953f8d-148b-4079-b4fa-365d83084e44\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-xwkdt"
Oct 31 01:04:11.564128 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:04:11.564106    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/e5089828-1c26-424e-b51b-d62ca3490b07-trusted-ca\") pod \"image-registry-6464b7c4f6-dfbph\" (UID: \"e5089828-1c26-424e-b51b-d62ca3490b07\") " pod="openshift-image-registry/image-registry-6464b7c4f6-dfbph"
Oct 31 01:04:11.564208 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:04:11.564135    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/e5089828-1c26-424e-b51b-d62ca3490b07-installation-pull-secrets\") pod \"image-registry-6464b7c4f6-dfbph\" (UID: \"e5089828-1c26-424e-b51b-d62ca3490b07\") " pod="openshift-image-registry/image-registry-6464b7c4f6-dfbph"
Oct 31 01:04:11.564208 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:04:11.564173    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c8n9s\" (UniqueName: \"kubernetes.io/projected/e5089828-1c26-424e-b51b-d62ca3490b07-kube-api-access-c8n9s\") pod \"image-registry-6464b7c4f6-dfbph\" (UID: \"e5089828-1c26-424e-b51b-d62ca3490b07\") " pod="openshift-image-registry/image-registry-6464b7c4f6-dfbph"
Oct 31 01:04:11.566791 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:04:11.566764    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/b6953f8d-148b-4079-b4fa-365d83084e44-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-xwkdt\" (UID: \"b6953f8d-148b-4079-b4fa-365d83084e44\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-xwkdt"
Oct 31 01:04:11.664884 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:04:11.664805    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e5089828-1c26-424e-b51b-d62ca3490b07-bound-sa-token\") pod \"image-registry-6464b7c4f6-dfbph\" (UID: \"e5089828-1c26-424e-b51b-d62ca3490b07\") " pod="openshift-image-registry/image-registry-6464b7c4f6-dfbph"
Oct 31 01:04:11.664884 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:04:11.664855    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/e5089828-1c26-424e-b51b-d62ca3490b07-trusted-ca\") pod \"image-registry-6464b7c4f6-dfbph\" (UID: \"e5089828-1c26-424e-b51b-d62ca3490b07\") " pod="openshift-image-registry/image-registry-6464b7c4f6-dfbph"
Oct 31 01:04:11.664884 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:04:11.664882    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/e5089828-1c26-424e-b51b-d62ca3490b07-installation-pull-secrets\") pod \"image-registry-6464b7c4f6-dfbph\" (UID: \"e5089828-1c26-424e-b51b-d62ca3490b07\") " pod="openshift-image-registry/image-registry-6464b7c4f6-dfbph"
Oct 31 01:04:11.665122 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:04:11.664915    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c8n9s\" (UniqueName: \"kubernetes.io/projected/e5089828-1c26-424e-b51b-d62ca3490b07-kube-api-access-c8n9s\") pod \"image-registry-6464b7c4f6-dfbph\" (UID: \"e5089828-1c26-424e-b51b-d62ca3490b07\") " pod="openshift-image-registry/image-registry-6464b7c4f6-dfbph"
Oct 31 01:04:11.665122 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:04:11.664938    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/e5089828-1c26-424e-b51b-d62ca3490b07-registry-certificates\") pod \"image-registry-6464b7c4f6-dfbph\" (UID: \"e5089828-1c26-424e-b51b-d62ca3490b07\") " pod="openshift-image-registry/image-registry-6464b7c4f6-dfbph"
Oct 31 01:04:11.665122 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:04:11.664971    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/e5089828-1c26-424e-b51b-d62ca3490b07-ca-trust-extracted\") pod \"image-registry-6464b7c4f6-dfbph\" (UID: \"e5089828-1c26-424e-b51b-d62ca3490b07\") " pod="openshift-image-registry/image-registry-6464b7c4f6-dfbph"
Oct 31 01:04:11.665122 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:04:11.665002    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/e5089828-1c26-424e-b51b-d62ca3490b07-image-registry-private-configuration\") pod \"image-registry-6464b7c4f6-dfbph\" (UID: \"e5089828-1c26-424e-b51b-d62ca3490b07\") " pod="openshift-image-registry/image-registry-6464b7c4f6-dfbph"
Oct 31 01:04:11.665122 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:04:11.665040    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e5089828-1c26-424e-b51b-d62ca3490b07-registry-tls\") pod \"image-registry-6464b7c4f6-dfbph\" (UID: \"e5089828-1c26-424e-b51b-d62ca3490b07\") " pod="openshift-image-registry/image-registry-6464b7c4f6-dfbph"
Oct 31 01:04:11.665611 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:04:11.665582    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/e5089828-1c26-424e-b51b-d62ca3490b07-ca-trust-extracted\") pod \"image-registry-6464b7c4f6-dfbph\" (UID: \"e5089828-1c26-424e-b51b-d62ca3490b07\") " pod="openshift-image-registry/image-registry-6464b7c4f6-dfbph"
Oct 31 01:04:11.666025 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:04:11.666002    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/e5089828-1c26-424e-b51b-d62ca3490b07-registry-certificates\") pod \"image-registry-6464b7c4f6-dfbph\" (UID: \"e5089828-1c26-424e-b51b-d62ca3490b07\") " pod="openshift-image-registry/image-registry-6464b7c4f6-dfbph"
Oct 31 01:04:11.666124 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:04:11.666087    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/e5089828-1c26-424e-b51b-d62ca3490b07-trusted-ca\") pod \"image-registry-6464b7c4f6-dfbph\" (UID: \"e5089828-1c26-424e-b51b-d62ca3490b07\") " pod="openshift-image-registry/image-registry-6464b7c4f6-dfbph"
Oct 31 01:04:11.667735 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:04:11.667710    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/e5089828-1c26-424e-b51b-d62ca3490b07-image-registry-private-configuration\") pod \"image-registry-6464b7c4f6-dfbph\" (UID: \"e5089828-1c26-424e-b51b-d62ca3490b07\") " pod="openshift-image-registry/image-registry-6464b7c4f6-dfbph"
Oct 31 01:04:11.668049 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:04:11.668028    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/e5089828-1c26-424e-b51b-d62ca3490b07-installation-pull-secrets\") pod \"image-registry-6464b7c4f6-dfbph\" (UID: \"e5089828-1c26-424e-b51b-d62ca3490b07\") " pod="openshift-image-registry/image-registry-6464b7c4f6-dfbph"
Oct 31 01:04:11.668238 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:04:11.668214    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e5089828-1c26-424e-b51b-d62ca3490b07-registry-tls\") pod \"image-registry-6464b7c4f6-dfbph\" (UID: \"e5089828-1c26-424e-b51b-d62ca3490b07\") " pod="openshift-image-registry/image-registry-6464b7c4f6-dfbph"
Oct 31 01:04:11.695061 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:04:11.695029    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-xwkdt"
Oct 31 01:04:11.721533 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:04:11.721264    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:04:11.722416 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:04:11.722392    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c8n9s\" (UniqueName: \"kubernetes.io/projected/e5089828-1c26-424e-b51b-d62ca3490b07-kube-api-access-c8n9s\") pod \"image-registry-6464b7c4f6-dfbph\" (UID: \"e5089828-1c26-424e-b51b-d62ca3490b07\") " pod="openshift-image-registry/image-registry-6464b7c4f6-dfbph"
Oct 31 01:04:11.722817 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:04:11.722800    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e5089828-1c26-424e-b51b-d62ca3490b07-bound-sa-token\") pod \"image-registry-6464b7c4f6-dfbph\" (UID: \"e5089828-1c26-424e-b51b-d62ca3490b07\") " pod="openshift-image-registry/image-registry-6464b7c4f6-dfbph"
Oct 31 01:04:11.796558 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:04:11.796523    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6464b7c4f6-dfbph"
Oct 31 01:04:12.072456 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:04:12.072372    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-xwkdt"]
Oct 31 01:04:12.075124 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:04:12.075095    4964 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6953f8d_148b_4079_b4fa_365d83084e44.slice/crio-5c66a9709e1b8a718fbf1be27f5baa445221ab8979248ec1ed3c9867e5b71ad8 WatchSource:0}: Error finding container 5c66a9709e1b8a718fbf1be27f5baa445221ab8979248ec1ed3c9867e5b71ad8: Status 404 returned error can't find the container with id 5c66a9709e1b8a718fbf1be27f5baa445221ab8979248ec1ed3c9867e5b71ad8
Oct 31 01:04:12.106613 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:04:12.106577    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:04:12.187761 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:04:12.187728    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6464b7c4f6-dfbph"]
Oct 31 01:04:12.191619 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:04:12.191590    4964 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5089828_1c26_424e_b51b_d62ca3490b07.slice/crio-5c11cbc6d1ca4dcd0ebc14f96acaa8a9ed8c7b412851bf1dea0a958d6caccec7 WatchSource:0}: Error finding container 5c11cbc6d1ca4dcd0ebc14f96acaa8a9ed8c7b412851bf1dea0a958d6caccec7: Status 404 returned error can't find the container with id 5c11cbc6d1ca4dcd0ebc14f96acaa8a9ed8c7b412851bf1dea0a958d6caccec7
Oct 31 01:04:12.305930 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:04:12.305887    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:04:12.878475 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:04:12.878397    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:04:12.959124 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:04:12.959091    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-xwkdt" event={"ID":"b6953f8d-148b-4079-b4fa-365d83084e44","Type":"ContainerStarted","Data":"5c66a9709e1b8a718fbf1be27f5baa445221ab8979248ec1ed3c9867e5b71ad8"}
Oct 31 01:04:12.961178 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:04:12.961150    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-q56x5" event={"ID":"7dd98c21-e0b1-4289-8fe8-036bf413d444","Type":"ContainerStarted","Data":"419125f048616655a7f31ed2a94cc4f0331563923f6e6affb96af3b646c4c375"}
Oct 31 01:04:12.962703 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:04:12.962680    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6464b7c4f6-dfbph" event={"ID":"e5089828-1c26-424e-b51b-d62ca3490b07","Type":"ContainerStarted","Data":"6b4acbdf3c6b9b67fc0241673c9cb2627649738681a0d36ce3c473a8f1efda2d"}
Oct 31 01:04:12.962812 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:04:12.962716    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6464b7c4f6-dfbph" event={"ID":"e5089828-1c26-424e-b51b-d62ca3490b07","Type":"ContainerStarted","Data":"5c11cbc6d1ca4dcd0ebc14f96acaa8a9ed8c7b412851bf1dea0a958d6caccec7"}
Oct 31 01:04:13.196507 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:04:13.196340    4964 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/image-registry-6464b7c4f6-dfbph" podStartSLOduration=2.196305848 podStartE2EDuration="2.196305848s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:04:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:04:13.194929408 +0000 UTC m=+21.671495802" watchObservedRunningTime="2026-10-31 01:04:13.196305848 +0000 UTC m=+21.672872223"
Oct 31 01:04:13.852001 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:04:13.851971    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-pghw2"
Oct 31 01:04:13.916815 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:04:13.916787    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-t7r6c"
Oct 31 01:04:13.969383 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:04:13.969353    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-xwkdt" event={"ID":"b6953f8d-148b-4079-b4fa-365d83084e44","Type":"ContainerStarted","Data":"7d1181a8d6cb5f93e0a412041d45d98d3e1660208191e88de96610ad809d7835"}
Oct 31 01:04:13.969722 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:04:13.969705    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-6464b7c4f6-dfbph"
Oct 31 01:04:13.969792 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:04:13.969745    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-xwkdt"
Oct 31 01:04:13.974172 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:04:13.974140    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-xwkdt"
Oct 31 01:04:14.135157 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:04:14.135045    4964 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-xwkdt" podStartSLOduration=1.642757679 podStartE2EDuration="3.135024494s" totalImagesPullingTime="1.492266815s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:04:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:04:14.134642356 +0000 UTC m=+22.611208727" watchObservedRunningTime="2026-10-31 01:04:14.135024494 +0000 UTC m=+22.611590875"
Oct 31 01:04:14.563733 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:04:14.563687    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:04:14.741174 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:04:14.741139    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:04:14.953834 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:04:14.953737    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:04:15.134920 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:04:15.134875    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:04:16.928392 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:04:16.928364    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-9257n"
Oct 31 01:04:17.246263 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:04:17.246225    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:04:17.490551 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:04:17.490507    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:04:17.673840 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:04:17.673756    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:04:18.256850 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:04:18.256810    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:04:18.432287 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:04:18.432257    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:04:19.026186 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:04:19.026159    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-75575b5db8-wtq4d"
Oct 31 01:04:19.048829 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:04:19.048794    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:04:19.333151 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:04:19.333061    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:04:19.531937 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:04:19.531901    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:04:19.780039 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:04:19.780004    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:04:19.979328 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:04:19.979285    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:04:21.269510 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:04:21.269470    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:04:21.425172 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:04:21.425144    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-56d8464d7c-vpwjc"
Oct 31 01:04:21.543687 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:04:21.543608    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-8mqbd"
Oct 31 01:04:21.993522 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:04:21.993488    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:04:22.185994 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:04:22.185954    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:04:22.436871 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:04:22.436783    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:04:22.636997 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:04:22.636962    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:04:22.683126 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:04:22.683096    4964 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-kwtlx"]
Oct 31 01:04:22.715278 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:04:22.715202    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-kwtlx"
Oct 31 01:04:22.764124 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:04:22.764024    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/24af6b7c-9cc3-43e6-be11-192cd1a3678d-node-exporter-tls\") pod \"node-exporter-kwtlx\" (UID: \"24af6b7c-9cc3-43e6-be11-192cd1a3678d\") " pod="openshift-monitoring/node-exporter-kwtlx"
Oct 31 01:04:22.764124 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:04:22.764076    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/24af6b7c-9cc3-43e6-be11-192cd1a3678d-metrics-client-ca\") pod \"node-exporter-kwtlx\" (UID: \"24af6b7c-9cc3-43e6-be11-192cd1a3678d\") " pod="openshift-monitoring/node-exporter-kwtlx"
Oct 31 01:04:22.764413 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:04:22.764183    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/24af6b7c-9cc3-43e6-be11-192cd1a3678d-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-kwtlx\" (UID: \"24af6b7c-9cc3-43e6-be11-192cd1a3678d\") " pod="openshift-monitoring/node-exporter-kwtlx"
Oct 31 01:04:22.764413 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:04:22.764232    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dpnrb\" (UniqueName: \"kubernetes.io/projected/24af6b7c-9cc3-43e6-be11-192cd1a3678d-kube-api-access-dpnrb\") pod \"node-exporter-kwtlx\" (UID: \"24af6b7c-9cc3-43e6-be11-192cd1a3678d\") " pod="openshift-monitoring/node-exporter-kwtlx"
Oct 31 01:04:22.764413 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:04:22.764255    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/24af6b7c-9cc3-43e6-be11-192cd1a3678d-sys\") pod \"node-exporter-kwtlx\" (UID: \"24af6b7c-9cc3-43e6-be11-192cd1a3678d\") " pod="openshift-monitoring/node-exporter-kwtlx"
Oct 31 01:04:22.764413 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:04:22.764281    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/24af6b7c-9cc3-43e6-be11-192cd1a3678d-root\") pod \"node-exporter-kwtlx\" (UID: \"24af6b7c-9cc3-43e6-be11-192cd1a3678d\") " pod="openshift-monitoring/node-exporter-kwtlx"
Oct 31 01:04:22.764413 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:04:22.764329    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/24af6b7c-9cc3-43e6-be11-192cd1a3678d-node-exporter-accelerators-collector-config\") pod \"node-exporter-kwtlx\" (UID: \"24af6b7c-9cc3-43e6-be11-192cd1a3678d\") " pod="openshift-monitoring/node-exporter-kwtlx"
Oct 31 01:04:22.764413 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:04:22.764360    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/24af6b7c-9cc3-43e6-be11-192cd1a3678d-node-exporter-wtmp\") pod \"node-exporter-kwtlx\" (UID: \"24af6b7c-9cc3-43e6-be11-192cd1a3678d\") " pod="openshift-monitoring/node-exporter-kwtlx"
Oct 31 01:04:22.764413 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:04:22.764407    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/24af6b7c-9cc3-43e6-be11-192cd1a3678d-node-exporter-textfile\") pod \"node-exporter-kwtlx\" (UID: \"24af6b7c-9cc3-43e6-be11-192cd1a3678d\") " pod="openshift-monitoring/node-exporter-kwtlx"
Oct 31 01:04:22.811457 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:04:22.811425    4964 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Oct 31 01:04:22.816366 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:04:22.816341    4964 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Oct 31 01:04:22.822680 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:04:22.822654    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:04:22.865588 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:04:22.865553    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/24af6b7c-9cc3-43e6-be11-192cd1a3678d-node-exporter-textfile\") pod \"node-exporter-kwtlx\" (UID: \"24af6b7c-9cc3-43e6-be11-192cd1a3678d\") " pod="openshift-monitoring/node-exporter-kwtlx"
Oct 31 01:04:22.865777 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:04:22.865607    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/24af6b7c-9cc3-43e6-be11-192cd1a3678d-node-exporter-tls\") pod \"node-exporter-kwtlx\" (UID: \"24af6b7c-9cc3-43e6-be11-192cd1a3678d\") " pod="openshift-monitoring/node-exporter-kwtlx"
Oct 31 01:04:22.865845 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:04:22.865820    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/24af6b7c-9cc3-43e6-be11-192cd1a3678d-metrics-client-ca\") pod \"node-exporter-kwtlx\" (UID: \"24af6b7c-9cc3-43e6-be11-192cd1a3678d\") " pod="openshift-monitoring/node-exporter-kwtlx"
Oct 31 01:04:22.865886 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:04:22.865870    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/24af6b7c-9cc3-43e6-be11-192cd1a3678d-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-kwtlx\" (UID: \"24af6b7c-9cc3-43e6-be11-192cd1a3678d\") " pod="openshift-monitoring/node-exporter-kwtlx"
Oct 31 01:04:22.865929 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:04:22.865915    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dpnrb\" (UniqueName: \"kubernetes.io/projected/24af6b7c-9cc3-43e6-be11-192cd1a3678d-kube-api-access-dpnrb\") pod \"node-exporter-kwtlx\" (UID: \"24af6b7c-9cc3-43e6-be11-192cd1a3678d\") " pod="openshift-monitoring/node-exporter-kwtlx"
Oct 31 01:04:22.865979 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:04:22.865926    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/24af6b7c-9cc3-43e6-be11-192cd1a3678d-node-exporter-textfile\") pod \"node-exporter-kwtlx\" (UID: \"24af6b7c-9cc3-43e6-be11-192cd1a3678d\") " pod="openshift-monitoring/node-exporter-kwtlx"
Oct 31 01:04:22.865979 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:04:22.865949    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/24af6b7c-9cc3-43e6-be11-192cd1a3678d-sys\") pod \"node-exporter-kwtlx\" (UID: \"24af6b7c-9cc3-43e6-be11-192cd1a3678d\") " pod="openshift-monitoring/node-exporter-kwtlx"
Oct 31 01:04:22.866087 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:04:22.865992    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/24af6b7c-9cc3-43e6-be11-192cd1a3678d-sys\") pod \"node-exporter-kwtlx\" (UID: \"24af6b7c-9cc3-43e6-be11-192cd1a3678d\") " pod="openshift-monitoring/node-exporter-kwtlx"
Oct 31 01:04:22.866087 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:04:22.865997    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/24af6b7c-9cc3-43e6-be11-192cd1a3678d-root\") pod \"node-exporter-kwtlx\" (UID: \"24af6b7c-9cc3-43e6-be11-192cd1a3678d\") " pod="openshift-monitoring/node-exporter-kwtlx"
Oct 31 01:04:22.866087 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:04:22.866010    4964 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Oct 31 01:04:22.866087 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:04:22.866039    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/24af6b7c-9cc3-43e6-be11-192cd1a3678d-root\") pod \"node-exporter-kwtlx\" (UID: \"24af6b7c-9cc3-43e6-be11-192cd1a3678d\") " pod="openshift-monitoring/node-exporter-kwtlx"
Oct 31 01:04:22.866087 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:04:22.866038    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/24af6b7c-9cc3-43e6-be11-192cd1a3678d-node-exporter-accelerators-collector-config\") pod \"node-exporter-kwtlx\" (UID: \"24af6b7c-9cc3-43e6-be11-192cd1a3678d\") " pod="openshift-monitoring/node-exporter-kwtlx"
Oct 31 01:04:22.866087 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:04:22.866082    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/24af6b7c-9cc3-43e6-be11-192cd1a3678d-node-exporter-wtmp\") pod \"node-exporter-kwtlx\" (UID: \"24af6b7c-9cc3-43e6-be11-192cd1a3678d\") " pod="openshift-monitoring/node-exporter-kwtlx"
Oct 31 01:04:22.866404 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:04:22.866251    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/24af6b7c-9cc3-43e6-be11-192cd1a3678d-node-exporter-wtmp\") pod \"node-exporter-kwtlx\" (UID: \"24af6b7c-9cc3-43e6-be11-192cd1a3678d\") " pod="openshift-monitoring/node-exporter-kwtlx"
Oct 31 01:04:22.866567 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:04:22.866546    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/24af6b7c-9cc3-43e6-be11-192cd1a3678d-metrics-client-ca\") pod \"node-exporter-kwtlx\" (UID: \"24af6b7c-9cc3-43e6-be11-192cd1a3678d\") " pod="openshift-monitoring/node-exporter-kwtlx"
Oct 31 01:04:22.866607 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:04:22.866588    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/24af6b7c-9cc3-43e6-be11-192cd1a3678d-node-exporter-accelerators-collector-config\") pod \"node-exporter-kwtlx\" (UID: \"24af6b7c-9cc3-43e6-be11-192cd1a3678d\") " pod="openshift-monitoring/node-exporter-kwtlx"
Oct 31 01:04:22.868629 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:04:22.868607    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/24af6b7c-9cc3-43e6-be11-192cd1a3678d-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-kwtlx\" (UID: \"24af6b7c-9cc3-43e6-be11-192cd1a3678d\") " pod="openshift-monitoring/node-exporter-kwtlx"
Oct 31 01:04:22.878195 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:04:22.878166    4964 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-ln2ts\""
Oct 31 01:04:22.878195 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:04:22.878166    4964 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Oct 31 01:04:22.888712 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:04:22.888677    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/24af6b7c-9cc3-43e6-be11-192cd1a3678d-node-exporter-tls\") pod \"node-exporter-kwtlx\" (UID: \"24af6b7c-9cc3-43e6-be11-192cd1a3678d\") " pod="openshift-monitoring/node-exporter-kwtlx"
Oct 31 01:04:22.922915 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:04:22.922889    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dpnrb\" (UniqueName: \"kubernetes.io/projected/24af6b7c-9cc3-43e6-be11-192cd1a3678d-kube-api-access-dpnrb\") pod \"node-exporter-kwtlx\" (UID: \"24af6b7c-9cc3-43e6-be11-192cd1a3678d\") " pod="openshift-monitoring/node-exporter-kwtlx"
Oct 31 01:04:23.004113 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:04:23.004067    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:04:23.072671 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:04:23.072636    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-kwtlx"
Oct 31 01:04:23.113030 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:04:23.113001    4964 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod24af6b7c_9cc3_43e6_be11_192cd1a3678d.slice/crio-1d9913a9f0dd9dc828b6402c917c594a3b62b1bb4c9128f78f79bb03171af848 WatchSource:0}: Error finding container 1d9913a9f0dd9dc828b6402c917c594a3b62b1bb4c9128f78f79bb03171af848: Status 404 returned error can't find the container with id 1d9913a9f0dd9dc828b6402c917c594a3b62b1bb4c9128f78f79bb03171af848
Oct 31 01:04:23.257431 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:04:23.257350    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:04:23.463720 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:04:23.463683    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:04:24.015183 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:04:24.015144    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-kwtlx" event={"ID":"24af6b7c-9cc3-43e6-be11-192cd1a3678d","Type":"ContainerStarted","Data":"1d9913a9f0dd9dc828b6402c917c594a3b62b1bb4c9128f78f79bb03171af848"}
Oct 31 01:04:24.105375 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:04:24.105310    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:04:24.621607 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:04:24.621575    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:04:24.987153 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:04:24.987122    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:04:26.025612 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:04:26.025570    4964 generic.go:356] "Generic (PLEG): container finished" podID="24af6b7c-9cc3-43e6-be11-192cd1a3678d" containerID="4dd517fc8572ee25f6eb3a254bc293d7b3c0d550ebea9d3ff84e044b2cf57c7e" exitCode=0
Oct 31 01:04:26.026006 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:04:26.025620    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-kwtlx" event={"ID":"24af6b7c-9cc3-43e6-be11-192cd1a3678d","Type":"ContainerDied","Data":"4dd517fc8572ee25f6eb3a254bc293d7b3c0d550ebea9d3ff84e044b2cf57c7e"}
Oct 31 01:04:26.872926 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:04:26.872886    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:04:27.033488 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:04:27.033450    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-kwtlx" event={"ID":"24af6b7c-9cc3-43e6-be11-192cd1a3678d","Type":"ContainerStarted","Data":"383050ff14f18005f86cf1ad03abd9e6955bdfa4b99dedb53046537dab5c80f4"}
Oct 31 01:04:27.033903 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:04:27.033501    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-kwtlx" event={"ID":"24af6b7c-9cc3-43e6-be11-192cd1a3678d","Type":"ContainerStarted","Data":"f9d4fbc4764a50bea7c1dfa7fb18c4a6c935897c6508e4bccefc583861800131"}
Oct 31 01:04:27.145708 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:04:27.145626    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:04:27.182697 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:04:27.182595    4964 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-kwtlx" podStartSLOduration=3.146933046 podStartE2EDuration="5.182575191s" totalImagesPullingTime="1.947265839s" totalInitContainerRuntime="88.376306ms" isStatefulPod=true podCreationTimestamp="2026-10-31 01:04:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:04:27.181957089 +0000 UTC m=+35.658523470" watchObservedRunningTime="2026-10-31 01:04:27.182575191 +0000 UTC m=+35.659141571"
Oct 31 01:04:27.322483 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:04:27.322443    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:04:27.522367 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:04:27.522331    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:04:27.705228 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:04:27.705189    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:04:27.937896 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:04:27.937809    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:04:28.112760 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:04:28.112731    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:04:28.335673 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:04:28.335642    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:04:31.988515 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:04:31.988387    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:04:33.644100 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:04:33.644052    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:04:33.880324 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:04:33.880288    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:04:34.977577 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:04:34.977552    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-6464b7c4f6-dfbph"
Oct 31 01:04:35.197098 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:04:35.197069    4964 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-75575b5db8-wtq4d"]
Oct 31 01:04:35.326727 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:04:35.326635    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:04:35.542702 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:04:35.542675    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:04:35.737687 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:04:35.737656    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:04:36.441036 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:04:36.440989    4964 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-image-registry/image-registry-56d8464d7c-vpwjc" podUID="ff4dcba2-df8d-4324-9e0c-abb23a5e6982" containerName="registry" containerID="cri-o://f52fc06311bb5c0fd0ea014c6b80a1c1e84c54d8e0c75f46a4db73d0b18c3fd7" gracePeriod=30
Oct 31 01:04:36.680664 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:04:36.680639    4964 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-56d8464d7c-vpwjc"
Oct 31 01:04:36.702921 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:04:36.702846    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:04:36.785126 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:04:36.785089    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ff4dcba2-df8d-4324-9e0c-abb23a5e6982-registry-tls\" (UniqueName: \"kubernetes.io/projected/ff4dcba2-df8d-4324-9e0c-abb23a5e6982-registry-tls\") pod \"ff4dcba2-df8d-4324-9e0c-abb23a5e6982\" (UID: \"ff4dcba2-df8d-4324-9e0c-abb23a5e6982\") "
Oct 31 01:04:36.785126 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:04:36.785131    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ff4dcba2-df8d-4324-9e0c-abb23a5e6982-trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ff4dcba2-df8d-4324-9e0c-abb23a5e6982-trusted-ca\") pod \"ff4dcba2-df8d-4324-9e0c-abb23a5e6982\" (UID: \"ff4dcba2-df8d-4324-9e0c-abb23a5e6982\") "
Oct 31 01:04:36.785425 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:04:36.785165    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ff4dcba2-df8d-4324-9e0c-abb23a5e6982-registry-certificates\" (UniqueName: \"kubernetes.io/configmap/ff4dcba2-df8d-4324-9e0c-abb23a5e6982-registry-certificates\") pod \"ff4dcba2-df8d-4324-9e0c-abb23a5e6982\" (UID: \"ff4dcba2-df8d-4324-9e0c-abb23a5e6982\") "
Oct 31 01:04:36.785425 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:04:36.785203    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ff4dcba2-df8d-4324-9e0c-abb23a5e6982-bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ff4dcba2-df8d-4324-9e0c-abb23a5e6982-bound-sa-token\") pod \"ff4dcba2-df8d-4324-9e0c-abb23a5e6982\" (UID: \"ff4dcba2-df8d-4324-9e0c-abb23a5e6982\") "
Oct 31 01:04:36.785425 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:04:36.785230    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff4dcba2-df8d-4324-9e0c-abb23a5e6982-ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/ff4dcba2-df8d-4324-9e0c-abb23a5e6982-ca-trust-extracted\") pod \"ff4dcba2-df8d-4324-9e0c-abb23a5e6982\" (UID: \"ff4dcba2-df8d-4324-9e0c-abb23a5e6982\") "
Oct 31 01:04:36.785425 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:04:36.785256    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ff4dcba2-df8d-4324-9e0c-abb23a5e6982-installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/ff4dcba2-df8d-4324-9e0c-abb23a5e6982-installation-pull-secrets\") pod \"ff4dcba2-df8d-4324-9e0c-abb23a5e6982\" (UID: \"ff4dcba2-df8d-4324-9e0c-abb23a5e6982\") "
Oct 31 01:04:36.785425 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:04:36.785332    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ff4dcba2-df8d-4324-9e0c-abb23a5e6982-kube-api-access-v9dnm\" (UniqueName: \"kubernetes.io/projected/ff4dcba2-df8d-4324-9e0c-abb23a5e6982-kube-api-access-v9dnm\") pod \"ff4dcba2-df8d-4324-9e0c-abb23a5e6982\" (UID: \"ff4dcba2-df8d-4324-9e0c-abb23a5e6982\") "
Oct 31 01:04:36.785425 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:04:36.785403    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ff4dcba2-df8d-4324-9e0c-abb23a5e6982-image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/ff4dcba2-df8d-4324-9e0c-abb23a5e6982-image-registry-private-configuration\") pod \"ff4dcba2-df8d-4324-9e0c-abb23a5e6982\" (UID: \"ff4dcba2-df8d-4324-9e0c-abb23a5e6982\") "
Oct 31 01:04:36.785713 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:04:36.785684    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ff4dcba2-df8d-4324-9e0c-abb23a5e6982-trusted-ca" pod "ff4dcba2-df8d-4324-9e0c-abb23a5e6982" (UID: "ff4dcba2-df8d-4324-9e0c-abb23a5e6982"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:04:36.785818 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:04:36.785774    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ff4dcba2-df8d-4324-9e0c-abb23a5e6982-registry-certificates" pod "ff4dcba2-df8d-4324-9e0c-abb23a5e6982" (UID: "ff4dcba2-df8d-4324-9e0c-abb23a5e6982"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:04:36.788065 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:04:36.788028    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff4dcba2-df8d-4324-9e0c-abb23a5e6982-bound-sa-token" pod "ff4dcba2-df8d-4324-9e0c-abb23a5e6982" (UID: "ff4dcba2-df8d-4324-9e0c-abb23a5e6982"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:04:36.788394 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:04:36.788366    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff4dcba2-df8d-4324-9e0c-abb23a5e6982-registry-tls" pod "ff4dcba2-df8d-4324-9e0c-abb23a5e6982" (UID: "ff4dcba2-df8d-4324-9e0c-abb23a5e6982"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:04:36.788513 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:04:36.788447    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ff4dcba2-df8d-4324-9e0c-abb23a5e6982-installation-pull-secrets" pod "ff4dcba2-df8d-4324-9e0c-abb23a5e6982" (UID: "ff4dcba2-df8d-4324-9e0c-abb23a5e6982"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:04:36.788564 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:04:36.788520    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ff4dcba2-df8d-4324-9e0c-abb23a5e6982-image-registry-private-configuration" pod "ff4dcba2-df8d-4324-9e0c-abb23a5e6982" (UID: "ff4dcba2-df8d-4324-9e0c-abb23a5e6982"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:04:36.788600 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:04:36.788582    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff4dcba2-df8d-4324-9e0c-abb23a5e6982-kube-api-access-v9dnm" pod "ff4dcba2-df8d-4324-9e0c-abb23a5e6982" (UID: "ff4dcba2-df8d-4324-9e0c-abb23a5e6982"). InnerVolumeSpecName "kube-api-access-v9dnm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:04:36.794433 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:04:36.794296    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff4dcba2-df8d-4324-9e0c-abb23a5e6982-ca-trust-extracted" pod "ff4dcba2-df8d-4324-9e0c-abb23a5e6982" (UID: "ff4dcba2-df8d-4324-9e0c-abb23a5e6982"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:04:36.883841 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:04:36.883807    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:04:36.889368 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:04:36.887268    4964 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ff4dcba2-df8d-4324-9e0c-abb23a5e6982-registry-tls\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:04:36.889368 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:04:36.887308    4964 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ff4dcba2-df8d-4324-9e0c-abb23a5e6982-trusted-ca\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:04:36.889368 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:04:36.887356    4964 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/ff4dcba2-df8d-4324-9e0c-abb23a5e6982-registry-certificates\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:04:36.889368 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:04:36.887381    4964 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ff4dcba2-df8d-4324-9e0c-abb23a5e6982-bound-sa-token\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:04:36.889368 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:04:36.887397    4964 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/ff4dcba2-df8d-4324-9e0c-abb23a5e6982-ca-trust-extracted\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:04:36.889368 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:04:36.887411    4964 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/ff4dcba2-df8d-4324-9e0c-abb23a5e6982-installation-pull-secrets\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:04:36.889368 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:04:36.887425    4964 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v9dnm\" (UniqueName: \"kubernetes.io/projected/ff4dcba2-df8d-4324-9e0c-abb23a5e6982-kube-api-access-v9dnm\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:04:36.889368 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:04:36.887446    4964 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/ff4dcba2-df8d-4324-9e0c-abb23a5e6982-image-registry-private-configuration\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:04:37.076965 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:04:37.076927    4964 generic.go:356] "Generic (PLEG): container finished" podID="ff4dcba2-df8d-4324-9e0c-abb23a5e6982" containerID="f52fc06311bb5c0fd0ea014c6b80a1c1e84c54d8e0c75f46a4db73d0b18c3fd7" exitCode=0
Oct 31 01:04:37.077154 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:04:37.076993    4964 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-56d8464d7c-vpwjc"
Oct 31 01:04:37.077154 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:04:37.077007    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-56d8464d7c-vpwjc" event={"ID":"ff4dcba2-df8d-4324-9e0c-abb23a5e6982","Type":"ContainerDied","Data":"f52fc06311bb5c0fd0ea014c6b80a1c1e84c54d8e0c75f46a4db73d0b18c3fd7"}
Oct 31 01:04:37.077154 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:04:37.077096    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-56d8464d7c-vpwjc" event={"ID":"ff4dcba2-df8d-4324-9e0c-abb23a5e6982","Type":"ContainerDied","Data":"9cd59c49b732bc348537c81f271393f2b7370197baf9479034c23d8b9260d572"}
Oct 31 01:04:37.077154 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:04:37.077123    4964 scope.go:122] "RemoveContainer" containerID="f52fc06311bb5c0fd0ea014c6b80a1c1e84c54d8e0c75f46a4db73d0b18c3fd7"
Oct 31 01:04:37.086150 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:04:37.086129    4964 scope.go:122] "RemoveContainer" containerID="f52fc06311bb5c0fd0ea014c6b80a1c1e84c54d8e0c75f46a4db73d0b18c3fd7"
Oct 31 01:04:37.086534 ip-10-0-133-197 kubenswrapper[4964]: E1031 01:04:37.086510    4964 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f52fc06311bb5c0fd0ea014c6b80a1c1e84c54d8e0c75f46a4db73d0b18c3fd7\": container with ID starting with f52fc06311bb5c0fd0ea014c6b80a1c1e84c54d8e0c75f46a4db73d0b18c3fd7 not found: ID does not exist" containerID="f52fc06311bb5c0fd0ea014c6b80a1c1e84c54d8e0c75f46a4db73d0b18c3fd7"
Oct 31 01:04:37.086638 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:04:37.086543    4964 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f52fc06311bb5c0fd0ea014c6b80a1c1e84c54d8e0c75f46a4db73d0b18c3fd7"} err="failed to get container status \"f52fc06311bb5c0fd0ea014c6b80a1c1e84c54d8e0c75f46a4db73d0b18c3fd7\": rpc error: code = NotFound desc = could not find container \"f52fc06311bb5c0fd0ea014c6b80a1c1e84c54d8e0c75f46a4db73d0b18c3fd7\": container with ID starting with f52fc06311bb5c0fd0ea014c6b80a1c1e84c54d8e0c75f46a4db73d0b18c3fd7 not found: ID does not exist"
Oct 31 01:04:37.240966 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:04:37.240927    4964 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-56d8464d7c-vpwjc"]
Oct 31 01:04:37.246959 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:04:37.246935    4964 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-56d8464d7c-vpwjc"]
Oct 31 01:04:37.775851 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:04:37.775818    4964 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ff4dcba2-df8d-4324-9e0c-abb23a5e6982" path="/var/lib/kubelet/pods/ff4dcba2-df8d-4324-9e0c-abb23a5e6982/volumes"
Oct 31 01:04:38.000383 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:04:38.000343    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:04:38.325300 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:04:38.325268    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:04:38.696864 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:04:38.696741    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:04:38.943478 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:04:38.943447    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:04:40.301128 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:04:40.301096    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:04:40.491766 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:04:40.491735    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:04:41.034799 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:04:41.034770    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:04:41.237269 ip-10-0-133-197 kubenswrapper[4964]: E1031 01:04:41.237236    4964 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q56x5" containerName="extractor"
Oct 31 01:04:41.254192 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:04:41.254163    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:04:41.473013 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:04:41.472973    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:04:45.248759 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:04:45.248704    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:04:46.249136 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:04:46.249089    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:04:46.423598 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:04:46.423567    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:04:53.001622 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:04:53.001586    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:04:53.197987 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:04:53.197935    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:04:53.646724 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:04:53.646682    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:04:53.925954 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:04:53.925847    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:04:56.154619 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:04:56.154585    4964 generic.go:356] "Generic (PLEG): container finished" podID="c8e98df5-b2f7-49dd-a565-ed60534da496" containerID="84ec5dc0d3ef6b0dc04ef56239f35a37a96800c7a0abc2dd95835fd6cd612524" exitCode=0
Oct 31 01:04:56.155016 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:04:56.154661    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-jkk5x" event={"ID":"c8e98df5-b2f7-49dd-a565-ed60534da496","Type":"ContainerDied","Data":"84ec5dc0d3ef6b0dc04ef56239f35a37a96800c7a0abc2dd95835fd6cd612524"}
Oct 31 01:04:56.155016 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:04:56.154983    4964 scope.go:122] "RemoveContainer" containerID="84ec5dc0d3ef6b0dc04ef56239f35a37a96800c7a0abc2dd95835fd6cd612524"
Oct 31 01:04:57.160474 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:04:57.160435    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-jkk5x" event={"ID":"c8e98df5-b2f7-49dd-a565-ed60534da496","Type":"ContainerStarted","Data":"49dfd8ee4f1271dd96486b384daad969679caa943f0ec1e93c3c84337c72ba0d"}
Oct 31 01:04:57.182648 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:04:57.182619    4964 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-insights/periodic-gathering-b2z6s-vxz9h"]
Oct 31 01:04:57.182852 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:04:57.182830    4964 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-insights/periodic-gathering-b2z6s-vxz9h" podUID="9122c2fd-f29f-4e74-9615-15078517a3c0" containerName="insights-gathering" containerID="cri-o://9532152a746063f8ad7d07b9839b23f02eedf132bfc3900c57a3d6cf1f9597ec" gracePeriod=30
Oct 31 01:04:57.429535 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:04:57.429513    4964 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-b2z6s-vxz9h"
Oct 31 01:04:57.460005 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:04:57.459970    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9122c2fd-f29f-4e74-9615-15078517a3c0-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9122c2fd-f29f-4e74-9615-15078517a3c0-trusted-ca-bundle\") pod \"9122c2fd-f29f-4e74-9615-15078517a3c0\" (UID: \"9122c2fd-f29f-4e74-9615-15078517a3c0\") "
Oct 31 01:04:57.460170 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:04:57.460089    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9122c2fd-f29f-4e74-9615-15078517a3c0-kube-api-access-r4l7n\" (UniqueName: \"kubernetes.io/projected/9122c2fd-f29f-4e74-9615-15078517a3c0-kube-api-access-r4l7n\") pod \"9122c2fd-f29f-4e74-9615-15078517a3c0\" (UID: \"9122c2fd-f29f-4e74-9615-15078517a3c0\") "
Oct 31 01:04:57.460170 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:04:57.460146    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9122c2fd-f29f-4e74-9615-15078517a3c0-archives-path\" (UniqueName: \"kubernetes.io/empty-dir/9122c2fd-f29f-4e74-9615-15078517a3c0-archives-path\") pod \"9122c2fd-f29f-4e74-9615-15078517a3c0\" (UID: \"9122c2fd-f29f-4e74-9615-15078517a3c0\") "
Oct 31 01:04:57.460286 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:04:57.460179    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9122c2fd-f29f-4e74-9615-15078517a3c0-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9122c2fd-f29f-4e74-9615-15078517a3c0-service-ca-bundle\") pod \"9122c2fd-f29f-4e74-9615-15078517a3c0\" (UID: \"9122c2fd-f29f-4e74-9615-15078517a3c0\") "
Oct 31 01:04:57.460502 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:04:57.460384    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9122c2fd-f29f-4e74-9615-15078517a3c0-trusted-ca-bundle" pod "9122c2fd-f29f-4e74-9615-15078517a3c0" (UID: "9122c2fd-f29f-4e74-9615-15078517a3c0"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:04:57.460575 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:04:57.460518    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9122c2fd-f29f-4e74-9615-15078517a3c0-archives-path" pod "9122c2fd-f29f-4e74-9615-15078517a3c0" (UID: "9122c2fd-f29f-4e74-9615-15078517a3c0"). InnerVolumeSpecName "archives-path". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:04:57.460626 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:04:57.460575    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9122c2fd-f29f-4e74-9615-15078517a3c0-service-ca-bundle" pod "9122c2fd-f29f-4e74-9615-15078517a3c0" (UID: "9122c2fd-f29f-4e74-9615-15078517a3c0"). InnerVolumeSpecName "service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:04:57.462616 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:04:57.462593    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9122c2fd-f29f-4e74-9615-15078517a3c0-kube-api-access-r4l7n" pod "9122c2fd-f29f-4e74-9615-15078517a3c0" (UID: "9122c2fd-f29f-4e74-9615-15078517a3c0"). InnerVolumeSpecName "kube-api-access-r4l7n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:04:57.561799 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:04:57.561763    4964 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9122c2fd-f29f-4e74-9615-15078517a3c0-trusted-ca-bundle\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:04:57.561961 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:04:57.561803    4964 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r4l7n\" (UniqueName: \"kubernetes.io/projected/9122c2fd-f29f-4e74-9615-15078517a3c0-kube-api-access-r4l7n\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:04:57.561961 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:04:57.561823    4964 reconciler_common.go:299] "Volume detached for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/9122c2fd-f29f-4e74-9615-15078517a3c0-archives-path\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:04:57.561961 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:04:57.561836    4964 reconciler_common.go:299] "Volume detached for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9122c2fd-f29f-4e74-9615-15078517a3c0-service-ca-bundle\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:04:57.636588 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:04:57.636555    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:04:57.825978 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:04:57.825944    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:04:58.165860 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:04:58.165774    4964 generic.go:356] "Generic (PLEG): container finished" podID="9122c2fd-f29f-4e74-9615-15078517a3c0" containerID="9532152a746063f8ad7d07b9839b23f02eedf132bfc3900c57a3d6cf1f9597ec" exitCode=2
Oct 31 01:04:58.165860 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:04:58.165822    4964 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-b2z6s-vxz9h"
Oct 31 01:04:58.166343 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:04:58.165858    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-b2z6s-vxz9h" event={"ID":"9122c2fd-f29f-4e74-9615-15078517a3c0","Type":"ContainerDied","Data":"9532152a746063f8ad7d07b9839b23f02eedf132bfc3900c57a3d6cf1f9597ec"}
Oct 31 01:04:58.166343 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:04:58.165903    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-b2z6s-vxz9h" event={"ID":"9122c2fd-f29f-4e74-9615-15078517a3c0","Type":"ContainerDied","Data":"91c64ba91f2d616e39faaef286d3e721f03f6ac5c7d010d6b674016160e7aa6e"}
Oct 31 01:04:58.166343 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:04:58.165925    4964 scope.go:122] "RemoveContainer" containerID="9532152a746063f8ad7d07b9839b23f02eedf132bfc3900c57a3d6cf1f9597ec"
Oct 31 01:04:58.174083 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:04:58.174067    4964 scope.go:122] "RemoveContainer" containerID="9532152a746063f8ad7d07b9839b23f02eedf132bfc3900c57a3d6cf1f9597ec"
Oct 31 01:04:58.174345 ip-10-0-133-197 kubenswrapper[4964]: E1031 01:04:58.174308    4964 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9532152a746063f8ad7d07b9839b23f02eedf132bfc3900c57a3d6cf1f9597ec\": container with ID starting with 9532152a746063f8ad7d07b9839b23f02eedf132bfc3900c57a3d6cf1f9597ec not found: ID does not exist" containerID="9532152a746063f8ad7d07b9839b23f02eedf132bfc3900c57a3d6cf1f9597ec"
Oct 31 01:04:58.174399 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:04:58.174354    4964 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9532152a746063f8ad7d07b9839b23f02eedf132bfc3900c57a3d6cf1f9597ec"} err="failed to get container status \"9532152a746063f8ad7d07b9839b23f02eedf132bfc3900c57a3d6cf1f9597ec\": rpc error: code = NotFound desc = could not find container \"9532152a746063f8ad7d07b9839b23f02eedf132bfc3900c57a3d6cf1f9597ec\": container with ID starting with 9532152a746063f8ad7d07b9839b23f02eedf132bfc3900c57a3d6cf1f9597ec not found: ID does not exist"
Oct 31 01:04:58.332837 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:04:58.332805    4964 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-insights/periodic-gathering-b2z6s-vxz9h"]
Oct 31 01:04:58.382276 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:04:58.382239    4964 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-insights/periodic-gathering-b2z6s-vxz9h"]
Oct 31 01:04:59.774476 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:04:59.774446    4964 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9122c2fd-f29f-4e74-9615-15078517a3c0" path="/var/lib/kubelet/pods/9122c2fd-f29f-4e74-9615-15078517a3c0/volumes"
Oct 31 01:05:00.224140 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:05:00.224106    4964 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-image-registry/image-registry-75575b5db8-wtq4d" podUID="aa112079-b749-404c-9081-113220707fcd" containerName="registry" containerID="cri-o://633d26971872c25b8ea849d33f1ef31d6c1ae5fb02e89c28a0c19568874f0f1c" gracePeriod=30
Oct 31 01:05:00.371488 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:05:00.371429    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:05:00.493898 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:05:00.493870    4964 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-75575b5db8-wtq4d"
Oct 31 01:05:00.567014 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:05:00.566986    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:05:00.587987 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:05:00.587961    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aa112079-b749-404c-9081-113220707fcd-registry-certificates\" (UniqueName: \"kubernetes.io/configmap/aa112079-b749-404c-9081-113220707fcd-registry-certificates\") pod \"aa112079-b749-404c-9081-113220707fcd\" (UID: \"aa112079-b749-404c-9081-113220707fcd\") "
Oct 31 01:05:00.588141 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:05:00.588004    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aa112079-b749-404c-9081-113220707fcd-ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/aa112079-b749-404c-9081-113220707fcd-ca-trust-extracted\") pod \"aa112079-b749-404c-9081-113220707fcd\" (UID: \"aa112079-b749-404c-9081-113220707fcd\") "
Oct 31 01:05:00.588141 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:05:00.588048    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aa112079-b749-404c-9081-113220707fcd-registry-tls\" (UniqueName: \"kubernetes.io/projected/aa112079-b749-404c-9081-113220707fcd-registry-tls\") pod \"aa112079-b749-404c-9081-113220707fcd\" (UID: \"aa112079-b749-404c-9081-113220707fcd\") "
Oct 31 01:05:00.588273 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:05:00.588146    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aa112079-b749-404c-9081-113220707fcd-bound-sa-token\" (UniqueName: \"kubernetes.io/projected/aa112079-b749-404c-9081-113220707fcd-bound-sa-token\") pod \"aa112079-b749-404c-9081-113220707fcd\" (UID: \"aa112079-b749-404c-9081-113220707fcd\") "
Oct 31 01:05:00.588273 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:05:00.588206    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/aa112079-b749-404c-9081-113220707fcd-installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/aa112079-b749-404c-9081-113220707fcd-installation-pull-secrets\") pod \"aa112079-b749-404c-9081-113220707fcd\" (UID: \"aa112079-b749-404c-9081-113220707fcd\") "
Oct 31 01:05:00.588273 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:05:00.588249    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aa112079-b749-404c-9081-113220707fcd-trusted-ca\" (UniqueName: \"kubernetes.io/configmap/aa112079-b749-404c-9081-113220707fcd-trusted-ca\") pod \"aa112079-b749-404c-9081-113220707fcd\" (UID: \"aa112079-b749-404c-9081-113220707fcd\") "
Oct 31 01:05:00.588500 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:05:00.588479    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aa112079-b749-404c-9081-113220707fcd-registry-certificates" pod "aa112079-b749-404c-9081-113220707fcd" (UID: "aa112079-b749-404c-9081-113220707fcd"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:05:00.588644 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:05:00.588623    4964 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/aa112079-b749-404c-9081-113220707fcd-registry-certificates\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:05:00.588770 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:05:00.588743    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aa112079-b749-404c-9081-113220707fcd-trusted-ca" pod "aa112079-b749-404c-9081-113220707fcd" (UID: "aa112079-b749-404c-9081-113220707fcd"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:05:00.591085 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:05:00.591059    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa112079-b749-404c-9081-113220707fcd-registry-tls" pod "aa112079-b749-404c-9081-113220707fcd" (UID: "aa112079-b749-404c-9081-113220707fcd"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:05:00.591201 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:05:00.591169    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aa112079-b749-404c-9081-113220707fcd-installation-pull-secrets" pod "aa112079-b749-404c-9081-113220707fcd" (UID: "aa112079-b749-404c-9081-113220707fcd"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:05:00.591435 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:05:00.591415    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa112079-b749-404c-9081-113220707fcd-bound-sa-token" pod "aa112079-b749-404c-9081-113220707fcd" (UID: "aa112079-b749-404c-9081-113220707fcd"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:05:00.594690 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:05:00.594665    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa112079-b749-404c-9081-113220707fcd-ca-trust-extracted" pod "aa112079-b749-404c-9081-113220707fcd" (UID: "aa112079-b749-404c-9081-113220707fcd"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:05:00.688854 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:05:00.688817    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/aa112079-b749-404c-9081-113220707fcd-image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/aa112079-b749-404c-9081-113220707fcd-image-registry-private-configuration\") pod \"aa112079-b749-404c-9081-113220707fcd\" (UID: \"aa112079-b749-404c-9081-113220707fcd\") "
Oct 31 01:05:00.689049 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:05:00.688862    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aa112079-b749-404c-9081-113220707fcd-kube-api-access-rpmxj\" (UniqueName: \"kubernetes.io/projected/aa112079-b749-404c-9081-113220707fcd-kube-api-access-rpmxj\") pod \"aa112079-b749-404c-9081-113220707fcd\" (UID: \"aa112079-b749-404c-9081-113220707fcd\") "
Oct 31 01:05:00.689120 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:05:00.689069    4964 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/aa112079-b749-404c-9081-113220707fcd-bound-sa-token\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:05:00.689120 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:05:00.689091    4964 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/aa112079-b749-404c-9081-113220707fcd-installation-pull-secrets\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:05:00.689120 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:05:00.689110    4964 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/aa112079-b749-404c-9081-113220707fcd-trusted-ca\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:05:00.689234 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:05:00.689124    4964 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/aa112079-b749-404c-9081-113220707fcd-ca-trust-extracted\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:05:00.689234 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:05:00.689139    4964 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/aa112079-b749-404c-9081-113220707fcd-registry-tls\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:05:00.691076 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:05:00.691050    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aa112079-b749-404c-9081-113220707fcd-image-registry-private-configuration" pod "aa112079-b749-404c-9081-113220707fcd" (UID: "aa112079-b749-404c-9081-113220707fcd"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:05:00.691182 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:05:00.691162    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa112079-b749-404c-9081-113220707fcd-kube-api-access-rpmxj" pod "aa112079-b749-404c-9081-113220707fcd" (UID: "aa112079-b749-404c-9081-113220707fcd"). InnerVolumeSpecName "kube-api-access-rpmxj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:05:00.742108 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:05:00.742006    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:05:00.789484 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:05:00.789454    4964 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/aa112079-b749-404c-9081-113220707fcd-image-registry-private-configuration\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:05:00.789484 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:05:00.789482    4964 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rpmxj\" (UniqueName: \"kubernetes.io/projected/aa112079-b749-404c-9081-113220707fcd-kube-api-access-rpmxj\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:05:01.181025 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:05:01.180949    4964 generic.go:356] "Generic (PLEG): container finished" podID="aa112079-b749-404c-9081-113220707fcd" containerID="633d26971872c25b8ea849d33f1ef31d6c1ae5fb02e89c28a0c19568874f0f1c" exitCode=0
Oct 31 01:05:01.181025 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:05:01.181000    4964 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-75575b5db8-wtq4d"
Oct 31 01:05:01.181211 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:05:01.181004    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-75575b5db8-wtq4d" event={"ID":"aa112079-b749-404c-9081-113220707fcd","Type":"ContainerDied","Data":"633d26971872c25b8ea849d33f1ef31d6c1ae5fb02e89c28a0c19568874f0f1c"}
Oct 31 01:05:01.181211 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:05:01.181103    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-75575b5db8-wtq4d" event={"ID":"aa112079-b749-404c-9081-113220707fcd","Type":"ContainerDied","Data":"6061e137a9ba52d29809eef437d64849764736661809b0de896d4eff81760b69"}
Oct 31 01:05:01.181211 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:05:01.181121    4964 scope.go:122] "RemoveContainer" containerID="633d26971872c25b8ea849d33f1ef31d6c1ae5fb02e89c28a0c19568874f0f1c"
Oct 31 01:05:01.189883 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:05:01.189867    4964 scope.go:122] "RemoveContainer" containerID="633d26971872c25b8ea849d33f1ef31d6c1ae5fb02e89c28a0c19568874f0f1c"
Oct 31 01:05:01.190138 ip-10-0-133-197 kubenswrapper[4964]: E1031 01:05:01.190120    4964 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"633d26971872c25b8ea849d33f1ef31d6c1ae5fb02e89c28a0c19568874f0f1c\": container with ID starting with 633d26971872c25b8ea849d33f1ef31d6c1ae5fb02e89c28a0c19568874f0f1c not found: ID does not exist" containerID="633d26971872c25b8ea849d33f1ef31d6c1ae5fb02e89c28a0c19568874f0f1c"
Oct 31 01:05:01.190186 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:05:01.190148    4964 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"633d26971872c25b8ea849d33f1ef31d6c1ae5fb02e89c28a0c19568874f0f1c"} err="failed to get container status \"633d26971872c25b8ea849d33f1ef31d6c1ae5fb02e89c28a0c19568874f0f1c\": rpc error: code = NotFound desc = could not find container \"633d26971872c25b8ea849d33f1ef31d6c1ae5fb02e89c28a0c19568874f0f1c\": container with ID starting with 633d26971872c25b8ea849d33f1ef31d6c1ae5fb02e89c28a0c19568874f0f1c not found: ID does not exist"
Oct 31 01:05:01.392927 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:05:01.392895    4964 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-75575b5db8-wtq4d"]
Oct 31 01:05:01.442739 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:05:01.442669    4964 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-75575b5db8-wtq4d"]
Oct 31 01:05:01.774263 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:05:01.774231    4964 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aa112079-b749-404c-9081-113220707fcd" path="/var/lib/kubelet/pods/aa112079-b749-404c-9081-113220707fcd/volumes"
Oct 31 01:05:02.187179 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:05:02.187108    4964 generic.go:356] "Generic (PLEG): container finished" podID="e54a9dc6-36e6-4f1b-9c3a-623a520e11da" containerID="2901f432a0e6351847808a4a8fe2e646d08659e910d2b351767f477cbde70b75" exitCode=0
Oct 31 01:05:02.187179 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:05:02.187175    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-pdzj4" event={"ID":"e54a9dc6-36e6-4f1b-9c3a-623a520e11da","Type":"ContainerDied","Data":"2901f432a0e6351847808a4a8fe2e646d08659e910d2b351767f477cbde70b75"}
Oct 31 01:05:02.187566 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:05:02.187426    4964 scope.go:122] "RemoveContainer" containerID="2901f432a0e6351847808a4a8fe2e646d08659e910d2b351767f477cbde70b75"
Oct 31 01:05:03.192697 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:05:03.192669    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-pdzj4" event={"ID":"e54a9dc6-36e6-4f1b-9c3a-623a520e11da","Type":"ContainerStarted","Data":"d88338524c21ebd1007d3889a46e907283204d42661f3cfa37d4bdfd0efbf1d6"}
Oct 31 01:05:03.248209 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:05:03.248181    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:05:03.491311 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:05:03.491279    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:05:06.293846 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:05:06.293810    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:05:06.526736 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:05:06.526705    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:05:11.236654 ip-10-0-133-197 kubenswrapper[4964]: E1031 01:05:11.236621    4964 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q56x5" containerName="extractor"
Oct 31 01:05:12.229677 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:05:12.229645    4964 generic.go:356] "Generic (PLEG): container finished" podID="c2be658b-ec92-4782-a78b-d2dccf2249cb" containerID="2fa284236f6fb4b670ac25d5b32b5e19dea2607e333f6376be9cd9cffac81ef5" exitCode=0
Oct 31 01:05:12.229820 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:05:12.229725    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-n5q9v" event={"ID":"c2be658b-ec92-4782-a78b-d2dccf2249cb","Type":"ContainerDied","Data":"2fa284236f6fb4b670ac25d5b32b5e19dea2607e333f6376be9cd9cffac81ef5"}
Oct 31 01:05:12.230056 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:05:12.230040    4964 scope.go:122] "RemoveContainer" containerID="2fa284236f6fb4b670ac25d5b32b5e19dea2607e333f6376be9cd9cffac81ef5"
Oct 31 01:05:13.234838 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:05:13.234805    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-n5q9v" event={"ID":"c2be658b-ec92-4782-a78b-d2dccf2249cb","Type":"ContainerStarted","Data":"66bcadcdaa30f38d5ce4df43c0bb2f66b02ba2fdd9a6090b9768258454426f8e"}
Oct 31 01:05:22.994556 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:05:22.994515    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:05:23.166661 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:05:23.166624    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:05:41.236124 ip-10-0-133-197 kubenswrapper[4964]: E1031 01:05:41.236086    4964 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q56x5" containerName="extractor"
Oct 31 01:05:41.366193 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:05:41.366155    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:05:42.928582 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:05:42.928527    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:05:43.484754 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:05:43.484719    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:06:11.236431 ip-10-0-133-197 kubenswrapper[4964]: E1031 01:06:11.236354    4964 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q56x5" containerName="extractor"
Oct 31 01:06:41.235921 ip-10-0-133-197 kubenswrapper[4964]: E1031 01:06:41.235886    4964 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q56x5" containerName="extractor"
Oct 31 01:06:41.728851 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:06:41.728822    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:06:54.053909 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:06:54.053872    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:07:11.236076 ip-10-0-133-197 kubenswrapper[4964]: E1031 01:07:11.236041    4964 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q56x5" containerName="extractor"
Oct 31 01:07:16.719881 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:07:16.719843    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:07:16.942225 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:07:16.942189    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:07:18.445794 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:07:18.445759    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:07:18.685858 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:07:18.685812    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:07:19.294917 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:07:19.294880    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:07:19.540062 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:07:19.540026    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:07:19.717780 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:07:19.717687    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:07:22.089387 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:07:22.089345    4964 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6b6qrk"]
Oct 31 01:07:22.089763 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:07:22.089663    4964 cpu_manager.go:414] "containerMap: removing container" podUID="aa112079-b749-404c-9081-113220707fcd" containerName="registry"
Oct 31 01:07:22.089763 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:07:22.089673    4964 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aa112079-b749-404c-9081-113220707fcd" containerName="registry"
Oct 31 01:07:22.089763 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:07:22.089688    4964 cpu_manager.go:414] "containerMap: removing container" podUID="9122c2fd-f29f-4e74-9615-15078517a3c0" containerName="insights-gathering"
Oct 31 01:07:22.089763 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:07:22.089692    4964 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9122c2fd-f29f-4e74-9615-15078517a3c0" containerName="insights-gathering"
Oct 31 01:07:22.089763 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:07:22.089704    4964 cpu_manager.go:414] "containerMap: removing container" podUID="ff4dcba2-df8d-4324-9e0c-abb23a5e6982" containerName="registry"
Oct 31 01:07:22.089763 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:07:22.089709    4964 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ff4dcba2-df8d-4324-9e0c-abb23a5e6982" containerName="registry"
Oct 31 01:07:22.089763 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:07:22.089764    4964 memory_manager.go:365] "RemoveStaleState removing state" podUID="aa112079-b749-404c-9081-113220707fcd" containerName="registry"
Oct 31 01:07:22.089972 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:07:22.089772    4964 memory_manager.go:365] "RemoveStaleState removing state" podUID="9122c2fd-f29f-4e74-9615-15078517a3c0" containerName="insights-gathering"
Oct 31 01:07:22.089972 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:07:22.089778    4964 memory_manager.go:365] "RemoveStaleState removing state" podUID="ff4dcba2-df8d-4324-9e0c-abb23a5e6982" containerName="registry"
Oct 31 01:07:22.092996 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:07:22.092979    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6b6qrk"
Oct 31 01:07:22.194102 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:07:22.194071    4964 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Oct 31 01:07:22.194102 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:07:22.194074    4964 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Oct 31 01:07:22.194347 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:07:22.194181    4964 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3-b83c8cba-dockercfg-hjtff\""
Oct 31 01:07:22.195435 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:07:22.195412    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6b6qrk"]
Oct 31 01:07:22.196239 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:07:22.196222    4964 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ap562g"]
Oct 31 01:07:22.199549 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:07:22.199534    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ap562g"
Oct 31 01:07:22.218604 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:07:22.218578    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5329a60f-784d-4486-83ad-0013d9f19254-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6b6qrk\" (UID: \"5329a60f-784d-4486-83ad-0013d9f19254\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6b6qrk"
Oct 31 01:07:22.218743 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:07:22.218621    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tzn2b\" (UniqueName: \"kubernetes.io/projected/5329a60f-784d-4486-83ad-0013d9f19254-kube-api-access-tzn2b\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6b6qrk\" (UID: \"5329a60f-784d-4486-83ad-0013d9f19254\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6b6qrk"
Oct 31 01:07:22.218743 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:07:22.218646    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5329a60f-784d-4486-83ad-0013d9f19254-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6b6qrk\" (UID: \"5329a60f-784d-4486-83ad-0013d9f19254\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6b6qrk"
Oct 31 01:07:22.248141 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:07:22.248116    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ap562g"]
Oct 31 01:07:22.251499 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:07:22.251477    4964 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"6541d93aaee335d6227a8778ab4e840bbe7f52-f98fbcf8-dockercfg-8f5dt\""
Oct 31 01:07:22.320050 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:07:22.320021    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5329a60f-784d-4486-83ad-0013d9f19254-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6b6qrk\" (UID: \"5329a60f-784d-4486-83ad-0013d9f19254\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6b6qrk"
Oct 31 01:07:22.320237 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:07:22.320078    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5329a60f-784d-4486-83ad-0013d9f19254-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6b6qrk\" (UID: \"5329a60f-784d-4486-83ad-0013d9f19254\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6b6qrk"
Oct 31 01:07:22.320237 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:07:22.320103    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d131714a-324f-4312-8b7a-ff717920829e-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ap562g\" (UID: \"d131714a-324f-4312-8b7a-ff717920829e\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ap562g"
Oct 31 01:07:22.320237 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:07:22.320120    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d131714a-324f-4312-8b7a-ff717920829e-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ap562g\" (UID: \"d131714a-324f-4312-8b7a-ff717920829e\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ap562g"
Oct 31 01:07:22.320237 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:07:22.320155    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jwlmb\" (UniqueName: \"kubernetes.io/projected/d131714a-324f-4312-8b7a-ff717920829e-kube-api-access-jwlmb\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ap562g\" (UID: \"d131714a-324f-4312-8b7a-ff717920829e\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ap562g"
Oct 31 01:07:22.320420 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:07:22.320273    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tzn2b\" (UniqueName: \"kubernetes.io/projected/5329a60f-784d-4486-83ad-0013d9f19254-kube-api-access-tzn2b\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6b6qrk\" (UID: \"5329a60f-784d-4486-83ad-0013d9f19254\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6b6qrk"
Oct 31 01:07:22.320469 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:07:22.320451    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5329a60f-784d-4486-83ad-0013d9f19254-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6b6qrk\" (UID: \"5329a60f-784d-4486-83ad-0013d9f19254\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6b6qrk"
Oct 31 01:07:22.320510 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:07:22.320461    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5329a60f-784d-4486-83ad-0013d9f19254-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6b6qrk\" (UID: \"5329a60f-784d-4486-83ad-0013d9f19254\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6b6qrk"
Oct 31 01:07:22.376831 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:07:22.376755    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tzn2b\" (UniqueName: \"kubernetes.io/projected/5329a60f-784d-4486-83ad-0013d9f19254-kube-api-access-tzn2b\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6b6qrk\" (UID: \"5329a60f-784d-4486-83ad-0013d9f19254\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6b6qrk"
Oct 31 01:07:22.408761 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:07:22.408728    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6b6qrk"
Oct 31 01:07:22.420970 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:07:22.420945    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d131714a-324f-4312-8b7a-ff717920829e-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ap562g\" (UID: \"d131714a-324f-4312-8b7a-ff717920829e\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ap562g"
Oct 31 01:07:22.421102 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:07:22.420983    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d131714a-324f-4312-8b7a-ff717920829e-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ap562g\" (UID: \"d131714a-324f-4312-8b7a-ff717920829e\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ap562g"
Oct 31 01:07:22.421102 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:07:22.421017    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jwlmb\" (UniqueName: \"kubernetes.io/projected/d131714a-324f-4312-8b7a-ff717920829e-kube-api-access-jwlmb\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ap562g\" (UID: \"d131714a-324f-4312-8b7a-ff717920829e\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ap562g"
Oct 31 01:07:22.421405 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:07:22.421382    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d131714a-324f-4312-8b7a-ff717920829e-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ap562g\" (UID: \"d131714a-324f-4312-8b7a-ff717920829e\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ap562g"
Oct 31 01:07:22.421405 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:07:22.421396    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d131714a-324f-4312-8b7a-ff717920829e-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ap562g\" (UID: \"d131714a-324f-4312-8b7a-ff717920829e\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ap562g"
Oct 31 01:07:22.478041 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:07:22.478007    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jwlmb\" (UniqueName: \"kubernetes.io/projected/d131714a-324f-4312-8b7a-ff717920829e-kube-api-access-jwlmb\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ap562g\" (UID: \"d131714a-324f-4312-8b7a-ff717920829e\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ap562g"
Oct 31 01:07:22.515021 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:07:22.514989    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ap562g"
Oct 31 01:07:22.783305 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:07:22.783239    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6b6qrk"]
Oct 31 01:07:22.785979 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:07:22.785954    4964 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5329a60f_784d_4486_83ad_0013d9f19254.slice/crio-d905a48adfc78a4365a95aa60ec99a9215837e8806b15fcdf726d6f8250ab2d5 WatchSource:0}: Error finding container d905a48adfc78a4365a95aa60ec99a9215837e8806b15fcdf726d6f8250ab2d5: Status 404 returned error can't find the container with id d905a48adfc78a4365a95aa60ec99a9215837e8806b15fcdf726d6f8250ab2d5
Oct 31 01:07:22.896978 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:07:22.896943    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ap562g"]
Oct 31 01:07:22.900784 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:07:22.900755    4964 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd131714a_324f_4312_8b7a_ff717920829e.slice/crio-7b4ddd588f69a8c73174c9cc193f84be8301f3d614c305f3e20405bbe065a96a WatchSource:0}: Error finding container 7b4ddd588f69a8c73174c9cc193f84be8301f3d614c305f3e20405bbe065a96a: Status 404 returned error can't find the container with id 7b4ddd588f69a8c73174c9cc193f84be8301f3d614c305f3e20405bbe065a96a
Oct 31 01:07:23.733690 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:07:23.733652    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ap562g" event={"ID":"d131714a-324f-4312-8b7a-ff717920829e","Type":"ContainerStarted","Data":"7b4ddd588f69a8c73174c9cc193f84be8301f3d614c305f3e20405bbe065a96a"}
Oct 31 01:07:23.734989 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:07:23.734960    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6b6qrk" event={"ID":"5329a60f-784d-4486-83ad-0013d9f19254","Type":"ContainerStarted","Data":"d905a48adfc78a4365a95aa60ec99a9215837e8806b15fcdf726d6f8250ab2d5"}
Oct 31 01:07:24.179072 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:07:24.178993    4964 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be717dwcr"]
Oct 31 01:07:24.183054 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:07:24.183028    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be717dwcr"
Oct 31 01:07:24.234721 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:07:24.234589    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/aa9df77e-e1a0-4084-9834-3cd7244cdc66-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be717dwcr\" (UID: \"aa9df77e-e1a0-4084-9834-3cd7244cdc66\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be717dwcr"
Oct 31 01:07:24.234994 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:07:24.234718    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/aa9df77e-e1a0-4084-9834-3cd7244cdc66-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be717dwcr\" (UID: \"aa9df77e-e1a0-4084-9834-3cd7244cdc66\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be717dwcr"
Oct 31 01:07:24.234994 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:07:24.234774    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-79dzf\" (UniqueName: \"kubernetes.io/projected/aa9df77e-e1a0-4084-9834-3cd7244cdc66-kube-api-access-79dzf\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be717dwcr\" (UID: \"aa9df77e-e1a0-4084-9834-3cd7244cdc66\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be717dwcr"
Oct 31 01:07:24.284815 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:07:24.284769    4964 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"e644421527c169d876fdb59235c927986d6bd2-9d2815fc-dockercfg-dwxgt\""
Oct 31 01:07:24.286650 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:07:24.286618    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be717dwcr"]
Oct 31 01:07:24.287759 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:07:24.287740    4964 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cwtrkb"]
Oct 31 01:07:24.292496 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:07:24.292464    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cwtrkb"
Oct 31 01:07:24.335781 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:07:24.335741    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/aa9df77e-e1a0-4084-9834-3cd7244cdc66-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be717dwcr\" (UID: \"aa9df77e-e1a0-4084-9834-3cd7244cdc66\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be717dwcr"
Oct 31 01:07:24.335984 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:07:24.335792    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/aa9df77e-e1a0-4084-9834-3cd7244cdc66-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be717dwcr\" (UID: \"aa9df77e-e1a0-4084-9834-3cd7244cdc66\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be717dwcr"
Oct 31 01:07:24.335984 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:07:24.335928    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-79dzf\" (UniqueName: \"kubernetes.io/projected/aa9df77e-e1a0-4084-9834-3cd7244cdc66-kube-api-access-79dzf\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be717dwcr\" (UID: \"aa9df77e-e1a0-4084-9834-3cd7244cdc66\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be717dwcr"
Oct 31 01:07:24.335984 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:07:24.335973    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sxstv\" (UniqueName: \"kubernetes.io/projected/83b2207c-b598-4a1c-bce5-636e7cf8f970-kube-api-access-sxstv\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cwtrkb\" (UID: \"83b2207c-b598-4a1c-bce5-636e7cf8f970\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cwtrkb"
Oct 31 01:07:24.336153 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:07:24.336057    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/83b2207c-b598-4a1c-bce5-636e7cf8f970-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cwtrkb\" (UID: \"83b2207c-b598-4a1c-bce5-636e7cf8f970\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cwtrkb"
Oct 31 01:07:24.336153 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:07:24.336084    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/83b2207c-b598-4a1c-bce5-636e7cf8f970-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cwtrkb\" (UID: \"83b2207c-b598-4a1c-bce5-636e7cf8f970\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cwtrkb"
Oct 31 01:07:24.336248 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:07:24.336199    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/aa9df77e-e1a0-4084-9834-3cd7244cdc66-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be717dwcr\" (UID: \"aa9df77e-e1a0-4084-9834-3cd7244cdc66\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be717dwcr"
Oct 31 01:07:24.336248 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:07:24.336209    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/aa9df77e-e1a0-4084-9834-3cd7244cdc66-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be717dwcr\" (UID: \"aa9df77e-e1a0-4084-9834-3cd7244cdc66\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be717dwcr"
Oct 31 01:07:24.345030 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:07:24.344996    4964 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"c7537007944577f64df34960ace3c69136c416-139ec370-dockercfg-5xqz2\""
Oct 31 01:07:24.354265 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:07:24.354235    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cwtrkb"]
Oct 31 01:07:24.395836 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:07:24.395798    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-79dzf\" (UniqueName: \"kubernetes.io/projected/aa9df77e-e1a0-4084-9834-3cd7244cdc66-kube-api-access-79dzf\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be717dwcr\" (UID: \"aa9df77e-e1a0-4084-9834-3cd7244cdc66\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be717dwcr"
Oct 31 01:07:24.437409 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:07:24.437298    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sxstv\" (UniqueName: \"kubernetes.io/projected/83b2207c-b598-4a1c-bce5-636e7cf8f970-kube-api-access-sxstv\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cwtrkb\" (UID: \"83b2207c-b598-4a1c-bce5-636e7cf8f970\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cwtrkb"
Oct 31 01:07:24.437592 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:07:24.437415    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/83b2207c-b598-4a1c-bce5-636e7cf8f970-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cwtrkb\" (UID: \"83b2207c-b598-4a1c-bce5-636e7cf8f970\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cwtrkb"
Oct 31 01:07:24.437592 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:07:24.437440    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/83b2207c-b598-4a1c-bce5-636e7cf8f970-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cwtrkb\" (UID: \"83b2207c-b598-4a1c-bce5-636e7cf8f970\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cwtrkb"
Oct 31 01:07:24.437945 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:07:24.437917    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/83b2207c-b598-4a1c-bce5-636e7cf8f970-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cwtrkb\" (UID: \"83b2207c-b598-4a1c-bce5-636e7cf8f970\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cwtrkb"
Oct 31 01:07:24.438062 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:07:24.437917    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/83b2207c-b598-4a1c-bce5-636e7cf8f970-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cwtrkb\" (UID: \"83b2207c-b598-4a1c-bce5-636e7cf8f970\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cwtrkb"
Oct 31 01:07:24.496718 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:07:24.496669    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sxstv\" (UniqueName: \"kubernetes.io/projected/83b2207c-b598-4a1c-bce5-636e7cf8f970-kube-api-access-sxstv\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cwtrkb\" (UID: \"83b2207c-b598-4a1c-bce5-636e7cf8f970\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cwtrkb"
Oct 31 01:07:24.506611 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:07:24.506583    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be717dwcr"
Oct 31 01:07:24.614030 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:07:24.613997    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cwtrkb"
Oct 31 01:07:24.903232 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:07:24.903108    4964 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa9df77e_e1a0_4084_9834_3cd7244cdc66.slice/crio-3bdc336e31d6bf2b63df67875f46161606659009e0dd6c0758f737bf5e7aba2f WatchSource:0}: Error finding container 3bdc336e31d6bf2b63df67875f46161606659009e0dd6c0758f737bf5e7aba2f: Status 404 returned error can't find the container with id 3bdc336e31d6bf2b63df67875f46161606659009e0dd6c0758f737bf5e7aba2f
Oct 31 01:07:24.905160 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:07:24.905121    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be717dwcr"]
Oct 31 01:07:25.035917 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:07:25.035885    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cwtrkb"]
Oct 31 01:07:25.041952 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:07:25.041923    4964 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83b2207c_b598_4a1c_bce5_636e7cf8f970.slice/crio-f8c31db4e59a6488525c81259aa64fd5cf0f7922972d4f13aeb1ee7c6f4b5d53 WatchSource:0}: Error finding container f8c31db4e59a6488525c81259aa64fd5cf0f7922972d4f13aeb1ee7c6f4b5d53: Status 404 returned error can't find the container with id f8c31db4e59a6488525c81259aa64fd5cf0f7922972d4f13aeb1ee7c6f4b5d53
Oct 31 01:07:25.069748 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:07:25.069714    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:07:25.255389 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:07:25.255274    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:07:25.749520 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:07:25.749476    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cwtrkb" event={"ID":"83b2207c-b598-4a1c-bce5-636e7cf8f970","Type":"ContainerStarted","Data":"f8c31db4e59a6488525c81259aa64fd5cf0f7922972d4f13aeb1ee7c6f4b5d53"}
Oct 31 01:07:25.751097 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:07:25.751039    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be717dwcr" event={"ID":"aa9df77e-e1a0-4084-9834-3cd7244cdc66","Type":"ContainerStarted","Data":"3bdc336e31d6bf2b63df67875f46161606659009e0dd6c0758f737bf5e7aba2f"}
Oct 31 01:07:25.978601 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:07:25.978565    4964 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-jn9hq"]
Oct 31 01:07:25.982398 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:07:25.982370    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-jn9hq"
Oct 31 01:07:26.034260 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:07:26.034138    4964 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-bnzrd\""
Oct 31 01:07:26.052858 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:07:26.052788    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/23f1b6ef-5cf9-4d1a-afdd-75a0c723727c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-jn9hq\" (UID: \"23f1b6ef-5cf9-4d1a-afdd-75a0c723727c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-jn9hq"
Oct 31 01:07:26.052858 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:07:26.052857    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/23f1b6ef-5cf9-4d1a-afdd-75a0c723727c-workload-certs\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-jn9hq\" (UID: \"23f1b6ef-5cf9-4d1a-afdd-75a0c723727c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-jn9hq"
Oct 31 01:07:26.053207 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:07:26.052885    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/23f1b6ef-5cf9-4d1a-afdd-75a0c723727c-istio-data\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-jn9hq\" (UID: \"23f1b6ef-5cf9-4d1a-afdd-75a0c723727c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-jn9hq"
Oct 31 01:07:26.053207 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:07:26.052923    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/23f1b6ef-5cf9-4d1a-afdd-75a0c723727c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-jn9hq\" (UID: \"23f1b6ef-5cf9-4d1a-afdd-75a0c723727c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-jn9hq"
Oct 31 01:07:26.053207 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:07:26.052953    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5vj4m\" (UniqueName: \"kubernetes.io/projected/23f1b6ef-5cf9-4d1a-afdd-75a0c723727c-kube-api-access-5vj4m\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-jn9hq\" (UID: \"23f1b6ef-5cf9-4d1a-afdd-75a0c723727c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-jn9hq"
Oct 31 01:07:26.053207 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:07:26.052984    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/23f1b6ef-5cf9-4d1a-afdd-75a0c723727c-credential-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-jn9hq\" (UID: \"23f1b6ef-5cf9-4d1a-afdd-75a0c723727c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-jn9hq"
Oct 31 01:07:26.053207 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:07:26.053018    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/23f1b6ef-5cf9-4d1a-afdd-75a0c723727c-istio-envoy\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-jn9hq\" (UID: \"23f1b6ef-5cf9-4d1a-afdd-75a0c723727c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-jn9hq"
Oct 31 01:07:26.053207 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:07:26.053081    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/23f1b6ef-5cf9-4d1a-afdd-75a0c723727c-istio-token\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-jn9hq\" (UID: \"23f1b6ef-5cf9-4d1a-afdd-75a0c723727c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-jn9hq"
Oct 31 01:07:26.053207 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:07:26.053124    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/23f1b6ef-5cf9-4d1a-afdd-75a0c723727c-workload-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-jn9hq\" (UID: \"23f1b6ef-5cf9-4d1a-afdd-75a0c723727c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-jn9hq"
Oct 31 01:07:26.055941 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:07:26.055913    4964 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Oct 31 01:07:26.088472 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:07:26.087210    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-jn9hq"]
Oct 31 01:07:26.127751 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:07:26.127710    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:07:26.154788 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:07:26.154005    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/23f1b6ef-5cf9-4d1a-afdd-75a0c723727c-workload-certs\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-jn9hq\" (UID: \"23f1b6ef-5cf9-4d1a-afdd-75a0c723727c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-jn9hq"
Oct 31 01:07:26.154788 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:07:26.154049    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/23f1b6ef-5cf9-4d1a-afdd-75a0c723727c-istio-data\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-jn9hq\" (UID: \"23f1b6ef-5cf9-4d1a-afdd-75a0c723727c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-jn9hq"
Oct 31 01:07:26.154788 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:07:26.154091    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/23f1b6ef-5cf9-4d1a-afdd-75a0c723727c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-jn9hq\" (UID: \"23f1b6ef-5cf9-4d1a-afdd-75a0c723727c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-jn9hq"
Oct 31 01:07:26.154788 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:07:26.154119    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5vj4m\" (UniqueName: \"kubernetes.io/projected/23f1b6ef-5cf9-4d1a-afdd-75a0c723727c-kube-api-access-5vj4m\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-jn9hq\" (UID: \"23f1b6ef-5cf9-4d1a-afdd-75a0c723727c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-jn9hq"
Oct 31 01:07:26.154788 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:07:26.154136    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/23f1b6ef-5cf9-4d1a-afdd-75a0c723727c-credential-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-jn9hq\" (UID: \"23f1b6ef-5cf9-4d1a-afdd-75a0c723727c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-jn9hq"
Oct 31 01:07:26.154788 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:07:26.154167    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/23f1b6ef-5cf9-4d1a-afdd-75a0c723727c-istio-envoy\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-jn9hq\" (UID: \"23f1b6ef-5cf9-4d1a-afdd-75a0c723727c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-jn9hq"
Oct 31 01:07:26.154788 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:07:26.154227    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/23f1b6ef-5cf9-4d1a-afdd-75a0c723727c-istio-token\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-jn9hq\" (UID: \"23f1b6ef-5cf9-4d1a-afdd-75a0c723727c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-jn9hq"
Oct 31 01:07:26.154788 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:07:26.154265    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/23f1b6ef-5cf9-4d1a-afdd-75a0c723727c-workload-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-jn9hq\" (UID: \"23f1b6ef-5cf9-4d1a-afdd-75a0c723727c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-jn9hq"
Oct 31 01:07:26.154788 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:07:26.154302    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/23f1b6ef-5cf9-4d1a-afdd-75a0c723727c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-jn9hq\" (UID: \"23f1b6ef-5cf9-4d1a-afdd-75a0c723727c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-jn9hq"
Oct 31 01:07:26.154788 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:07:26.154490    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/23f1b6ef-5cf9-4d1a-afdd-75a0c723727c-workload-certs\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-jn9hq\" (UID: \"23f1b6ef-5cf9-4d1a-afdd-75a0c723727c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-jn9hq"
Oct 31 01:07:26.154788 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:07:26.154496    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/23f1b6ef-5cf9-4d1a-afdd-75a0c723727c-istio-data\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-jn9hq\" (UID: \"23f1b6ef-5cf9-4d1a-afdd-75a0c723727c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-jn9hq"
Oct 31 01:07:26.155519 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:07:26.155007    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/23f1b6ef-5cf9-4d1a-afdd-75a0c723727c-credential-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-jn9hq\" (UID: \"23f1b6ef-5cf9-4d1a-afdd-75a0c723727c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-jn9hq"
Oct 31 01:07:26.155519 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:07:26.155233    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/23f1b6ef-5cf9-4d1a-afdd-75a0c723727c-workload-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-jn9hq\" (UID: \"23f1b6ef-5cf9-4d1a-afdd-75a0c723727c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-jn9hq"
Oct 31 01:07:26.155519 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:07:26.155456    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/23f1b6ef-5cf9-4d1a-afdd-75a0c723727c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-jn9hq\" (UID: \"23f1b6ef-5cf9-4d1a-afdd-75a0c723727c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-jn9hq"
Oct 31 01:07:26.171785 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:07:26.169341    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/23f1b6ef-5cf9-4d1a-afdd-75a0c723727c-istio-envoy\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-jn9hq\" (UID: \"23f1b6ef-5cf9-4d1a-afdd-75a0c723727c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-jn9hq"
Oct 31 01:07:26.171785 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:07:26.169984    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/23f1b6ef-5cf9-4d1a-afdd-75a0c723727c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-jn9hq\" (UID: \"23f1b6ef-5cf9-4d1a-afdd-75a0c723727c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-jn9hq"
Oct 31 01:07:26.211792 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:07:26.211730    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/23f1b6ef-5cf9-4d1a-afdd-75a0c723727c-istio-token\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-jn9hq\" (UID: \"23f1b6ef-5cf9-4d1a-afdd-75a0c723727c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-jn9hq"
Oct 31 01:07:26.212628 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:07:26.212553    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5vj4m\" (UniqueName: \"kubernetes.io/projected/23f1b6ef-5cf9-4d1a-afdd-75a0c723727c-kube-api-access-5vj4m\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-jn9hq\" (UID: \"23f1b6ef-5cf9-4d1a-afdd-75a0c723727c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-jn9hq"
Oct 31 01:07:26.302445 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:07:26.302364    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-jn9hq"
Oct 31 01:07:26.442292 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:07:26.442261    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:07:26.705095 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:07:26.704994    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:07:26.761100 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:07:26.761068    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-jn9hq"]
Oct 31 01:07:27.765658 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:07:27.765623    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-jn9hq" event={"ID":"23f1b6ef-5cf9-4d1a-afdd-75a0c723727c","Type":"ContainerStarted","Data":"008b15c6027a0d7553cfeb88e439b14100336de2a7c2c03295baaa3f53e2f155"}
Oct 31 01:07:30.279872 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:07:30.279831    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:07:32.799286 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:07:32.799250    4964 generic.go:356] "Generic (PLEG): container finished" podID="d131714a-324f-4312-8b7a-ff717920829e" containerID="ff3e209b6b7d2951798395979619a20e162157860536947272d56801bdbf4c23" exitCode=0
Oct 31 01:07:32.799887 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:07:32.799368    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ap562g" event={"ID":"d131714a-324f-4312-8b7a-ff717920829e","Type":"ContainerDied","Data":"ff3e209b6b7d2951798395979619a20e162157860536947272d56801bdbf4c23"}
Oct 31 01:07:32.802641 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:07:32.802615    4964 generic.go:356] "Generic (PLEG): container finished" podID="5329a60f-784d-4486-83ad-0013d9f19254" containerID="2f66ebf39815b85cb6f0554a58e865d3715cb1418138a6effe38efda61a3bad9" exitCode=0
Oct 31 01:07:32.802740 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:07:32.802669    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6b6qrk" event={"ID":"5329a60f-784d-4486-83ad-0013d9f19254","Type":"ContainerDied","Data":"2f66ebf39815b85cb6f0554a58e865d3715cb1418138a6effe38efda61a3bad9"}
Oct 31 01:07:33.410288 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:07:33.410249    4964 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 01:07:33.410419 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:07:33.410376    4964 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 01:07:33.410492 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:07:33.410430    4964 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 01:07:33.809292 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:07:33.809255    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-jn9hq" event={"ID":"23f1b6ef-5cf9-4d1a-afdd-75a0c723727c","Type":"ContainerStarted","Data":"b8cc5e4779b99a864ec1494eabe62d70b306a08294a77c9d1066df055e523b2c"}
Oct 31 01:07:33.978448 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:07:33.978385    4964 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-jn9hq" podStartSLOduration=2.339880451 podStartE2EDuration="8.978366623s" totalImagesPullingTime="6.638486172s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:07:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:07:33.977550503 +0000 UTC m=+222.454116882" watchObservedRunningTime="2026-10-31 01:07:33.978366623 +0000 UTC m=+222.454933000"
Oct 31 01:07:34.303253 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:07:34.303213    4964 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-jn9hq"
Oct 31 01:07:34.308723 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:07:34.308694    4964 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-jn9hq"
Oct 31 01:07:34.814336 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:07:34.814283    4964 generic.go:356] "Generic (PLEG): container finished" podID="83b2207c-b598-4a1c-bce5-636e7cf8f970" containerID="0fc227df4efec206be7603aea3f4392d45c44e81e63f58a680e65ac081c7cd8c" exitCode=0
Oct 31 01:07:34.814687 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:07:34.814398    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cwtrkb" event={"ID":"83b2207c-b598-4a1c-bce5-636e7cf8f970","Type":"ContainerDied","Data":"0fc227df4efec206be7603aea3f4392d45c44e81e63f58a680e65ac081c7cd8c"}
Oct 31 01:07:34.815984 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:07:34.815963    4964 generic.go:356] "Generic (PLEG): container finished" podID="aa9df77e-e1a0-4084-9834-3cd7244cdc66" containerID="9582ba1dc4de0f05122df1c98ccd413c0938a938fa32def56c7c37b601e903a5" exitCode=0
Oct 31 01:07:34.816092 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:07:34.816049    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be717dwcr" event={"ID":"aa9df77e-e1a0-4084-9834-3cd7244cdc66","Type":"ContainerDied","Data":"9582ba1dc4de0f05122df1c98ccd413c0938a938fa32def56c7c37b601e903a5"}
Oct 31 01:07:34.818025 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:07:34.817997    4964 generic.go:356] "Generic (PLEG): container finished" podID="5329a60f-784d-4486-83ad-0013d9f19254" containerID="a7806f0b7f541e162baac0d836b7ca8966d13a653a53a28d5918a563e680e795" exitCode=0
Oct 31 01:07:34.818919 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:07:34.818894    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6b6qrk" event={"ID":"5329a60f-784d-4486-83ad-0013d9f19254","Type":"ContainerDied","Data":"a7806f0b7f541e162baac0d836b7ca8966d13a653a53a28d5918a563e680e795"}
Oct 31 01:07:34.819006 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:07:34.818950    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-jn9hq"
Oct 31 01:07:34.819774 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:07:34.819756    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-jn9hq"
Oct 31 01:07:35.825051 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:07:35.825016    4964 generic.go:356] "Generic (PLEG): container finished" podID="d131714a-324f-4312-8b7a-ff717920829e" containerID="48bcd571db184d1fc6cde991a30cb258fa989671f8c99d5c5e7e362f53829071" exitCode=0
Oct 31 01:07:35.825562 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:07:35.825146    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ap562g" event={"ID":"d131714a-324f-4312-8b7a-ff717920829e","Type":"ContainerDied","Data":"48bcd571db184d1fc6cde991a30cb258fa989671f8c99d5c5e7e362f53829071"}
Oct 31 01:07:35.827484 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:07:35.827397    4964 generic.go:356] "Generic (PLEG): container finished" podID="aa9df77e-e1a0-4084-9834-3cd7244cdc66" containerID="2b07198bb15fd227aaccda171d6a0f53646e9d350783b1b99d21d25b7e4eea96" exitCode=0
Oct 31 01:07:35.827484 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:07:35.827435    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be717dwcr" event={"ID":"aa9df77e-e1a0-4084-9834-3cd7244cdc66","Type":"ContainerDied","Data":"2b07198bb15fd227aaccda171d6a0f53646e9d350783b1b99d21d25b7e4eea96"}
Oct 31 01:07:38.846786 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:07:38.846748    4964 generic.go:356] "Generic (PLEG): container finished" podID="83b2207c-b598-4a1c-bce5-636e7cf8f970" containerID="faadcc27cb071239d37633c050c4990ad2791a0e38ac73ca66376201e33905cc" exitCode=0
Oct 31 01:07:38.847242 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:07:38.846837    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cwtrkb" event={"ID":"83b2207c-b598-4a1c-bce5-636e7cf8f970","Type":"ContainerDied","Data":"faadcc27cb071239d37633c050c4990ad2791a0e38ac73ca66376201e33905cc"}
Oct 31 01:07:41.260704 ip-10-0-133-197 kubenswrapper[4964]: E1031 01:07:41.260615    4964 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q56x5" containerName="extractor"
Oct 31 01:07:42.095647 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:07:42.095613    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:07:44.885555 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:07:44.884974    4964 generic.go:356] "Generic (PLEG): container finished" podID="5329a60f-784d-4486-83ad-0013d9f19254" containerID="946d57a4cf08ff63b8fcae1b29ce7c2fc7c43063542ef94868460f4e87dc0fab" exitCode=0
Oct 31 01:07:44.885555 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:07:44.885123    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6b6qrk" event={"ID":"5329a60f-784d-4486-83ad-0013d9f19254","Type":"ContainerDied","Data":"946d57a4cf08ff63b8fcae1b29ce7c2fc7c43063542ef94868460f4e87dc0fab"}
Oct 31 01:07:45.891939 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:07:45.891900    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ap562g" event={"ID":"d131714a-324f-4312-8b7a-ff717920829e","Type":"ContainerStarted","Data":"208d9199162d28b685ca5eed4a971f6f25b87e79bafedbd08719626ff1f59a49"}
Oct 31 01:07:45.894021 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:07:45.893998    4964 generic.go:356] "Generic (PLEG): container finished" podID="83b2207c-b598-4a1c-bce5-636e7cf8f970" containerID="288cbb08acb281ebdc592639a3b4eed32ca82895dc27d06dd3dd8b1a942cc7fc" exitCode=0
Oct 31 01:07:45.894125 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:07:45.894075    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cwtrkb" event={"ID":"83b2207c-b598-4a1c-bce5-636e7cf8f970","Type":"ContainerDied","Data":"288cbb08acb281ebdc592639a3b4eed32ca82895dc27d06dd3dd8b1a942cc7fc"}
Oct 31 01:07:46.009598 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:07:46.009547    4964 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ap562g" podStartSLOduration=2.954713005 podStartE2EDuration="24.00953321s" totalImagesPullingTime="21.054820205s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:07:22 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:07:46.007648846 +0000 UTC m=+234.484215225" watchObservedRunningTime="2026-10-31 01:07:46.00953321 +0000 UTC m=+234.486099586"
Oct 31 01:07:46.041604 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:07:46.041576    4964 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6b6qrk"
Oct 31 01:07:46.164513 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:07:46.164486    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5329a60f-784d-4486-83ad-0013d9f19254-kube-api-access-tzn2b\" (UniqueName: \"kubernetes.io/projected/5329a60f-784d-4486-83ad-0013d9f19254-kube-api-access-tzn2b\") pod \"5329a60f-784d-4486-83ad-0013d9f19254\" (UID: \"5329a60f-784d-4486-83ad-0013d9f19254\") "
Oct 31 01:07:46.164642 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:07:46.164526    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5329a60f-784d-4486-83ad-0013d9f19254-bundle\" (UniqueName: \"kubernetes.io/empty-dir/5329a60f-784d-4486-83ad-0013d9f19254-bundle\") pod \"5329a60f-784d-4486-83ad-0013d9f19254\" (UID: \"5329a60f-784d-4486-83ad-0013d9f19254\") "
Oct 31 01:07:46.164642 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:07:46.164560    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5329a60f-784d-4486-83ad-0013d9f19254-util\" (UniqueName: \"kubernetes.io/empty-dir/5329a60f-784d-4486-83ad-0013d9f19254-util\") pod \"5329a60f-784d-4486-83ad-0013d9f19254\" (UID: \"5329a60f-784d-4486-83ad-0013d9f19254\") "
Oct 31 01:07:46.164958 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:07:46.164931    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5329a60f-784d-4486-83ad-0013d9f19254-bundle" pod "5329a60f-784d-4486-83ad-0013d9f19254" (UID: "5329a60f-784d-4486-83ad-0013d9f19254"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:07:46.166879 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:07:46.166854    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5329a60f-784d-4486-83ad-0013d9f19254-kube-api-access-tzn2b" pod "5329a60f-784d-4486-83ad-0013d9f19254" (UID: "5329a60f-784d-4486-83ad-0013d9f19254"). InnerVolumeSpecName "kube-api-access-tzn2b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:07:46.168280 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:07:46.168253    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5329a60f-784d-4486-83ad-0013d9f19254-util" pod "5329a60f-784d-4486-83ad-0013d9f19254" (UID: "5329a60f-784d-4486-83ad-0013d9f19254"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:07:46.265804 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:07:46.265772    4964 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tzn2b\" (UniqueName: \"kubernetes.io/projected/5329a60f-784d-4486-83ad-0013d9f19254-kube-api-access-tzn2b\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:07:46.265804 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:07:46.265805    4964 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5329a60f-784d-4486-83ad-0013d9f19254-bundle\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:07:46.265989 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:07:46.265816    4964 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5329a60f-784d-4486-83ad-0013d9f19254-util\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:07:46.653836 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:07:46.653802    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:07:46.894632 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:07:46.894588    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:07:46.901201 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:07:46.901171    4964 generic.go:356] "Generic (PLEG): container finished" podID="aa9df77e-e1a0-4084-9834-3cd7244cdc66" containerID="b2bd18fe85b46b587975e61e456b4c999af2c3b7363916417de48998c0dee30c" exitCode=0
Oct 31 01:07:46.901391 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:07:46.901215    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be717dwcr" event={"ID":"aa9df77e-e1a0-4084-9834-3cd7244cdc66","Type":"ContainerDied","Data":"b2bd18fe85b46b587975e61e456b4c999af2c3b7363916417de48998c0dee30c"}
Oct 31 01:07:46.903683 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:07:46.903656    4964 generic.go:356] "Generic (PLEG): container finished" podID="d131714a-324f-4312-8b7a-ff717920829e" containerID="208d9199162d28b685ca5eed4a971f6f25b87e79bafedbd08719626ff1f59a49" exitCode=0
Oct 31 01:07:46.903812 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:07:46.903733    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ap562g" event={"ID":"d131714a-324f-4312-8b7a-ff717920829e","Type":"ContainerDied","Data":"208d9199162d28b685ca5eed4a971f6f25b87e79bafedbd08719626ff1f59a49"}
Oct 31 01:07:46.905747 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:07:46.905685    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6b6qrk" event={"ID":"5329a60f-784d-4486-83ad-0013d9f19254","Type":"ContainerDied","Data":"d905a48adfc78a4365a95aa60ec99a9215837e8806b15fcdf726d6f8250ab2d5"}
Oct 31 01:07:46.905747 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:07:46.905719    4964 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6b6qrk"
Oct 31 01:07:46.905747 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:07:46.905734    4964 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d905a48adfc78a4365a95aa60ec99a9215837e8806b15fcdf726d6f8250ab2d5"
Oct 31 01:07:47.053130 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:07:47.053105    4964 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cwtrkb"
Oct 31 01:07:47.172906 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:07:47.172822    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/83b2207c-b598-4a1c-bce5-636e7cf8f970-util\" (UniqueName: \"kubernetes.io/empty-dir/83b2207c-b598-4a1c-bce5-636e7cf8f970-util\") pod \"83b2207c-b598-4a1c-bce5-636e7cf8f970\" (UID: \"83b2207c-b598-4a1c-bce5-636e7cf8f970\") "
Oct 31 01:07:47.173087 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:07:47.172940    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/83b2207c-b598-4a1c-bce5-636e7cf8f970-kube-api-access-sxstv\" (UniqueName: \"kubernetes.io/projected/83b2207c-b598-4a1c-bce5-636e7cf8f970-kube-api-access-sxstv\") pod \"83b2207c-b598-4a1c-bce5-636e7cf8f970\" (UID: \"83b2207c-b598-4a1c-bce5-636e7cf8f970\") "
Oct 31 01:07:47.173087 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:07:47.172996    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/83b2207c-b598-4a1c-bce5-636e7cf8f970-bundle\" (UniqueName: \"kubernetes.io/empty-dir/83b2207c-b598-4a1c-bce5-636e7cf8f970-bundle\") pod \"83b2207c-b598-4a1c-bce5-636e7cf8f970\" (UID: \"83b2207c-b598-4a1c-bce5-636e7cf8f970\") "
Oct 31 01:07:47.173631 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:07:47.173606    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83b2207c-b598-4a1c-bce5-636e7cf8f970-bundle" pod "83b2207c-b598-4a1c-bce5-636e7cf8f970" (UID: "83b2207c-b598-4a1c-bce5-636e7cf8f970"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:07:47.175277 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:07:47.175254    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/83b2207c-b598-4a1c-bce5-636e7cf8f970-kube-api-access-sxstv" pod "83b2207c-b598-4a1c-bce5-636e7cf8f970" (UID: "83b2207c-b598-4a1c-bce5-636e7cf8f970"). InnerVolumeSpecName "kube-api-access-sxstv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:07:47.176869 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:07:47.176847    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83b2207c-b598-4a1c-bce5-636e7cf8f970-util" pod "83b2207c-b598-4a1c-bce5-636e7cf8f970" (UID: "83b2207c-b598-4a1c-bce5-636e7cf8f970"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:07:47.274476 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:07:47.274443    4964 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sxstv\" (UniqueName: \"kubernetes.io/projected/83b2207c-b598-4a1c-bce5-636e7cf8f970-kube-api-access-sxstv\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:07:47.274476 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:07:47.274475    4964 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/83b2207c-b598-4a1c-bce5-636e7cf8f970-bundle\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:07:47.274476 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:07:47.274484    4964 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/83b2207c-b598-4a1c-bce5-636e7cf8f970-util\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:07:47.912096 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:07:47.912060    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cwtrkb" event={"ID":"83b2207c-b598-4a1c-bce5-636e7cf8f970","Type":"ContainerDied","Data":"f8c31db4e59a6488525c81259aa64fd5cf0f7922972d4f13aeb1ee7c6f4b5d53"}
Oct 31 01:07:47.912488 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:07:47.912100    4964 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cwtrkb"
Oct 31 01:07:47.912488 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:07:47.912115    4964 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f8c31db4e59a6488525c81259aa64fd5cf0f7922972d4f13aeb1ee7c6f4b5d53"
Oct 31 01:07:48.071943 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:07:48.071914    4964 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be717dwcr"
Oct 31 01:07:48.081573 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:07:48.081546    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aa9df77e-e1a0-4084-9834-3cd7244cdc66-kube-api-access-79dzf\" (UniqueName: \"kubernetes.io/projected/aa9df77e-e1a0-4084-9834-3cd7244cdc66-kube-api-access-79dzf\") pod \"aa9df77e-e1a0-4084-9834-3cd7244cdc66\" (UID: \"aa9df77e-e1a0-4084-9834-3cd7244cdc66\") "
Oct 31 01:07:48.081727 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:07:48.081602    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aa9df77e-e1a0-4084-9834-3cd7244cdc66-bundle\" (UniqueName: \"kubernetes.io/empty-dir/aa9df77e-e1a0-4084-9834-3cd7244cdc66-bundle\") pod \"aa9df77e-e1a0-4084-9834-3cd7244cdc66\" (UID: \"aa9df77e-e1a0-4084-9834-3cd7244cdc66\") "
Oct 31 01:07:48.082143 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:07:48.082117    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa9df77e-e1a0-4084-9834-3cd7244cdc66-bundle" pod "aa9df77e-e1a0-4084-9834-3cd7244cdc66" (UID: "aa9df77e-e1a0-4084-9834-3cd7244cdc66"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:07:48.083727 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:07:48.083704    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa9df77e-e1a0-4084-9834-3cd7244cdc66-kube-api-access-79dzf" pod "aa9df77e-e1a0-4084-9834-3cd7244cdc66" (UID: "aa9df77e-e1a0-4084-9834-3cd7244cdc66"). InnerVolumeSpecName "kube-api-access-79dzf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:07:48.087880 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:07:48.087864    4964 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ap562g"
Oct 31 01:07:48.182693 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:07:48.182652    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d131714a-324f-4312-8b7a-ff717920829e-kube-api-access-jwlmb\" (UniqueName: \"kubernetes.io/projected/d131714a-324f-4312-8b7a-ff717920829e-kube-api-access-jwlmb\") pod \"d131714a-324f-4312-8b7a-ff717920829e\" (UID: \"d131714a-324f-4312-8b7a-ff717920829e\") "
Oct 31 01:07:48.182868 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:07:48.182723    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d131714a-324f-4312-8b7a-ff717920829e-bundle\" (UniqueName: \"kubernetes.io/empty-dir/d131714a-324f-4312-8b7a-ff717920829e-bundle\") pod \"d131714a-324f-4312-8b7a-ff717920829e\" (UID: \"d131714a-324f-4312-8b7a-ff717920829e\") "
Oct 31 01:07:48.182868 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:07:48.182749    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aa9df77e-e1a0-4084-9834-3cd7244cdc66-util\" (UniqueName: \"kubernetes.io/empty-dir/aa9df77e-e1a0-4084-9834-3cd7244cdc66-util\") pod \"aa9df77e-e1a0-4084-9834-3cd7244cdc66\" (UID: \"aa9df77e-e1a0-4084-9834-3cd7244cdc66\") "
Oct 31 01:07:48.182961 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:07:48.182944    4964 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/aa9df77e-e1a0-4084-9834-3cd7244cdc66-bundle\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:07:48.182997 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:07:48.182969    4964 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-79dzf\" (UniqueName: \"kubernetes.io/projected/aa9df77e-e1a0-4084-9834-3cd7244cdc66-kube-api-access-79dzf\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:07:48.183578 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:07:48.183544    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d131714a-324f-4312-8b7a-ff717920829e-bundle" pod "d131714a-324f-4312-8b7a-ff717920829e" (UID: "d131714a-324f-4312-8b7a-ff717920829e"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:07:48.184983 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:07:48.184963    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d131714a-324f-4312-8b7a-ff717920829e-kube-api-access-jwlmb" pod "d131714a-324f-4312-8b7a-ff717920829e" (UID: "d131714a-324f-4312-8b7a-ff717920829e"). InnerVolumeSpecName "kube-api-access-jwlmb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:07:48.188733 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:07:48.188711    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa9df77e-e1a0-4084-9834-3cd7244cdc66-util" pod "aa9df77e-e1a0-4084-9834-3cd7244cdc66" (UID: "aa9df77e-e1a0-4084-9834-3cd7244cdc66"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:07:48.283377 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:07:48.283341    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d131714a-324f-4312-8b7a-ff717920829e-util\" (UniqueName: \"kubernetes.io/empty-dir/d131714a-324f-4312-8b7a-ff717920829e-util\") pod \"d131714a-324f-4312-8b7a-ff717920829e\" (UID: \"d131714a-324f-4312-8b7a-ff717920829e\") "
Oct 31 01:07:48.283546 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:07:48.283504    4964 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d131714a-324f-4312-8b7a-ff717920829e-bundle\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:07:48.283546 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:07:48.283518    4964 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/aa9df77e-e1a0-4084-9834-3cd7244cdc66-util\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:07:48.283546 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:07:48.283527    4964 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jwlmb\" (UniqueName: \"kubernetes.io/projected/d131714a-324f-4312-8b7a-ff717920829e-kube-api-access-jwlmb\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:07:48.289286 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:07:48.289258    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d131714a-324f-4312-8b7a-ff717920829e-util" pod "d131714a-324f-4312-8b7a-ff717920829e" (UID: "d131714a-324f-4312-8b7a-ff717920829e"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:07:48.384691 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:07:48.384604    4964 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d131714a-324f-4312-8b7a-ff717920829e-util\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:07:48.918537 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:07:48.918498    4964 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ap562g"
Oct 31 01:07:48.918982 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:07:48.918502    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ap562g" event={"ID":"d131714a-324f-4312-8b7a-ff717920829e","Type":"ContainerDied","Data":"7b4ddd588f69a8c73174c9cc193f84be8301f3d614c305f3e20405bbe065a96a"}
Oct 31 01:07:48.918982 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:07:48.918630    4964 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7b4ddd588f69a8c73174c9cc193f84be8301f3d614c305f3e20405bbe065a96a"
Oct 31 01:07:48.920358 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:07:48.920330    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be717dwcr" event={"ID":"aa9df77e-e1a0-4084-9834-3cd7244cdc66","Type":"ContainerDied","Data":"3bdc336e31d6bf2b63df67875f46161606659009e0dd6c0758f737bf5e7aba2f"}
Oct 31 01:07:48.920358 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:07:48.920357    4964 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be717dwcr"
Oct 31 01:07:48.920549 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:07:48.920374    4964 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3bdc336e31d6bf2b63df67875f46161606659009e0dd6c0758f737bf5e7aba2f"
Oct 31 01:07:55.200170 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:07:55.200131    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:07:56.744898 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:07:56.744857    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:07:58.404937 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:07:58.404901    4964 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-xzzcq"]
Oct 31 01:07:58.405359 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:07:58.405249    4964 cpu_manager.go:414] "containerMap: removing container" podUID="83b2207c-b598-4a1c-bce5-636e7cf8f970" containerName="pull"
Oct 31 01:07:58.405359 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:07:58.405262    4964 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="83b2207c-b598-4a1c-bce5-636e7cf8f970" containerName="pull"
Oct 31 01:07:58.405359 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:07:58.405275    4964 cpu_manager.go:414] "containerMap: removing container" podUID="83b2207c-b598-4a1c-bce5-636e7cf8f970" containerName="util"
Oct 31 01:07:58.405359 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:07:58.405279    4964 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="83b2207c-b598-4a1c-bce5-636e7cf8f970" containerName="util"
Oct 31 01:07:58.405359 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:07:58.405288    4964 cpu_manager.go:414] "containerMap: removing container" podUID="aa9df77e-e1a0-4084-9834-3cd7244cdc66" containerName="util"
Oct 31 01:07:58.405359 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:07:58.405293    4964 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aa9df77e-e1a0-4084-9834-3cd7244cdc66" containerName="util"
Oct 31 01:07:58.405359 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:07:58.405305    4964 cpu_manager.go:414] "containerMap: removing container" podUID="83b2207c-b598-4a1c-bce5-636e7cf8f970" containerName="extract"
Oct 31 01:07:58.405359 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:07:58.405310    4964 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="83b2207c-b598-4a1c-bce5-636e7cf8f970" containerName="extract"
Oct 31 01:07:58.405359 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:07:58.405344    4964 cpu_manager.go:414] "containerMap: removing container" podUID="d131714a-324f-4312-8b7a-ff717920829e" containerName="extract"
Oct 31 01:07:58.405359 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:07:58.405351    4964 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d131714a-324f-4312-8b7a-ff717920829e" containerName="extract"
Oct 31 01:07:58.405359 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:07:58.405360    4964 cpu_manager.go:414] "containerMap: removing container" podUID="aa9df77e-e1a0-4084-9834-3cd7244cdc66" containerName="extract"
Oct 31 01:07:58.405359 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:07:58.405365    4964 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aa9df77e-e1a0-4084-9834-3cd7244cdc66" containerName="extract"
Oct 31 01:07:58.405839 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:07:58.405381    4964 cpu_manager.go:414] "containerMap: removing container" podUID="5329a60f-784d-4486-83ad-0013d9f19254" containerName="util"
Oct 31 01:07:58.405839 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:07:58.405386    4964 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5329a60f-784d-4486-83ad-0013d9f19254" containerName="util"
Oct 31 01:07:58.405839 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:07:58.405396    4964 cpu_manager.go:414] "containerMap: removing container" podUID="d131714a-324f-4312-8b7a-ff717920829e" containerName="util"
Oct 31 01:07:58.405839 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:07:58.405400    4964 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d131714a-324f-4312-8b7a-ff717920829e" containerName="util"
Oct 31 01:07:58.405839 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:07:58.405405    4964 cpu_manager.go:414] "containerMap: removing container" podUID="aa9df77e-e1a0-4084-9834-3cd7244cdc66" containerName="pull"
Oct 31 01:07:58.405839 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:07:58.405410    4964 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aa9df77e-e1a0-4084-9834-3cd7244cdc66" containerName="pull"
Oct 31 01:07:58.405839 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:07:58.405415    4964 cpu_manager.go:414] "containerMap: removing container" podUID="5329a60f-784d-4486-83ad-0013d9f19254" containerName="extract"
Oct 31 01:07:58.405839 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:07:58.405418    4964 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5329a60f-784d-4486-83ad-0013d9f19254" containerName="extract"
Oct 31 01:07:58.405839 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:07:58.405427    4964 cpu_manager.go:414] "containerMap: removing container" podUID="5329a60f-784d-4486-83ad-0013d9f19254" containerName="pull"
Oct 31 01:07:58.405839 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:07:58.405430    4964 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5329a60f-784d-4486-83ad-0013d9f19254" containerName="pull"
Oct 31 01:07:58.405839 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:07:58.405435    4964 cpu_manager.go:414] "containerMap: removing container" podUID="d131714a-324f-4312-8b7a-ff717920829e" containerName="pull"
Oct 31 01:07:58.405839 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:07:58.405439    4964 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d131714a-324f-4312-8b7a-ff717920829e" containerName="pull"
Oct 31 01:07:58.405839 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:07:58.405497    4964 memory_manager.go:365] "RemoveStaleState removing state" podUID="83b2207c-b598-4a1c-bce5-636e7cf8f970" containerName="extract"
Oct 31 01:07:58.405839 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:07:58.405508    4964 memory_manager.go:365] "RemoveStaleState removing state" podUID="d131714a-324f-4312-8b7a-ff717920829e" containerName="extract"
Oct 31 01:07:58.405839 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:07:58.405515    4964 memory_manager.go:365] "RemoveStaleState removing state" podUID="aa9df77e-e1a0-4084-9834-3cd7244cdc66" containerName="extract"
Oct 31 01:07:58.405839 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:07:58.405522    4964 memory_manager.go:365] "RemoveStaleState removing state" podUID="5329a60f-784d-4486-83ad-0013d9f19254" containerName="extract"
Oct 31 01:07:58.407947 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:07:58.407931    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-xzzcq"
Oct 31 01:07:58.456704 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:07:58.456682    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/23955186-e493-4f42-8e4b-fe079c3ad7ca-tmp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-xzzcq\" (UID: \"23955186-e493-4f42-8e4b-fe079c3ad7ca\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-xzzcq"
Oct 31 01:07:58.456823 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:07:58.456722    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vvkpv\" (UniqueName: \"kubernetes.io/projected/23955186-e493-4f42-8e4b-fe079c3ad7ca-kube-api-access-vvkpv\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-xzzcq\" (UID: \"23955186-e493-4f42-8e4b-fe079c3ad7ca\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-xzzcq"
Oct 31 01:07:58.459920 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:07:58.459898    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-xzzcq"]
Oct 31 01:07:58.468740 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:07:58.468718    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:07:58.507522 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:07:58.507500    4964 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\""
Oct 31 01:07:58.507621 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:07:58.507528    4964 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-wrt6v\""
Oct 31 01:07:58.507621 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:07:58.507547    4964 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\""
Oct 31 01:07:58.557336 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:07:58.557289    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vvkpv\" (UniqueName: \"kubernetes.io/projected/23955186-e493-4f42-8e4b-fe079c3ad7ca-kube-api-access-vvkpv\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-xzzcq\" (UID: \"23955186-e493-4f42-8e4b-fe079c3ad7ca\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-xzzcq"
Oct 31 01:07:58.557445 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:07:58.557379    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/23955186-e493-4f42-8e4b-fe079c3ad7ca-tmp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-xzzcq\" (UID: \"23955186-e493-4f42-8e4b-fe079c3ad7ca\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-xzzcq"
Oct 31 01:07:58.557677 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:07:58.557662    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/23955186-e493-4f42-8e4b-fe079c3ad7ca-tmp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-xzzcq\" (UID: \"23955186-e493-4f42-8e4b-fe079c3ad7ca\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-xzzcq"
Oct 31 01:07:58.614509 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:07:58.614478    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vvkpv\" (UniqueName: \"kubernetes.io/projected/23955186-e493-4f42-8e4b-fe079c3ad7ca-kube-api-access-vvkpv\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-xzzcq\" (UID: \"23955186-e493-4f42-8e4b-fe079c3ad7ca\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-xzzcq"
Oct 31 01:07:58.661856 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:07:58.661780    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:07:58.717668 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:07:58.717645    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-xzzcq"
Oct 31 01:07:59.191725 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:07:59.191675    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-xzzcq"]
Oct 31 01:07:59.194420 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:07:59.194390    4964 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod23955186_e493_4f42_8e4b_fe079c3ad7ca.slice/crio-fbd68c77667a69c9a6f29aaaf2bed8daea925da98aa347a8868dae1d2960451e WatchSource:0}: Error finding container fbd68c77667a69c9a6f29aaaf2bed8daea925da98aa347a8868dae1d2960451e: Status 404 returned error can't find the container with id fbd68c77667a69c9a6f29aaaf2bed8daea925da98aa347a8868dae1d2960451e
Oct 31 01:07:59.629508 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:07:59.629477    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:07:59.760849 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:07:59.760821    4964 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-559fd7ff56-m5c2x"]
Oct 31 01:07:59.763132 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:07:59.763117    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-559fd7ff56-m5c2x"
Oct 31 01:07:59.766371 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:07:59.766347    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5cfd9ec8-fd50-421c-8744-18c0b03903d6-tmp\") pod \"opendatahub-operator-controller-manager-559fd7ff56-m5c2x\" (UID: \"5cfd9ec8-fd50-421c-8744-18c0b03903d6\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-559fd7ff56-m5c2x"
Oct 31 01:07:59.766524 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:07:59.766389    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/5cfd9ec8-fd50-421c-8744-18c0b03903d6-webhook-cert\") pod \"opendatahub-operator-controller-manager-559fd7ff56-m5c2x\" (UID: \"5cfd9ec8-fd50-421c-8744-18c0b03903d6\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-559fd7ff56-m5c2x"
Oct 31 01:07:59.766524 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:07:59.766456    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wgj9j\" (UniqueName: \"kubernetes.io/projected/5cfd9ec8-fd50-421c-8744-18c0b03903d6-kube-api-access-wgj9j\") pod \"opendatahub-operator-controller-manager-559fd7ff56-m5c2x\" (UID: \"5cfd9ec8-fd50-421c-8744-18c0b03903d6\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-559fd7ff56-m5c2x"
Oct 31 01:07:59.766524 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:07:59.766500    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/5cfd9ec8-fd50-421c-8744-18c0b03903d6-apiservice-cert\") pod \"opendatahub-operator-controller-manager-559fd7ff56-m5c2x\" (UID: \"5cfd9ec8-fd50-421c-8744-18c0b03903d6\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-559fd7ff56-m5c2x"
Oct 31 01:07:59.766524 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:07:59.766523    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/5cfd9ec8-fd50-421c-8744-18c0b03903d6-metrics-cert\") pod \"opendatahub-operator-controller-manager-559fd7ff56-m5c2x\" (UID: \"5cfd9ec8-fd50-421c-8744-18c0b03903d6\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-559fd7ff56-m5c2x"
Oct 31 01:07:59.766723 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:07:59.766634    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/5cfd9ec8-fd50-421c-8744-18c0b03903d6-manifests\") pod \"opendatahub-operator-controller-manager-559fd7ff56-m5c2x\" (UID: \"5cfd9ec8-fd50-421c-8744-18c0b03903d6\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-559fd7ff56-m5c2x"
Oct 31 01:07:59.841124 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:07:59.841101    4964 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\""
Oct 31 01:07:59.841826 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:07:59.841807    4964 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\""
Oct 31 01:07:59.867997 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:07:59.867968    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wgj9j\" (UniqueName: \"kubernetes.io/projected/5cfd9ec8-fd50-421c-8744-18c0b03903d6-kube-api-access-wgj9j\") pod \"opendatahub-operator-controller-manager-559fd7ff56-m5c2x\" (UID: \"5cfd9ec8-fd50-421c-8744-18c0b03903d6\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-559fd7ff56-m5c2x"
Oct 31 01:07:59.868120 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:07:59.868009    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/5cfd9ec8-fd50-421c-8744-18c0b03903d6-apiservice-cert\") pod \"opendatahub-operator-controller-manager-559fd7ff56-m5c2x\" (UID: \"5cfd9ec8-fd50-421c-8744-18c0b03903d6\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-559fd7ff56-m5c2x"
Oct 31 01:07:59.868120 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:07:59.868036    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/5cfd9ec8-fd50-421c-8744-18c0b03903d6-metrics-cert\") pod \"opendatahub-operator-controller-manager-559fd7ff56-m5c2x\" (UID: \"5cfd9ec8-fd50-421c-8744-18c0b03903d6\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-559fd7ff56-m5c2x"
Oct 31 01:07:59.868241 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:07:59.868183    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/5cfd9ec8-fd50-421c-8744-18c0b03903d6-manifests\") pod \"opendatahub-operator-controller-manager-559fd7ff56-m5c2x\" (UID: \"5cfd9ec8-fd50-421c-8744-18c0b03903d6\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-559fd7ff56-m5c2x"
Oct 31 01:07:59.868241 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:07:59.868218    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5cfd9ec8-fd50-421c-8744-18c0b03903d6-tmp\") pod \"opendatahub-operator-controller-manager-559fd7ff56-m5c2x\" (UID: \"5cfd9ec8-fd50-421c-8744-18c0b03903d6\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-559fd7ff56-m5c2x"
Oct 31 01:07:59.868399 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:07:59.868279    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/5cfd9ec8-fd50-421c-8744-18c0b03903d6-webhook-cert\") pod \"opendatahub-operator-controller-manager-559fd7ff56-m5c2x\" (UID: \"5cfd9ec8-fd50-421c-8744-18c0b03903d6\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-559fd7ff56-m5c2x"
Oct 31 01:07:59.868543 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:07:59.868525    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/5cfd9ec8-fd50-421c-8744-18c0b03903d6-manifests\") pod \"opendatahub-operator-controller-manager-559fd7ff56-m5c2x\" (UID: \"5cfd9ec8-fd50-421c-8744-18c0b03903d6\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-559fd7ff56-m5c2x"
Oct 31 01:07:59.870546 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:07:59.870520    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5cfd9ec8-fd50-421c-8744-18c0b03903d6-tmp\") pod \"opendatahub-operator-controller-manager-559fd7ff56-m5c2x\" (UID: \"5cfd9ec8-fd50-421c-8744-18c0b03903d6\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-559fd7ff56-m5c2x"
Oct 31 01:07:59.870649 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:07:59.870626    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/5cfd9ec8-fd50-421c-8744-18c0b03903d6-apiservice-cert\") pod \"opendatahub-operator-controller-manager-559fd7ff56-m5c2x\" (UID: \"5cfd9ec8-fd50-421c-8744-18c0b03903d6\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-559fd7ff56-m5c2x"
Oct 31 01:07:59.870713 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:07:59.870656    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/5cfd9ec8-fd50-421c-8744-18c0b03903d6-webhook-cert\") pod \"opendatahub-operator-controller-manager-559fd7ff56-m5c2x\" (UID: \"5cfd9ec8-fd50-421c-8744-18c0b03903d6\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-559fd7ff56-m5c2x"
Oct 31 01:07:59.875343 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:07:59.875304    4964 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\""
Oct 31 01:07:59.875479 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:07:59.875463    4964 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-9qxzz\""
Oct 31 01:07:59.875540 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:07:59.875474    4964 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\""
Oct 31 01:07:59.875540 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:07:59.875488    4964 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\""
Oct 31 01:07:59.880782 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:07:59.880727    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/5cfd9ec8-fd50-421c-8744-18c0b03903d6-metrics-cert\") pod \"opendatahub-operator-controller-manager-559fd7ff56-m5c2x\" (UID: \"5cfd9ec8-fd50-421c-8744-18c0b03903d6\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-559fd7ff56-m5c2x"
Oct 31 01:07:59.944105 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:07:59.944086    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wgj9j\" (UniqueName: \"kubernetes.io/projected/5cfd9ec8-fd50-421c-8744-18c0b03903d6-kube-api-access-wgj9j\") pod \"opendatahub-operator-controller-manager-559fd7ff56-m5c2x\" (UID: \"5cfd9ec8-fd50-421c-8744-18c0b03903d6\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-559fd7ff56-m5c2x"
Oct 31 01:07:59.966222 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:07:59.966179    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-xzzcq" event={"ID":"23955186-e493-4f42-8e4b-fe079c3ad7ca","Type":"ContainerStarted","Data":"fbd68c77667a69c9a6f29aaaf2bed8daea925da98aa347a8868dae1d2960451e"}
Oct 31 01:08:00.067956 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:08:00.067929    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-559fd7ff56-m5c2x"]
Oct 31 01:08:00.075954 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:08:00.075921    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-559fd7ff56-m5c2x"
Oct 31 01:08:00.206343 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:08:00.206248    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:08:00.455040 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:08:00.455010    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:08:00.575113 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:08:00.575092    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-559fd7ff56-m5c2x"]
Oct 31 01:08:00.576949 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:08:00.576925    4964 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5cfd9ec8_fd50_421c_8744_18c0b03903d6.slice/crio-1c6d00d0b6976ad913b7191355b2039607a11ce775814db05185732936ff124c WatchSource:0}: Error finding container 1c6d00d0b6976ad913b7191355b2039607a11ce775814db05185732936ff124c: Status 404 returned error can't find the container with id 1c6d00d0b6976ad913b7191355b2039607a11ce775814db05185732936ff124c
Oct 31 01:08:00.736581 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:08:00.736549    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:08:00.975176 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:08:00.975142    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-559fd7ff56-m5c2x" event={"ID":"5cfd9ec8-fd50-421c-8744-18c0b03903d6","Type":"ContainerStarted","Data":"1c6d00d0b6976ad913b7191355b2039607a11ce775814db05185732936ff124c"}
Oct 31 01:08:00.999786 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:08:00.999742    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:08:01.234439 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:08:01.234363    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:08:01.984646 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:08:01.984609    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:08:02.207966 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:08:02.207841    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:08:02.854672 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:08:02.854629    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:08:02.989155 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:08:02.989119    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-xzzcq" event={"ID":"23955186-e493-4f42-8e4b-fe079c3ad7ca","Type":"ContainerStarted","Data":"d43ce99eba0a77b6f47359c4e50bfb987ae583cb7f7cc42e7bb91f5709f762c6"}
Oct 31 01:08:03.178779 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:08:03.178741    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:08:03.214059 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:08:03.213999    4964 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-xzzcq" podStartSLOduration=1.7344759669999998 podStartE2EDuration="5.213980375s" totalImagesPullingTime="3.479504408s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:07:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:08:03.162521483 +0000 UTC m=+251.639087860" watchObservedRunningTime="2026-10-31 01:08:03.213980375 +0000 UTC m=+251.690546759"
Oct 31 01:08:04.714800 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:08:04.714766    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:08:04.945900 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:08:04.945872    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:08:04.999173 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:08:04.999146    4964 generic.go:356] "Generic (PLEG): container finished" podID="5cfd9ec8-fd50-421c-8744-18c0b03903d6" containerID="f59dfd24f3adafdf2ab3ddd11a41220ff490deed692773ca205d169047cb1792" exitCode=0
Oct 31 01:08:04.999296 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:08:04.999228    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-559fd7ff56-m5c2x" event={"ID":"5cfd9ec8-fd50-421c-8744-18c0b03903d6","Type":"ContainerDied","Data":"f59dfd24f3adafdf2ab3ddd11a41220ff490deed692773ca205d169047cb1792"}
Oct 31 01:08:05.595913 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:08:05.595874    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:08:06.008479 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:08:06.008442    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-559fd7ff56-m5c2x" event={"ID":"5cfd9ec8-fd50-421c-8744-18c0b03903d6","Type":"ContainerStarted","Data":"f8a153c5e508f3e1b470ed776062956276bbc24225bac12af315429651c60ac1"}
Oct 31 01:08:06.008861 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:08:06.008584    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-559fd7ff56-m5c2x"
Oct 31 01:08:06.107824 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:08:06.107724    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:08:06.248279 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:08:06.248223    4964 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-559fd7ff56-m5c2x" podStartSLOduration=2.305292664 podStartE2EDuration="7.248209342s" totalImagesPullingTime="4.352583798s" totalInitContainerRuntime="590.33288ms" isStatefulPod=false podCreationTimestamp="2026-10-31 01:07:59 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:08:06.24702323 +0000 UTC m=+254.723589623" watchObservedRunningTime="2026-10-31 01:08:06.248209342 +0000 UTC m=+254.724775721"
Oct 31 01:08:06.851464 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:08:06.851425    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:08:07.129925 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:08:07.129841    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:08:07.506370 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:08:07.506342    4964 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-4mfd5"]
Oct 31 01:08:07.508775 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:08:07.508760    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-4mfd5"
Oct 31 01:08:07.529211 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:08:07.529184    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5kfxj\" (UniqueName: \"kubernetes.io/projected/2ddf195b-447a-460c-8b02-ca432fda1fbc-kube-api-access-5kfxj\") pod \"cert-manager-cainjector-5c57bbd4fc-4mfd5\" (UID: \"2ddf195b-447a-460c-8b02-ca432fda1fbc\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-4mfd5"
Oct 31 01:08:07.529376 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:08:07.529236    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2ddf195b-447a-460c-8b02-ca432fda1fbc-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-4mfd5\" (UID: \"2ddf195b-447a-460c-8b02-ca432fda1fbc\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-4mfd5"
Oct 31 01:08:07.561379 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:08:07.561351    4964 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Oct 31 01:08:07.561478 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:08:07.561379    4964 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Oct 31 01:08:07.561478 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:08:07.561399    4964 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-9vx2s\""
Oct 31 01:08:07.562975 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:08:07.562950    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:08:07.582624 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:08:07.582601    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-4mfd5"]
Oct 31 01:08:07.630482 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:08:07.630457    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2ddf195b-447a-460c-8b02-ca432fda1fbc-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-4mfd5\" (UID: \"2ddf195b-447a-460c-8b02-ca432fda1fbc\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-4mfd5"
Oct 31 01:08:07.630581 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:08:07.630514    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5kfxj\" (UniqueName: \"kubernetes.io/projected/2ddf195b-447a-460c-8b02-ca432fda1fbc-kube-api-access-5kfxj\") pod \"cert-manager-cainjector-5c57bbd4fc-4mfd5\" (UID: \"2ddf195b-447a-460c-8b02-ca432fda1fbc\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-4mfd5"
Oct 31 01:08:07.696563 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:08:07.696531    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2ddf195b-447a-460c-8b02-ca432fda1fbc-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-4mfd5\" (UID: \"2ddf195b-447a-460c-8b02-ca432fda1fbc\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-4mfd5"
Oct 31 01:08:07.696658 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:08:07.696614    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5kfxj\" (UniqueName: \"kubernetes.io/projected/2ddf195b-447a-460c-8b02-ca432fda1fbc-kube-api-access-5kfxj\") pod \"cert-manager-cainjector-5c57bbd4fc-4mfd5\" (UID: \"2ddf195b-447a-460c-8b02-ca432fda1fbc\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-4mfd5"
Oct 31 01:08:07.827636 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:08:07.827554    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-4mfd5"
Oct 31 01:08:08.034380 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:08:08.034348    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:08:08.207298 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:08:08.207275    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-4mfd5"]
Oct 31 01:08:08.209965 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:08:08.209939    4964 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2ddf195b_447a_460c_8b02_ca432fda1fbc.slice/crio-632699ca1592a358cefa9f0ff10db902b2e2b3673332169f74a54c4dc8a62355 WatchSource:0}: Error finding container 632699ca1592a358cefa9f0ff10db902b2e2b3673332169f74a54c4dc8a62355: Status 404 returned error can't find the container with id 632699ca1592a358cefa9f0ff10db902b2e2b3673332169f74a54c4dc8a62355
Oct 31 01:08:09.021814 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:08:09.021784    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-4mfd5" event={"ID":"2ddf195b-447a-460c-8b02-ca432fda1fbc","Type":"ContainerStarted","Data":"632699ca1592a358cefa9f0ff10db902b2e2b3673332169f74a54c4dc8a62355"}
Oct 31 01:08:09.226471 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:08:09.226437    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:08:11.218916 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:08:11.218884    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:08:11.243267 ip-10-0-133-197 kubenswrapper[4964]: E1031 01:08:11.243222    4964 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q56x5" containerName="extractor"
Oct 31 01:08:12.042852 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:08:12.042818    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-4mfd5" event={"ID":"2ddf195b-447a-460c-8b02-ca432fda1fbc","Type":"ContainerStarted","Data":"142eb11baeb144454ea33c3452bdafa9d0187c41fc7c33dc184aaffc0b7b801c"}
Oct 31 01:08:12.160419 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:08:12.160308    4964 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-4mfd5" podStartSLOduration=1.565573042 podStartE2EDuration="5.160295779s" totalImagesPullingTime="3.594722737s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:08:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:08:12.158555953 +0000 UTC m=+260.635122333" watchObservedRunningTime="2026-10-31 01:08:12.160295779 +0000 UTC m=+260.636862159"
Oct 31 01:08:12.729228 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:08:12.729194    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:08:13.388677 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:08:13.388639    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:08:13.629537 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:08:13.629509    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:08:14.524191 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:08:14.524158    4964 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-cz9bm"]
Oct 31 01:08:14.552921 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:08:14.552897    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-cz9bm"
Oct 31 01:08:14.579010 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:08:14.578991    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-cz9bm"]
Oct 31 01:08:14.588939 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:08:14.588913    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n5rdj\" (UniqueName: \"kubernetes.io/projected/f5252add-5cab-4a62-ad5a-5b5d554ed0da-kube-api-access-n5rdj\") pod \"kuadrant-operator-controller-manager-cf8469d7b-cz9bm\" (UID: \"f5252add-5cab-4a62-ad5a-5b5d554ed0da\") " pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-cz9bm"
Oct 31 01:08:14.589045 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:08:14.588945    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/f5252add-5cab-4a62-ad5a-5b5d554ed0da-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-cf8469d7b-cz9bm\" (UID: \"f5252add-5cab-4a62-ad5a-5b5d554ed0da\") " pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-cz9bm"
Oct 31 01:08:14.626152 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:08:14.626122    4964 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Oct 31 01:08:14.626152 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:08:14.626135    4964 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Oct 31 01:08:14.626152 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:08:14.626134    4964 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-8rtl9\""
Oct 31 01:08:14.689880 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:08:14.689857    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/f5252add-5cab-4a62-ad5a-5b5d554ed0da-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-cf8469d7b-cz9bm\" (UID: \"f5252add-5cab-4a62-ad5a-5b5d554ed0da\") " pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-cz9bm"
Oct 31 01:08:14.689996 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:08:14.689948    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n5rdj\" (UniqueName: \"kubernetes.io/projected/f5252add-5cab-4a62-ad5a-5b5d554ed0da-kube-api-access-n5rdj\") pod \"kuadrant-operator-controller-manager-cf8469d7b-cz9bm\" (UID: \"f5252add-5cab-4a62-ad5a-5b5d554ed0da\") " pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-cz9bm"
Oct 31 01:08:14.690234 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:08:14.690215    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/f5252add-5cab-4a62-ad5a-5b5d554ed0da-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-cf8469d7b-cz9bm\" (UID: \"f5252add-5cab-4a62-ad5a-5b5d554ed0da\") " pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-cz9bm"
Oct 31 01:08:14.748148 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:08:14.748117    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n5rdj\" (UniqueName: \"kubernetes.io/projected/f5252add-5cab-4a62-ad5a-5b5d554ed0da-kube-api-access-n5rdj\") pod \"kuadrant-operator-controller-manager-cf8469d7b-cz9bm\" (UID: \"f5252add-5cab-4a62-ad5a-5b5d554ed0da\") " pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-cz9bm"
Oct 31 01:08:14.878416 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:08:14.878308    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-cz9bm"
Oct 31 01:08:15.254536 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:08:15.254507    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:08:15.382012 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:08:15.381990    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-cz9bm"]
Oct 31 01:08:15.385923 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:08:15.385895    4964 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5252add_5cab_4a62_ad5a_5b5d554ed0da.slice/crio-c0b57d7b3b5c55db97900cd85cc0c847de6ba2efe208a4b0cc30c4e0f1c5b14e WatchSource:0}: Error finding container c0b57d7b3b5c55db97900cd85cc0c847de6ba2efe208a4b0cc30c4e0f1c5b14e: Status 404 returned error can't find the container with id c0b57d7b3b5c55db97900cd85cc0c847de6ba2efe208a4b0cc30c4e0f1c5b14e
Oct 31 01:08:15.816013 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:08:15.815978    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:08:16.060623 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:08:16.060594    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-cz9bm" event={"ID":"f5252add-5cab-4a62-ad5a-5b5d554ed0da","Type":"ContainerStarted","Data":"c0b57d7b3b5c55db97900cd85cc0c847de6ba2efe208a4b0cc30c4e0f1c5b14e"}
Oct 31 01:08:16.958949 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:08:16.958912    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:08:17.015562 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:08:17.015539    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-559fd7ff56-m5c2x"
Oct 31 01:08:17.322724 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:08:17.322690    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:08:18.372936 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:08:18.372901    4964 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-8557cbdb84-2nnh7"]
Oct 31 01:08:18.378401 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:08:18.378375    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-2nnh7"
Oct 31 01:08:18.421257 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:08:18.421220    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g5dxj\" (UniqueName: \"kubernetes.io/projected/9b628e58-c287-4716-adbb-6bbdf0dad9fb-kube-api-access-g5dxj\") pod \"dns-operator-controller-manager-8557cbdb84-2nnh7\" (UID: \"9b628e58-c287-4716-adbb-6bbdf0dad9fb\") " pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-2nnh7"
Oct 31 01:08:18.484413 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:08:18.484381    4964 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\""
Oct 31 01:08:18.484562 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:08:18.484428    4964 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-sg7b7\""
Oct 31 01:08:18.486442 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:08:18.486419    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-8557cbdb84-2nnh7"]
Oct 31 01:08:18.522009 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:08:18.521977    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g5dxj\" (UniqueName: \"kubernetes.io/projected/9b628e58-c287-4716-adbb-6bbdf0dad9fb-kube-api-access-g5dxj\") pod \"dns-operator-controller-manager-8557cbdb84-2nnh7\" (UID: \"9b628e58-c287-4716-adbb-6bbdf0dad9fb\") " pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-2nnh7"
Oct 31 01:08:18.589782 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:08:18.589749    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g5dxj\" (UniqueName: \"kubernetes.io/projected/9b628e58-c287-4716-adbb-6bbdf0dad9fb-kube-api-access-g5dxj\") pod \"dns-operator-controller-manager-8557cbdb84-2nnh7\" (UID: \"9b628e58-c287-4716-adbb-6bbdf0dad9fb\") " pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-2nnh7"
Oct 31 01:08:18.693614 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:08:18.693536    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-2nnh7"
Oct 31 01:08:19.092626 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:08:19.092605    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-8557cbdb84-2nnh7"]
Oct 31 01:08:19.095117 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:08:19.095089    4964 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b628e58_c287_4716_adbb_6bbdf0dad9fb.slice/crio-360ee19128e7f9dd6931229d6af4255acd250da542f5917d23d3f4e4715221fa WatchSource:0}: Error finding container 360ee19128e7f9dd6931229d6af4255acd250da542f5917d23d3f4e4715221fa: Status 404 returned error can't find the container with id 360ee19128e7f9dd6931229d6af4255acd250da542f5917d23d3f4e4715221fa
Oct 31 01:08:19.469806 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:08:19.469774    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:08:20.080961 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:08:20.080932    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-2nnh7" event={"ID":"9b628e58-c287-4716-adbb-6bbdf0dad9fb","Type":"ContainerStarted","Data":"360ee19128e7f9dd6931229d6af4255acd250da542f5917d23d3f4e4715221fa"}
Oct 31 01:08:20.848742 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:08:20.848652    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:08:21.414366 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:08:21.414337    4964 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5778bb4449-svrmp"]
Oct 31 01:08:21.416503 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:08:21.416487    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-svrmp"
Oct 31 01:08:21.470423 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:08:21.470401    4964 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-84ndz\""
Oct 31 01:08:21.520865 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:08:21.520837    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5778bb4449-svrmp"]
Oct 31 01:08:21.551662 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:08:21.551595    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gzhqz\" (UniqueName: \"kubernetes.io/projected/25e6b2b1-a331-483e-86e8-8585418a8e37-kube-api-access-gzhqz\") pod \"limitador-operator-controller-manager-5778bb4449-svrmp\" (UID: \"25e6b2b1-a331-483e-86e8-8585418a8e37\") " pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-svrmp"
Oct 31 01:08:21.652540 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:08:21.652510    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gzhqz\" (UniqueName: \"kubernetes.io/projected/25e6b2b1-a331-483e-86e8-8585418a8e37-kube-api-access-gzhqz\") pod \"limitador-operator-controller-manager-5778bb4449-svrmp\" (UID: \"25e6b2b1-a331-483e-86e8-8585418a8e37\") " pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-svrmp"
Oct 31 01:08:21.713050 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:08:21.713016    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gzhqz\" (UniqueName: \"kubernetes.io/projected/25e6b2b1-a331-483e-86e8-8585418a8e37-kube-api-access-gzhqz\") pod \"limitador-operator-controller-manager-5778bb4449-svrmp\" (UID: \"25e6b2b1-a331-483e-86e8-8585418a8e37\") " pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-svrmp"
Oct 31 01:08:21.727758 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:08:21.727736    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-svrmp"
Oct 31 01:08:22.028069 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:08:22.028036    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:08:22.095341 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:08:22.094894    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-cz9bm" event={"ID":"f5252add-5cab-4a62-ad5a-5b5d554ed0da","Type":"ContainerStarted","Data":"6b6b69c80f48330af87066f017de3db0683053c33bc3873c055e03c717e7b40a"}
Oct 31 01:08:22.095341 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:08:22.094990    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-cz9bm"
Oct 31 01:08:22.112158 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:08:22.112137    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5778bb4449-svrmp"]
Oct 31 01:08:22.115096 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:08:22.115074    4964 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod25e6b2b1_a331_483e_86e8_8585418a8e37.slice/crio-46ea1ef6af46ca39a62a3c5decd4ee52a99c059d54b3d5d10faaeadc8bab163d WatchSource:0}: Error finding container 46ea1ef6af46ca39a62a3c5decd4ee52a99c059d54b3d5d10faaeadc8bab163d: Status 404 returned error can't find the container with id 46ea1ef6af46ca39a62a3c5decd4ee52a99c059d54b3d5d10faaeadc8bab163d
Oct 31 01:08:22.215716 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:08:22.215669    4964 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-cz9bm" podStartSLOduration=2.316509839 podStartE2EDuration="8.21565577s" totalImagesPullingTime="5.899145931s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:08:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:08:22.215039043 +0000 UTC m=+270.691605422" watchObservedRunningTime="2026-10-31 01:08:22.21565577 +0000 UTC m=+270.692222150"
Oct 31 01:08:22.633205 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:08:22.633155    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:08:22.973167 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:08:22.973138    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:08:23.101577 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:08:23.101542    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-svrmp" event={"ID":"25e6b2b1-a331-483e-86e8-8585418a8e37","Type":"ContainerStarted","Data":"46ea1ef6af46ca39a62a3c5decd4ee52a99c059d54b3d5d10faaeadc8bab163d"}
Oct 31 01:08:23.103692 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:08:23.103620    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-2nnh7" event={"ID":"9b628e58-c287-4716-adbb-6bbdf0dad9fb","Type":"ContainerStarted","Data":"fe69e2c5f1558b48884dfbe35476493b8d973726622aba8ec5601e8c85e2c23e"}
Oct 31 01:08:23.221949 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:08:23.221900    4964 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-2nnh7" podStartSLOduration=1.358123075 podStartE2EDuration="5.221887227s" totalImagesPullingTime="3.863764152s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:08:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:08:23.221264262 +0000 UTC m=+271.697830641" watchObservedRunningTime="2026-10-31 01:08:23.221887227 +0000 UTC m=+271.698453605"
Oct 31 01:08:24.109637 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:08:24.109607    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-2nnh7"
Oct 31 01:08:25.115589 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:08:25.115555    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-svrmp" event={"ID":"25e6b2b1-a331-483e-86e8-8585418a8e37","Type":"ContainerStarted","Data":"38ad8283c5e47fc6da6e8b061aab1701abd196b80f8d8b3e38138f089fc097e5"}
Oct 31 01:08:25.127370 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:08:25.127344    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:08:25.235957 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:08:25.235910    4964 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-svrmp" podStartSLOduration=1.7872382770000002 podStartE2EDuration="4.23589815s" totalImagesPullingTime="2.448659873s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:08:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:08:25.234170146 +0000 UTC m=+273.710736523" watchObservedRunningTime="2026-10-31 01:08:25.23589815 +0000 UTC m=+273.712464530"
Oct 31 01:08:25.318692 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:08:25.318664    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:08:26.120246 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:08:26.120222    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-svrmp"
Oct 31 01:08:26.739570 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:08:26.739536    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:08:26.964172 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:08:26.964133    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:08:28.626099 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:08:28.626066    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:08:28.865796 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:08:28.865760    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:08:29.231528 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:08:29.231491    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:08:29.422217 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:08:29.422184    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:08:29.611452 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:08:29.611369    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:08:29.802997 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:08:29.802967    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:08:33.106436 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:08:33.106404    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-cz9bm"
Oct 31 01:08:33.572272 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:08:33.572239    4964 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-9mrpk"]
Oct 31 01:08:33.619250 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:08:33.619225    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-9mrpk"
Oct 31 01:08:33.654772 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:08:33.654746    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-9mrpk"]
Oct 31 01:08:33.755456 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:08:33.755426    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-65zgs\" (UniqueName: \"kubernetes.io/projected/0446b6be-06eb-4fde-858b-a36f492f2209-kube-api-access-65zgs\") pod \"kuadrant-operator-controller-manager-678d87766f-9mrpk\" (UID: \"0446b6be-06eb-4fde-858b-a36f492f2209\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-9mrpk"
Oct 31 01:08:33.755597 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:08:33.755480    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/0446b6be-06eb-4fde-858b-a36f492f2209-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-9mrpk\" (UID: \"0446b6be-06eb-4fde-858b-a36f492f2209\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-9mrpk"
Oct 31 01:08:33.856577 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:08:33.856503    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/0446b6be-06eb-4fde-858b-a36f492f2209-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-9mrpk\" (UID: \"0446b6be-06eb-4fde-858b-a36f492f2209\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-9mrpk"
Oct 31 01:08:33.856720 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:08:33.856589    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-65zgs\" (UniqueName: \"kubernetes.io/projected/0446b6be-06eb-4fde-858b-a36f492f2209-kube-api-access-65zgs\") pod \"kuadrant-operator-controller-manager-678d87766f-9mrpk\" (UID: \"0446b6be-06eb-4fde-858b-a36f492f2209\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-9mrpk"
Oct 31 01:08:33.856974 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:08:33.856955    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/0446b6be-06eb-4fde-858b-a36f492f2209-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-9mrpk\" (UID: \"0446b6be-06eb-4fde-858b-a36f492f2209\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-9mrpk"
Oct 31 01:08:33.922368 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:08:33.922333    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-65zgs\" (UniqueName: \"kubernetes.io/projected/0446b6be-06eb-4fde-858b-a36f492f2209-kube-api-access-65zgs\") pod \"kuadrant-operator-controller-manager-678d87766f-9mrpk\" (UID: \"0446b6be-06eb-4fde-858b-a36f492f2209\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-9mrpk"
Oct 31 01:08:33.946002 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:08:33.945977    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-9mrpk"
Oct 31 01:08:34.440911 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:08:34.440887    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-9mrpk"]
Oct 31 01:08:34.443724 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:08:34.443688    4964 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0446b6be_06eb_4fde_858b_a36f492f2209.slice/crio-ecd5860194ead63517744abe832407baa685fe34ddb160e14d4f7514607ec76a WatchSource:0}: Error finding container ecd5860194ead63517744abe832407baa685fe34ddb160e14d4f7514607ec76a: Status 404 returned error can't find the container with id ecd5860194ead63517744abe832407baa685fe34ddb160e14d4f7514607ec76a
Oct 31 01:08:35.117899 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:08:35.117870    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-2nnh7"
Oct 31 01:08:35.164061 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:08:35.164027    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-9mrpk" event={"ID":"0446b6be-06eb-4fde-858b-a36f492f2209","Type":"ContainerStarted","Data":"a617df6b4920ed5431f39a4374361e3cd28b5cc326ab1c9f232efe796f69d928"}
Oct 31 01:08:35.164217 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:08:35.164084    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-9mrpk" event={"ID":"0446b6be-06eb-4fde-858b-a36f492f2209","Type":"ContainerStarted","Data":"ecd5860194ead63517744abe832407baa685fe34ddb160e14d4f7514607ec76a"}
Oct 31 01:08:35.164217 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:08:35.164120    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-9mrpk"
Oct 31 01:08:35.359354 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:08:35.359291    4964 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-9mrpk" podStartSLOduration=2.359277058 podStartE2EDuration="2.359277058s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:08:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:08:35.357158485 +0000 UTC m=+283.833724866" watchObservedRunningTime="2026-10-31 01:08:35.359277058 +0000 UTC m=+283.835843437"
Oct 31 01:08:35.509727 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:08:35.509697    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:08:35.857552 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:08:35.857453    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:08:37.126762 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:08:37.126736    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-svrmp"
Oct 31 01:08:40.304129 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:08:40.304089    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:08:40.495669 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:08:40.495636    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:08:41.236057 ip-10-0-133-197 kubenswrapper[4964]: E1031 01:08:41.236026    4964 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q56x5" containerName="extractor"
Oct 31 01:08:42.412729 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:08:42.412680    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:08:45.034814 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:08:45.034777    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:08:45.232431 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:08:45.232393    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:08:46.171426 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:08:46.171397    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-9mrpk"
Oct 31 01:08:46.391778 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:08:46.391746    4964 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-cz9bm"]
Oct 31 01:08:46.392006 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:08:46.391973    4964 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-cz9bm" podUID="f5252add-5cab-4a62-ad5a-5b5d554ed0da" containerName="manager" containerID="cri-o://6b6b69c80f48330af87066f017de3db0683053c33bc3873c055e03c717e7b40a" gracePeriod=10
Oct 31 01:08:46.630395 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:08:46.630373    4964 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-cz9bm"
Oct 31 01:08:46.756023 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:08:46.755947    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f5252add-5cab-4a62-ad5a-5b5d554ed0da-extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/f5252add-5cab-4a62-ad5a-5b5d554ed0da-extensions-socket-volume\") pod \"f5252add-5cab-4a62-ad5a-5b5d554ed0da\" (UID: \"f5252add-5cab-4a62-ad5a-5b5d554ed0da\") "
Oct 31 01:08:46.756023 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:08:46.756007    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f5252add-5cab-4a62-ad5a-5b5d554ed0da-kube-api-access-n5rdj\" (UniqueName: \"kubernetes.io/projected/f5252add-5cab-4a62-ad5a-5b5d554ed0da-kube-api-access-n5rdj\") pod \"f5252add-5cab-4a62-ad5a-5b5d554ed0da\" (UID: \"f5252add-5cab-4a62-ad5a-5b5d554ed0da\") "
Oct 31 01:08:46.756436 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:08:46.756413    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5252add-5cab-4a62-ad5a-5b5d554ed0da-extensions-socket-volume" pod "f5252add-5cab-4a62-ad5a-5b5d554ed0da" (UID: "f5252add-5cab-4a62-ad5a-5b5d554ed0da"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:08:46.758564 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:08:46.758543    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f5252add-5cab-4a62-ad5a-5b5d554ed0da-kube-api-access-n5rdj" pod "f5252add-5cab-4a62-ad5a-5b5d554ed0da" (UID: "f5252add-5cab-4a62-ad5a-5b5d554ed0da"). InnerVolumeSpecName "kube-api-access-n5rdj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:08:46.857563 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:08:46.857534    4964 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/f5252add-5cab-4a62-ad5a-5b5d554ed0da-extensions-socket-volume\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:08:46.857563 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:08:46.857560    4964 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n5rdj\" (UniqueName: \"kubernetes.io/projected/f5252add-5cab-4a62-ad5a-5b5d554ed0da-kube-api-access-n5rdj\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:08:47.221468 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:08:47.221432    4964 generic.go:356] "Generic (PLEG): container finished" podID="f5252add-5cab-4a62-ad5a-5b5d554ed0da" containerID="6b6b69c80f48330af87066f017de3db0683053c33bc3873c055e03c717e7b40a" exitCode=0
Oct 31 01:08:47.221874 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:08:47.221485    4964 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-cz9bm"
Oct 31 01:08:47.221874 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:08:47.221517    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-cz9bm" event={"ID":"f5252add-5cab-4a62-ad5a-5b5d554ed0da","Type":"ContainerDied","Data":"6b6b69c80f48330af87066f017de3db0683053c33bc3873c055e03c717e7b40a"}
Oct 31 01:08:47.221874 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:08:47.221574    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-cz9bm" event={"ID":"f5252add-5cab-4a62-ad5a-5b5d554ed0da","Type":"ContainerDied","Data":"c0b57d7b3b5c55db97900cd85cc0c847de6ba2efe208a4b0cc30c4e0f1c5b14e"}
Oct 31 01:08:47.221874 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:08:47.221596    4964 scope.go:122] "RemoveContainer" containerID="6b6b69c80f48330af87066f017de3db0683053c33bc3873c055e03c717e7b40a"
Oct 31 01:08:47.230421 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:08:47.230400    4964 scope.go:122] "RemoveContainer" containerID="6b6b69c80f48330af87066f017de3db0683053c33bc3873c055e03c717e7b40a"
Oct 31 01:08:47.230677 ip-10-0-133-197 kubenswrapper[4964]: E1031 01:08:47.230657    4964 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6b6b69c80f48330af87066f017de3db0683053c33bc3873c055e03c717e7b40a\": container with ID starting with 6b6b69c80f48330af87066f017de3db0683053c33bc3873c055e03c717e7b40a not found: ID does not exist" containerID="6b6b69c80f48330af87066f017de3db0683053c33bc3873c055e03c717e7b40a"
Oct 31 01:08:47.230744 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:08:47.230690    4964 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6b6b69c80f48330af87066f017de3db0683053c33bc3873c055e03c717e7b40a"} err="failed to get container status \"6b6b69c80f48330af87066f017de3db0683053c33bc3873c055e03c717e7b40a\": rpc error: code = NotFound desc = could not find container \"6b6b69c80f48330af87066f017de3db0683053c33bc3873c055e03c717e7b40a\": container with ID starting with 6b6b69c80f48330af87066f017de3db0683053c33bc3873c055e03c717e7b40a not found: ID does not exist"
Oct 31 01:08:47.400496 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:08:47.400470    4964 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-cz9bm"]
Oct 31 01:08:47.449976 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:08:47.449945    4964 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-cz9bm"]
Oct 31 01:08:47.774750 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:08:47.774716    4964 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f5252add-5cab-4a62-ad5a-5b5d554ed0da" path="/var/lib/kubelet/pods/f5252add-5cab-4a62-ad5a-5b5d554ed0da/volumes"
Oct 31 01:08:49.405519 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:08:49.405485    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:08:49.665158 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:08:49.665041    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:08:50.030798 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:08:50.030764    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:08:50.228898 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:08:50.228865    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:08:51.793185 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:08:51.793163    4964 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-9257n_f0c0ada7-7688-4dc2-97fc-813e4d0c4ef3/console-operator/2.log"
Oct 31 01:08:51.793561 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:08:51.793198    4964 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-9257n_f0c0ada7-7688-4dc2-97fc-813e4d0c4ef3/console-operator/2.log"
Oct 31 01:08:51.803773 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:08:51.803750    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:08:52.009087 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:08:52.009059    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:08:52.342605 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:08:52.342575    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:08:52.353645 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:08:52.353623    4964 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6b748b6585wg7kb"]
Oct 31 01:08:52.354563 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:08:52.354136    4964 cpu_manager.go:414] "containerMap: removing container" podUID="f5252add-5cab-4a62-ad5a-5b5d554ed0da" containerName="manager"
Oct 31 01:08:52.354563 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:08:52.354159    4964 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f5252add-5cab-4a62-ad5a-5b5d554ed0da" containerName="manager"
Oct 31 01:08:52.354563 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:08:52.354281    4964 memory_manager.go:365] "RemoveStaleState removing state" podUID="f5252add-5cab-4a62-ad5a-5b5d554ed0da" containerName="manager"
Oct 31 01:08:52.409067 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:08:52.409047    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6b748b6585wg7kb"
Oct 31 01:08:52.419531 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:08:52.419510    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6b748b6585wg7kb"]
Oct 31 01:08:52.466444 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:08:52.466425    4964 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-kjzbk\""
Oct 31 01:08:52.536576 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:08:52.536548    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:08:52.601849 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:08:52.601790    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5b65b0b2-79a7-4116-a40e-78e1414c0071-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6b748b6585wg7kb\" (UID: \"5b65b0b2-79a7-4116-a40e-78e1414c0071\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6b748b6585wg7kb"
Oct 31 01:08:52.601849 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:08:52.601831    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9nr24\" (UniqueName: \"kubernetes.io/projected/5b65b0b2-79a7-4116-a40e-78e1414c0071-kube-api-access-9nr24\") pod \"data-science-gateway-data-science-gateway-class-6b748b6585wg7kb\" (UID: \"5b65b0b2-79a7-4116-a40e-78e1414c0071\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6b748b6585wg7kb"
Oct 31 01:08:52.601849 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:08:52.601848    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5b65b0b2-79a7-4116-a40e-78e1414c0071-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6b748b6585wg7kb\" (UID: \"5b65b0b2-79a7-4116-a40e-78e1414c0071\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6b748b6585wg7kb"
Oct 31 01:08:52.602053 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:08:52.601991    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5b65b0b2-79a7-4116-a40e-78e1414c0071-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6b748b6585wg7kb\" (UID: \"5b65b0b2-79a7-4116-a40e-78e1414c0071\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6b748b6585wg7kb"
Oct 31 01:08:52.602053 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:08:52.602024    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b65b0b2-79a7-4116-a40e-78e1414c0071-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6b748b6585wg7kb\" (UID: \"5b65b0b2-79a7-4116-a40e-78e1414c0071\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6b748b6585wg7kb"
Oct 31 01:08:52.602053 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:08:52.602042    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5b65b0b2-79a7-4116-a40e-78e1414c0071-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6b748b6585wg7kb\" (UID: \"5b65b0b2-79a7-4116-a40e-78e1414c0071\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6b748b6585wg7kb"
Oct 31 01:08:52.602178 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:08:52.602095    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b65b0b2-79a7-4116-a40e-78e1414c0071-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6b748b6585wg7kb\" (UID: \"5b65b0b2-79a7-4116-a40e-78e1414c0071\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6b748b6585wg7kb"
Oct 31 01:08:52.602178 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:08:52.602134    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5b65b0b2-79a7-4116-a40e-78e1414c0071-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6b748b6585wg7kb\" (UID: \"5b65b0b2-79a7-4116-a40e-78e1414c0071\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6b748b6585wg7kb"
Oct 31 01:08:52.602243 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:08:52.602201    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5b65b0b2-79a7-4116-a40e-78e1414c0071-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6b748b6585wg7kb\" (UID: \"5b65b0b2-79a7-4116-a40e-78e1414c0071\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6b748b6585wg7kb"
Oct 31 01:08:52.703200 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:08:52.703173    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9nr24\" (UniqueName: \"kubernetes.io/projected/5b65b0b2-79a7-4116-a40e-78e1414c0071-kube-api-access-9nr24\") pod \"data-science-gateway-data-science-gateway-class-6b748b6585wg7kb\" (UID: \"5b65b0b2-79a7-4116-a40e-78e1414c0071\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6b748b6585wg7kb"
Oct 31 01:08:52.703200 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:08:52.703201    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5b65b0b2-79a7-4116-a40e-78e1414c0071-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6b748b6585wg7kb\" (UID: \"5b65b0b2-79a7-4116-a40e-78e1414c0071\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6b748b6585wg7kb"
Oct 31 01:08:52.703422 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:08:52.703230    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5b65b0b2-79a7-4116-a40e-78e1414c0071-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6b748b6585wg7kb\" (UID: \"5b65b0b2-79a7-4116-a40e-78e1414c0071\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6b748b6585wg7kb"
Oct 31 01:08:52.703422 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:08:52.703251    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b65b0b2-79a7-4116-a40e-78e1414c0071-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6b748b6585wg7kb\" (UID: \"5b65b0b2-79a7-4116-a40e-78e1414c0071\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6b748b6585wg7kb"
Oct 31 01:08:52.703422 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:08:52.703266    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5b65b0b2-79a7-4116-a40e-78e1414c0071-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6b748b6585wg7kb\" (UID: \"5b65b0b2-79a7-4116-a40e-78e1414c0071\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6b748b6585wg7kb"
Oct 31 01:08:52.703422 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:08:52.703282    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b65b0b2-79a7-4116-a40e-78e1414c0071-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6b748b6585wg7kb\" (UID: \"5b65b0b2-79a7-4116-a40e-78e1414c0071\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6b748b6585wg7kb"
Oct 31 01:08:52.703422 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:08:52.703367    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5b65b0b2-79a7-4116-a40e-78e1414c0071-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6b748b6585wg7kb\" (UID: \"5b65b0b2-79a7-4116-a40e-78e1414c0071\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6b748b6585wg7kb"
Oct 31 01:08:52.703422 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:08:52.703418    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5b65b0b2-79a7-4116-a40e-78e1414c0071-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6b748b6585wg7kb\" (UID: \"5b65b0b2-79a7-4116-a40e-78e1414c0071\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6b748b6585wg7kb"
Oct 31 01:08:52.703732 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:08:52.703462    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5b65b0b2-79a7-4116-a40e-78e1414c0071-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6b748b6585wg7kb\" (UID: \"5b65b0b2-79a7-4116-a40e-78e1414c0071\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6b748b6585wg7kb"
Oct 31 01:08:52.703732 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:08:52.703634    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b65b0b2-79a7-4116-a40e-78e1414c0071-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6b748b6585wg7kb\" (UID: \"5b65b0b2-79a7-4116-a40e-78e1414c0071\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6b748b6585wg7kb"
Oct 31 01:08:52.703732 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:08:52.703690    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5b65b0b2-79a7-4116-a40e-78e1414c0071-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6b748b6585wg7kb\" (UID: \"5b65b0b2-79a7-4116-a40e-78e1414c0071\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6b748b6585wg7kb"
Oct 31 01:08:52.703893 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:08:52.703770    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5b65b0b2-79a7-4116-a40e-78e1414c0071-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6b748b6585wg7kb\" (UID: \"5b65b0b2-79a7-4116-a40e-78e1414c0071\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6b748b6585wg7kb"
Oct 31 01:08:52.703893 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:08:52.703823    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b65b0b2-79a7-4116-a40e-78e1414c0071-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6b748b6585wg7kb\" (UID: \"5b65b0b2-79a7-4116-a40e-78e1414c0071\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6b748b6585wg7kb"
Oct 31 01:08:52.704266 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:08:52.704243    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5b65b0b2-79a7-4116-a40e-78e1414c0071-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6b748b6585wg7kb\" (UID: \"5b65b0b2-79a7-4116-a40e-78e1414c0071\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6b748b6585wg7kb"
Oct 31 01:08:52.705694 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:08:52.705674    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5b65b0b2-79a7-4116-a40e-78e1414c0071-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6b748b6585wg7kb\" (UID: \"5b65b0b2-79a7-4116-a40e-78e1414c0071\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6b748b6585wg7kb"
Oct 31 01:08:52.706022 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:08:52.706001    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5b65b0b2-79a7-4116-a40e-78e1414c0071-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6b748b6585wg7kb\" (UID: \"5b65b0b2-79a7-4116-a40e-78e1414c0071\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6b748b6585wg7kb"
Oct 31 01:08:52.730870 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:08:52.730850    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:08:52.761475 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:08:52.761452    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9nr24\" (UniqueName: \"kubernetes.io/projected/5b65b0b2-79a7-4116-a40e-78e1414c0071-kube-api-access-9nr24\") pod \"data-science-gateway-data-science-gateway-class-6b748b6585wg7kb\" (UID: \"5b65b0b2-79a7-4116-a40e-78e1414c0071\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6b748b6585wg7kb"
Oct 31 01:08:52.761783 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:08:52.761766    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5b65b0b2-79a7-4116-a40e-78e1414c0071-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6b748b6585wg7kb\" (UID: \"5b65b0b2-79a7-4116-a40e-78e1414c0071\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6b748b6585wg7kb"
Oct 31 01:08:53.037158 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:08:53.037125    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6b748b6585wg7kb"
Oct 31 01:08:53.494310 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:08:53.494287    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6b748b6585wg7kb"]
Oct 31 01:08:53.496675 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:08:53.496641    4964 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b65b0b2_79a7_4116_a40e_78e1414c0071.slice/crio-f28bf2169158ae84099cc8a528e7ddaf3516c5848d89d31965e3af77375a28e4 WatchSource:0}: Error finding container f28bf2169158ae84099cc8a528e7ddaf3516c5848d89d31965e3af77375a28e4: Status 404 returned error can't find the container with id f28bf2169158ae84099cc8a528e7ddaf3516c5848d89d31965e3af77375a28e4
Oct 31 01:08:53.498774 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:08:53.498741    4964 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 01:08:53.498851 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:08:53.498806    4964 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 01:08:53.498851 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:08:53.498833    4964 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 01:08:54.261178 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:08:54.261141    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6b748b6585wg7kb" event={"ID":"5b65b0b2-79a7-4116-a40e-78e1414c0071","Type":"ContainerStarted","Data":"97a99df4f8d2d1b864b270f91cc038294a5114e1533391133f22e9eefaaeac46"}
Oct 31 01:08:54.261576 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:08:54.261220    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6b748b6585wg7kb" event={"ID":"5b65b0b2-79a7-4116-a40e-78e1414c0071","Type":"ContainerStarted","Data":"f28bf2169158ae84099cc8a528e7ddaf3516c5848d89d31965e3af77375a28e4"}
Oct 31 01:08:54.429985 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:08:54.429935    4964 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6b748b6585wg7kb" podStartSLOduration=2.42992157 podStartE2EDuration="2.42992157s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:08:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:08:54.428284942 +0000 UTC m=+302.904851354" watchObservedRunningTime="2026-10-31 01:08:54.42992157 +0000 UTC m=+302.906487949"
Oct 31 01:08:54.782269 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:08:54.782237    4964 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-66db79789b-2jk8v"]
Oct 31 01:08:54.784506 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:08:54.784484    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-controller-manager-66db79789b-2jk8v"
Oct 31 01:08:54.837093 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:08:54.837066    4964 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"webhook-server-cert\""
Oct 31 01:08:54.837093 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:08:54.837086    4964 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"metrics-server-cert\""
Oct 31 01:08:54.837256 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:08:54.837171    4964 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"jobset-controller-manager-dockercfg-59pwr\""
Oct 31 01:08:54.886305 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:08:54.886280    4964 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"openshift-service-ca.crt\""
Oct 31 01:08:54.886305 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:08:54.886310    4964 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"kube-root-ca.crt\""
Oct 31 01:08:54.886480 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:08:54.886346    4964 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"jobset-manager-config\""
Oct 31 01:08:54.888015 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:08:54.887990    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-66db79789b-2jk8v"]
Oct 31 01:08:54.921268 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:08:54.921247    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ec0a1197-55db-4519-9e0e-670c36ff799c-cert\") pod \"jobset-controller-manager-66db79789b-2jk8v\" (UID: \"ec0a1197-55db-4519-9e0e-670c36ff799c\") " pod="openshift-jobset-operator/jobset-controller-manager-66db79789b-2jk8v"
Oct 31 01:08:54.921411 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:08:54.921301    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vzv74\" (UniqueName: \"kubernetes.io/projected/ec0a1197-55db-4519-9e0e-670c36ff799c-kube-api-access-vzv74\") pod \"jobset-controller-manager-66db79789b-2jk8v\" (UID: \"ec0a1197-55db-4519-9e0e-670c36ff799c\") " pod="openshift-jobset-operator/jobset-controller-manager-66db79789b-2jk8v"
Oct 31 01:08:54.921515 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:08:54.921493    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/ec0a1197-55db-4519-9e0e-670c36ff799c-manager-config\") pod \"jobset-controller-manager-66db79789b-2jk8v\" (UID: \"ec0a1197-55db-4519-9e0e-670c36ff799c\") " pod="openshift-jobset-operator/jobset-controller-manager-66db79789b-2jk8v"
Oct 31 01:08:54.921571 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:08:54.921555    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ec0a1197-55db-4519-9e0e-670c36ff799c-metrics-certs\") pod \"jobset-controller-manager-66db79789b-2jk8v\" (UID: \"ec0a1197-55db-4519-9e0e-670c36ff799c\") " pod="openshift-jobset-operator/jobset-controller-manager-66db79789b-2jk8v"
Oct 31 01:08:55.022381 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:08:55.022360    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ec0a1197-55db-4519-9e0e-670c36ff799c-cert\") pod \"jobset-controller-manager-66db79789b-2jk8v\" (UID: \"ec0a1197-55db-4519-9e0e-670c36ff799c\") " pod="openshift-jobset-operator/jobset-controller-manager-66db79789b-2jk8v"
Oct 31 01:08:55.022476 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:08:55.022406    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vzv74\" (UniqueName: \"kubernetes.io/projected/ec0a1197-55db-4519-9e0e-670c36ff799c-kube-api-access-vzv74\") pod \"jobset-controller-manager-66db79789b-2jk8v\" (UID: \"ec0a1197-55db-4519-9e0e-670c36ff799c\") " pod="openshift-jobset-operator/jobset-controller-manager-66db79789b-2jk8v"
Oct 31 01:08:55.022476 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:08:55.022427    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/ec0a1197-55db-4519-9e0e-670c36ff799c-manager-config\") pod \"jobset-controller-manager-66db79789b-2jk8v\" (UID: \"ec0a1197-55db-4519-9e0e-670c36ff799c\") " pod="openshift-jobset-operator/jobset-controller-manager-66db79789b-2jk8v"
Oct 31 01:08:55.022476 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:08:55.022453    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ec0a1197-55db-4519-9e0e-670c36ff799c-metrics-certs\") pod \"jobset-controller-manager-66db79789b-2jk8v\" (UID: \"ec0a1197-55db-4519-9e0e-670c36ff799c\") " pod="openshift-jobset-operator/jobset-controller-manager-66db79789b-2jk8v"
Oct 31 01:08:55.023134 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:08:55.023114    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/ec0a1197-55db-4519-9e0e-670c36ff799c-manager-config\") pod \"jobset-controller-manager-66db79789b-2jk8v\" (UID: \"ec0a1197-55db-4519-9e0e-670c36ff799c\") " pod="openshift-jobset-operator/jobset-controller-manager-66db79789b-2jk8v"
Oct 31 01:08:55.024892 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:08:55.024863    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ec0a1197-55db-4519-9e0e-670c36ff799c-cert\") pod \"jobset-controller-manager-66db79789b-2jk8v\" (UID: \"ec0a1197-55db-4519-9e0e-670c36ff799c\") " pod="openshift-jobset-operator/jobset-controller-manager-66db79789b-2jk8v"
Oct 31 01:08:55.024994 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:08:55.024906    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ec0a1197-55db-4519-9e0e-670c36ff799c-metrics-certs\") pod \"jobset-controller-manager-66db79789b-2jk8v\" (UID: \"ec0a1197-55db-4519-9e0e-670c36ff799c\") " pod="openshift-jobset-operator/jobset-controller-manager-66db79789b-2jk8v"
Oct 31 01:08:55.038170 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:08:55.038126    4964 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6b748b6585wg7kb"
Oct 31 01:08:55.042450 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:08:55.042431    4964 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6b748b6585wg7kb"
Oct 31 01:08:55.079377 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:08:55.079352    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vzv74\" (UniqueName: \"kubernetes.io/projected/ec0a1197-55db-4519-9e0e-670c36ff799c-kube-api-access-vzv74\") pod \"jobset-controller-manager-66db79789b-2jk8v\" (UID: \"ec0a1197-55db-4519-9e0e-670c36ff799c\") " pod="openshift-jobset-operator/jobset-controller-manager-66db79789b-2jk8v"
Oct 31 01:08:55.096390 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:08:55.096370    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-controller-manager-66db79789b-2jk8v"
Oct 31 01:08:55.266773 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:08:55.266746    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6b748b6585wg7kb"
Oct 31 01:08:55.267732 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:08:55.267710    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6b748b6585wg7kb"
Oct 31 01:08:55.467048 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:08:55.467024    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-66db79789b-2jk8v"]
Oct 31 01:08:55.468946 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:08:55.468919    4964 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec0a1197_55db_4519_9e0e_670c36ff799c.slice/crio-1c000213c8c12b36093597cac62008d142c35b590304d9de1f0e725b3e1378e9 WatchSource:0}: Error finding container 1c000213c8c12b36093597cac62008d142c35b590304d9de1f0e725b3e1378e9: Status 404 returned error can't find the container with id 1c000213c8c12b36093597cac62008d142c35b590304d9de1f0e725b3e1378e9
Oct 31 01:08:55.470927 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:08:55.470914    4964 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 31 01:08:56.273932 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:08:56.273897    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-controller-manager-66db79789b-2jk8v" event={"ID":"ec0a1197-55db-4519-9e0e-670c36ff799c","Type":"ContainerStarted","Data":"1c000213c8c12b36093597cac62008d142c35b590304d9de1f0e725b3e1378e9"}
Oct 31 01:08:56.419504 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:08:56.419475    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:08:56.650040 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:08:56.649964    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:08:57.052948 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:08:57.052907    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:08:59.166929 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:08:59.166893    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:09:00.295594 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:09:00.295513    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-controller-manager-66db79789b-2jk8v" event={"ID":"ec0a1197-55db-4519-9e0e-670c36ff799c","Type":"ContainerStarted","Data":"16e49eef9b1485ad13aa55b94453c3bd8bdbec5fadb6378b72f744ff28b63e2b"}
Oct 31 01:09:00.295594 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:09:00.295572    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-jobset-operator/jobset-controller-manager-66db79789b-2jk8v"
Oct 31 01:09:00.410340 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:09:00.410271    4964 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-jobset-operator/jobset-controller-manager-66db79789b-2jk8v" podStartSLOduration=2.519458143 podStartE2EDuration="6.410259693s" totalImagesPullingTime="3.89080155s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:08:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:09:00.40906315 +0000 UTC m=+308.885629530" watchObservedRunningTime="2026-10-31 01:09:00.410259693 +0000 UTC m=+308.886826149"
Oct 31 01:09:01.174156 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:09:01.174120    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:09:01.354109 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:09:01.354075    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:09:05.971668 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:09:05.971628    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:09:06.149328 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:09:06.149292    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:09:07.896203 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:09:07.896158    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:09:08.251608 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:09:08.251571    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:09:08.443291 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:09:08.443256    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:09:09.599554 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:09:09.599516    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:09:09.772672 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:09:09.772640    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:09:11.171713 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:09:11.171683    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:09:11.237026 ip-10-0-133-197 kubenswrapper[4964]: E1031 01:09:11.236996    4964 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q56x5" containerName="extractor"
Oct 31 01:09:11.306545 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:09:11.306519    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-jobset-operator/jobset-controller-manager-66db79789b-2jk8v"
Oct 31 01:09:11.350025 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:09:11.350000    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:09:11.572452 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:09:11.572424    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:09:11.753926 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:09:11.753892    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:09:11.946018 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:09:11.945946    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:09:17.272208 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:09:17.272169    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:09:17.446120 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:09:17.446084    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:09:18.215503 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:09:18.215472    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:09:18.382663 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:09:18.382625    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:09:22.982822 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:09:22.982782    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:09:23.214013 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:09:23.213980    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:09:27.785799 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:09:27.785761    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:09:28.080790 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:09:28.080707    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:09:32.575403 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:09:32.575373    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:09:32.766780 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:09:32.766746    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:09:33.338593 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:09:33.338557    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:09:41.236192 ip-10-0-133-197 kubenswrapper[4964]: E1031 01:09:41.236146    4964 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q56x5" containerName="extractor"
Oct 31 01:09:42.761813 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:09:42.761774    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:09:52.786880 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:09:52.786838    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:10:03.615221 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:10:03.615179    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:10:03.841464 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:10:03.841424    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:10:04.083133 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:10:04.083101    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:10:09.093541 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:10:09.093507    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:10:09.263189 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:10:09.263155    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:10:09.450462 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:10:09.450388    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:10:09.620012 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:10:09.619979    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:10:09.820309 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:10:09.820272    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:10:10.000867 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:10:10.000827    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:10:10.190966 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:10:10.190877    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:10:10.288258 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:10:10.288235    4964 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-769875db56-z94np"]
Oct 31 01:10:10.291689 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:10:10.291673    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-769875db56-z94np"
Oct 31 01:10:10.342131 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:10:10.342105    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vmhfc\" (UniqueName: \"kubernetes.io/projected/baeff60c-ce67-45a0-a76b-52220ba1065f-kube-api-access-vmhfc\") pod \"odh-model-controller-769875db56-z94np\" (UID: \"baeff60c-ce67-45a0-a76b-52220ba1065f\") " pod="opendatahub/odh-model-controller-769875db56-z94np"
Oct 31 01:10:10.342254 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:10:10.342166    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/baeff60c-ce67-45a0-a76b-52220ba1065f-metrics-certs\") pod \"odh-model-controller-769875db56-z94np\" (UID: \"baeff60c-ce67-45a0-a76b-52220ba1065f\") " pod="opendatahub/odh-model-controller-769875db56-z94np"
Oct 31 01:10:10.342354 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:10:10.342303    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/baeff60c-ce67-45a0-a76b-52220ba1065f-cert\") pod \"odh-model-controller-769875db56-z94np\" (UID: \"baeff60c-ce67-45a0-a76b-52220ba1065f\") " pod="opendatahub/odh-model-controller-769875db56-z94np"
Oct 31 01:10:10.348813 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:10:10.348784    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-769875db56-z94np"]
Oct 31 01:10:10.396261 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:10:10.396231    4964 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Oct 31 01:10:10.396261 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:10:10.396260    4964 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Oct 31 01:10:10.396493 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:10:10.396303    4964 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\""
Oct 31 01:10:10.396493 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:10:10.396405    4964 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-vmps8\""
Oct 31 01:10:10.396591 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:10:10.396519    4964 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-metrics-tls\""
Oct 31 01:10:10.442866 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:10:10.442797    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/baeff60c-ce67-45a0-a76b-52220ba1065f-cert\") pod \"odh-model-controller-769875db56-z94np\" (UID: \"baeff60c-ce67-45a0-a76b-52220ba1065f\") " pod="opendatahub/odh-model-controller-769875db56-z94np"
Oct 31 01:10:10.442866 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:10:10.442834    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vmhfc\" (UniqueName: \"kubernetes.io/projected/baeff60c-ce67-45a0-a76b-52220ba1065f-kube-api-access-vmhfc\") pod \"odh-model-controller-769875db56-z94np\" (UID: \"baeff60c-ce67-45a0-a76b-52220ba1065f\") " pod="opendatahub/odh-model-controller-769875db56-z94np"
Oct 31 01:10:10.443013 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:10:10.442875    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/baeff60c-ce67-45a0-a76b-52220ba1065f-metrics-certs\") pod \"odh-model-controller-769875db56-z94np\" (UID: \"baeff60c-ce67-45a0-a76b-52220ba1065f\") " pod="opendatahub/odh-model-controller-769875db56-z94np"
Oct 31 01:10:10.445383 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:10:10.445353    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/baeff60c-ce67-45a0-a76b-52220ba1065f-metrics-certs\") pod \"odh-model-controller-769875db56-z94np\" (UID: \"baeff60c-ce67-45a0-a76b-52220ba1065f\") " pod="opendatahub/odh-model-controller-769875db56-z94np"
Oct 31 01:10:10.445570 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:10:10.445386    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/baeff60c-ce67-45a0-a76b-52220ba1065f-cert\") pod \"odh-model-controller-769875db56-z94np\" (UID: \"baeff60c-ce67-45a0-a76b-52220ba1065f\") " pod="opendatahub/odh-model-controller-769875db56-z94np"
Oct 31 01:10:10.503942 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:10:10.503914    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vmhfc\" (UniqueName: \"kubernetes.io/projected/baeff60c-ce67-45a0-a76b-52220ba1065f-kube-api-access-vmhfc\") pod \"odh-model-controller-769875db56-z94np\" (UID: \"baeff60c-ce67-45a0-a76b-52220ba1065f\") " pod="opendatahub/odh-model-controller-769875db56-z94np"
Oct 31 01:10:10.604097 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:10:10.604073    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-769875db56-z94np"
Oct 31 01:10:10.975793 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:10:10.975771    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-769875db56-z94np"]
Oct 31 01:10:10.977980 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:10:10.977950    4964 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbaeff60c_ce67_45a0_a76b_52220ba1065f.slice/crio-7d262c5255f9e77241ae534128f61f8656cf971935e100419a4740c31a42ef04 WatchSource:0}: Error finding container 7d262c5255f9e77241ae534128f61f8656cf971935e100419a4740c31a42ef04: Status 404 returned error can't find the container with id 7d262c5255f9e77241ae534128f61f8656cf971935e100419a4740c31a42ef04
Oct 31 01:10:11.235913 ip-10-0-133-197 kubenswrapper[4964]: E1031 01:10:11.235832    4964 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q56x5" containerName="extractor"
Oct 31 01:10:11.355255 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:10:11.355218    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:10:11.640376 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:10:11.640289    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-769875db56-z94np" event={"ID":"baeff60c-ce67-45a0-a76b-52220ba1065f","Type":"ContainerStarted","Data":"7d262c5255f9e77241ae534128f61f8656cf971935e100419a4740c31a42ef04"}
Oct 31 01:10:11.963559 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:10:11.963472    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:10:12.154990 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:10:12.154950    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:10:14.656414 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:10:14.656379    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-769875db56-z94np" event={"ID":"baeff60c-ce67-45a0-a76b-52220ba1065f","Type":"ContainerStarted","Data":"4995a574c05bed7e519936bba1f11b8ea2d7da094974f9259b2abb0f1e1b6924"}
Oct 31 01:10:14.657004 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:10:14.656604    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-769875db56-z94np"
Oct 31 01:10:14.772155 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:10:14.772109    4964 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/odh-model-controller-769875db56-z94np" podStartSLOduration=1.7879823830000001 podStartE2EDuration="4.772095077s" totalImagesPullingTime="2.984112694s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:10:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:10:14.769979089 +0000 UTC m=+383.246545470" watchObservedRunningTime="2026-10-31 01:10:14.772095077 +0000 UTC m=+383.248661448"
Oct 31 01:10:15.557469 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:10:15.557429    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:10:15.784198 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:10:15.784155    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:10:16.571072 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:10:16.571023    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:10:16.750236 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:10:16.750204    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:10:16.928701 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:10:16.928623    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:10:17.151261 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:10:17.151231    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:10:17.352952 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:10:17.352918    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:10:17.537230 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:10:17.537196    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:10:17.713967 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:10:17.713887    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:10:17.957279 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:10:17.957251    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:10:25.663469 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:10:25.663437    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-769875db56-z94np"
Oct 31 01:10:41.236749 ip-10-0-133-197 kubenswrapper[4964]: E1031 01:10:41.236671    4964 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q56x5" containerName="extractor"
Oct 31 01:10:43.093063 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:10:43.093019    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:10:45.607097 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:10:45.607058    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:10:45.805167 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:10:45.805135    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:10:47.746796 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:10:47.746758    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:10:47.982424 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:10:47.982390    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:10:48.161152 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:10:48.161065    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:10:48.753050 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:10:48.753016    4964 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-6889b8b577-pb2t7"]
Oct 31 01:10:48.756303 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:10:48.756284    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6889b8b577-pb2t7"
Oct 31 01:10:48.858028 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:10:48.857997    4964 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-x26rq\""
Oct 31 01:10:48.858028 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:10:48.858025    4964 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 01:10:48.858028 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:10:48.858025    4964 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Oct 31 01:10:48.858290 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:10:48.857992    4964 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\""
Oct 31 01:10:48.858290 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:10:48.858090    4964 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 01:10:48.860296 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:10:48.860270    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6889b8b577-pb2t7"]
Oct 31 01:10:48.870372 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:10:48.870350    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a5a09bbd-0dcf-4a46-a38b-e88708df6273-openshift-service-ca\") pod \"maas-api-6889b8b577-pb2t7\" (UID: \"a5a09bbd-0dcf-4a46-a38b-e88708df6273\") " pod="odh-ai-gateway-infra/maas-api-6889b8b577-pb2t7"
Oct 31 01:10:48.870452 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:10:48.870378    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a5a09bbd-0dcf-4a46-a38b-e88708df6273-metrics-certs\") pod \"maas-api-6889b8b577-pb2t7\" (UID: \"a5a09bbd-0dcf-4a46-a38b-e88708df6273\") " pod="odh-ai-gateway-infra/maas-api-6889b8b577-pb2t7"
Oct 31 01:10:48.870452 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:10:48.870401    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-67tr5\" (UniqueName: \"kubernetes.io/projected/a5a09bbd-0dcf-4a46-a38b-e88708df6273-kube-api-access-67tr5\") pod \"maas-api-6889b8b577-pb2t7\" (UID: \"a5a09bbd-0dcf-4a46-a38b-e88708df6273\") " pod="odh-ai-gateway-infra/maas-api-6889b8b577-pb2t7"
Oct 31 01:10:48.870541 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:10:48.870529    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a5a09bbd-0dcf-4a46-a38b-e88708df6273-maas-api-tls\") pod \"maas-api-6889b8b577-pb2t7\" (UID: \"a5a09bbd-0dcf-4a46-a38b-e88708df6273\") " pod="odh-ai-gateway-infra/maas-api-6889b8b577-pb2t7"
Oct 31 01:10:48.971895 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:10:48.971857    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a5a09bbd-0dcf-4a46-a38b-e88708df6273-metrics-certs\") pod \"maas-api-6889b8b577-pb2t7\" (UID: \"a5a09bbd-0dcf-4a46-a38b-e88708df6273\") " pod="odh-ai-gateway-infra/maas-api-6889b8b577-pb2t7"
Oct 31 01:10:48.971895 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:10:48.971899    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-67tr5\" (UniqueName: \"kubernetes.io/projected/a5a09bbd-0dcf-4a46-a38b-e88708df6273-kube-api-access-67tr5\") pod \"maas-api-6889b8b577-pb2t7\" (UID: \"a5a09bbd-0dcf-4a46-a38b-e88708df6273\") " pod="odh-ai-gateway-infra/maas-api-6889b8b577-pb2t7"
Oct 31 01:10:48.972101 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:10:48.971948    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a5a09bbd-0dcf-4a46-a38b-e88708df6273-maas-api-tls\") pod \"maas-api-6889b8b577-pb2t7\" (UID: \"a5a09bbd-0dcf-4a46-a38b-e88708df6273\") " pod="odh-ai-gateway-infra/maas-api-6889b8b577-pb2t7"
Oct 31 01:10:48.972101 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:10:48.972048    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a5a09bbd-0dcf-4a46-a38b-e88708df6273-openshift-service-ca\") pod \"maas-api-6889b8b577-pb2t7\" (UID: \"a5a09bbd-0dcf-4a46-a38b-e88708df6273\") " pod="odh-ai-gateway-infra/maas-api-6889b8b577-pb2t7"
Oct 31 01:10:48.972825 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:10:48.972729    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a5a09bbd-0dcf-4a46-a38b-e88708df6273-openshift-service-ca\") pod \"maas-api-6889b8b577-pb2t7\" (UID: \"a5a09bbd-0dcf-4a46-a38b-e88708df6273\") " pod="odh-ai-gateway-infra/maas-api-6889b8b577-pb2t7"
Oct 31 01:10:48.974610 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:10:48.974586    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a5a09bbd-0dcf-4a46-a38b-e88708df6273-maas-api-tls\") pod \"maas-api-6889b8b577-pb2t7\" (UID: \"a5a09bbd-0dcf-4a46-a38b-e88708df6273\") " pod="odh-ai-gateway-infra/maas-api-6889b8b577-pb2t7"
Oct 31 01:10:48.974734 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:10:48.974712    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a5a09bbd-0dcf-4a46-a38b-e88708df6273-metrics-certs\") pod \"maas-api-6889b8b577-pb2t7\" (UID: \"a5a09bbd-0dcf-4a46-a38b-e88708df6273\") " pod="odh-ai-gateway-infra/maas-api-6889b8b577-pb2t7"
Oct 31 01:10:49.029261 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:10:49.029181    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-67tr5\" (UniqueName: \"kubernetes.io/projected/a5a09bbd-0dcf-4a46-a38b-e88708df6273-kube-api-access-67tr5\") pod \"maas-api-6889b8b577-pb2t7\" (UID: \"a5a09bbd-0dcf-4a46-a38b-e88708df6273\") " pod="odh-ai-gateway-infra/maas-api-6889b8b577-pb2t7"
Oct 31 01:10:49.067508 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:10:49.067475    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6889b8b577-pb2t7"
Oct 31 01:10:49.438930 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:10:49.438906    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6889b8b577-pb2t7"]
Oct 31 01:10:49.441507 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:10:49.441460    4964 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5a09bbd_0dcf_4a46_a38b_e88708df6273.slice/crio-b9bfecf2221048f192e9053d2808d3c17df62811b5832b696ebf89361b412299 WatchSource:0}: Error finding container b9bfecf2221048f192e9053d2808d3c17df62811b5832b696ebf89361b412299: Status 404 returned error can't find the container with id b9bfecf2221048f192e9053d2808d3c17df62811b5832b696ebf89361b412299
Oct 31 01:10:49.829832 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:10:49.829798    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6889b8b577-pb2t7" event={"ID":"a5a09bbd-0dcf-4a46-a38b-e88708df6273","Type":"ContainerStarted","Data":"b9bfecf2221048f192e9053d2808d3c17df62811b5832b696ebf89361b412299"}
Oct 31 01:10:53.110750 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:10:53.110712    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:10:56.870815 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:10:56.870771    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6889b8b577-pb2t7" event={"ID":"a5a09bbd-0dcf-4a46-a38b-e88708df6273","Type":"ContainerStarted","Data":"6e078db67a49e27f7f26bab843c97128bee2d93d10282111f5f6f9f7b4969c1f"}
Oct 31 01:10:56.871207 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:10:56.870860    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-6889b8b577-pb2t7"
Oct 31 01:10:56.987339 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:10:56.987278    4964 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-6889b8b577-pb2t7" podStartSLOduration=2.385656464 podStartE2EDuration="8.98726584s" totalImagesPullingTime="6.601609376s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:10:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:10:56.985845133 +0000 UTC m=+425.462411513" watchObservedRunningTime="2026-10-31 01:10:56.98726584 +0000 UTC m=+425.463832220"
Oct 31 01:11:02.881230 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:11:02.881202    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-6889b8b577-pb2t7"
Oct 31 01:11:03.456965 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:11:03.456929    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:11:03.684512 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:11:03.684443    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:11:03.908333 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:11:03.908290    4964 generic.go:356] "Generic (PLEG): container finished" podID="23f1b6ef-5cf9-4d1a-afdd-75a0c723727c" containerID="b8cc5e4779b99a864ec1494eabe62d70b306a08294a77c9d1066df055e523b2c" exitCode=0
Oct 31 01:11:03.908736 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:11:03.908373    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-jn9hq" event={"ID":"23f1b6ef-5cf9-4d1a-afdd-75a0c723727c","Type":"ContainerDied","Data":"b8cc5e4779b99a864ec1494eabe62d70b306a08294a77c9d1066df055e523b2c"}
Oct 31 01:11:03.908787 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:11:03.908750    4964 scope.go:122] "RemoveContainer" containerID="b8cc5e4779b99a864ec1494eabe62d70b306a08294a77c9d1066df055e523b2c"
Oct 31 01:11:03.909845 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:11:03.909819    4964 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 01:11:03.909913 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:11:03.909904    4964 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 01:11:03.909949 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:11:03.909935    4964 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 01:11:03.938784 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:11:03.938760    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:11:04.819027 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:11:04.818998    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-jn9hq"
Oct 31 01:11:04.919306 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:11:04.919273    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-jn9hq" event={"ID":"23f1b6ef-5cf9-4d1a-afdd-75a0c723727c","Type":"ContainerStarted","Data":"132299ecb89272458db55c532ab4f33a4f465573a9e338b86e64a0500bcca42f"}
Oct 31 01:11:05.307895 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:11:05.307866    4964 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-jn9hq"
Oct 31 01:11:05.308241 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:11:05.308218    4964 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-jn9hq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.34:15021/healthz/ready\": dial tcp 10.132.0.34:15021: connect: connection refused" start-of-body=
Oct 31 01:11:05.308430 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:11:05.308403    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-jn9hq" podUID="23f1b6ef-5cf9-4d1a-afdd-75a0c723727c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.34:15021/healthz/ready\": dial tcp 10.132.0.34:15021: connect: connection refused"
Oct 31 01:11:06.306469 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:11:06.306432    4964 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-jn9hq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.34:15021/healthz/ready\": dial tcp 10.132.0.34:15021: connect: connection refused" start-of-body=
Oct 31 01:11:06.307085 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:11:06.307040    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-jn9hq" podUID="23f1b6ef-5cf9-4d1a-afdd-75a0c723727c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.34:15021/healthz/ready\": dial tcp 10.132.0.34:15021: connect: connection refused"
Oct 31 01:11:07.312344 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:11:07.305229    4964 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-jn9hq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.34:15021/healthz/ready\": dial tcp 10.132.0.34:15021: connect: connection refused" start-of-body=
Oct 31 01:11:07.312344 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:11:07.305386    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-jn9hq" podUID="23f1b6ef-5cf9-4d1a-afdd-75a0c723727c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.34:15021/healthz/ready\": dial tcp 10.132.0.34:15021: connect: connection refused"
Oct 31 01:11:08.307795 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:11:08.307755    4964 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-jn9hq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.34:15021/healthz/ready\": dial tcp 10.132.0.34:15021: connect: connection refused" start-of-body=
Oct 31 01:11:08.308090 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:11:08.308059    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-jn9hq" podUID="23f1b6ef-5cf9-4d1a-afdd-75a0c723727c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.34:15021/healthz/ready\": dial tcp 10.132.0.34:15021: connect: connection refused"
Oct 31 01:11:09.308789 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:11:09.308751    4964 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-jn9hq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.34:15021/healthz/ready\": dial tcp 10.132.0.34:15021: connect: connection refused" start-of-body=
Oct 31 01:11:09.309450 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:11:09.309418    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-jn9hq" podUID="23f1b6ef-5cf9-4d1a-afdd-75a0c723727c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.34:15021/healthz/ready\": dial tcp 10.132.0.34:15021: connect: connection refused"
Oct 31 01:11:10.306672 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:11:10.306642    4964 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-jn9hq"
Oct 31 01:11:10.307057 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:11:10.307037    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-jn9hq"
Oct 31 01:11:10.307679 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:11:10.307660    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-jn9hq"
Oct 31 01:11:11.237061 ip-10-0-133-197 kubenswrapper[4964]: E1031 01:11:11.237027    4964 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q56x5" containerName="extractor"
Oct 31 01:11:12.401384 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:11:12.401341    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:11:12.950898 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:11:12.950869    4964 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-6889b8b577-pb2t7"]
Oct 31 01:11:12.951086 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:11:12.951053    4964 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-6889b8b577-pb2t7" podUID="a5a09bbd-0dcf-4a46-a38b-e88708df6273" containerName="maas-api" containerID="cri-o://6e078db67a49e27f7f26bab843c97128bee2d93d10282111f5f6f9f7b4969c1f" gracePeriod=30
Oct 31 01:11:13.209552 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:11:13.209499    4964 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6889b8b577-pb2t7"
Oct 31 01:11:13.291715 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:11:13.291685    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a5a09bbd-0dcf-4a46-a38b-e88708df6273-metrics-certs\" (UniqueName: \"kubernetes.io/secret/a5a09bbd-0dcf-4a46-a38b-e88708df6273-metrics-certs\") pod \"a5a09bbd-0dcf-4a46-a38b-e88708df6273\" (UID: \"a5a09bbd-0dcf-4a46-a38b-e88708df6273\") "
Oct 31 01:11:13.291880 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:11:13.291755    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a5a09bbd-0dcf-4a46-a38b-e88708df6273-kube-api-access-67tr5\" (UniqueName: \"kubernetes.io/projected/a5a09bbd-0dcf-4a46-a38b-e88708df6273-kube-api-access-67tr5\") pod \"a5a09bbd-0dcf-4a46-a38b-e88708df6273\" (UID: \"a5a09bbd-0dcf-4a46-a38b-e88708df6273\") "
Oct 31 01:11:13.291880 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:11:13.291800    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a5a09bbd-0dcf-4a46-a38b-e88708df6273-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a5a09bbd-0dcf-4a46-a38b-e88708df6273-maas-api-tls\") pod \"a5a09bbd-0dcf-4a46-a38b-e88708df6273\" (UID: \"a5a09bbd-0dcf-4a46-a38b-e88708df6273\") "
Oct 31 01:11:13.291880 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:11:13.291866    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a5a09bbd-0dcf-4a46-a38b-e88708df6273-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a5a09bbd-0dcf-4a46-a38b-e88708df6273-openshift-service-ca\") pod \"a5a09bbd-0dcf-4a46-a38b-e88708df6273\" (UID: \"a5a09bbd-0dcf-4a46-a38b-e88708df6273\") "
Oct 31 01:11:13.292280 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:11:13.292251    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a5a09bbd-0dcf-4a46-a38b-e88708df6273-openshift-service-ca" pod "a5a09bbd-0dcf-4a46-a38b-e88708df6273" (UID: "a5a09bbd-0dcf-4a46-a38b-e88708df6273"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:11:13.294147 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:11:13.294121    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a5a09bbd-0dcf-4a46-a38b-e88708df6273-kube-api-access-67tr5" pod "a5a09bbd-0dcf-4a46-a38b-e88708df6273" (UID: "a5a09bbd-0dcf-4a46-a38b-e88708df6273"). InnerVolumeSpecName "kube-api-access-67tr5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:11:13.294147 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:11:13.294139    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a5a09bbd-0dcf-4a46-a38b-e88708df6273-metrics-certs" pod "a5a09bbd-0dcf-4a46-a38b-e88708df6273" (UID: "a5a09bbd-0dcf-4a46-a38b-e88708df6273"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:11:13.294260 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:11:13.294164    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a5a09bbd-0dcf-4a46-a38b-e88708df6273-maas-api-tls" pod "a5a09bbd-0dcf-4a46-a38b-e88708df6273" (UID: "a5a09bbd-0dcf-4a46-a38b-e88708df6273"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:11:13.392946 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:11:13.392916    4964 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-67tr5\" (UniqueName: \"kubernetes.io/projected/a5a09bbd-0dcf-4a46-a38b-e88708df6273-kube-api-access-67tr5\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:11:13.392946 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:11:13.392941    4964 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a5a09bbd-0dcf-4a46-a38b-e88708df6273-maas-api-tls\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:11:13.392946 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:11:13.392951    4964 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a5a09bbd-0dcf-4a46-a38b-e88708df6273-openshift-service-ca\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:11:13.393162 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:11:13.392960    4964 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a5a09bbd-0dcf-4a46-a38b-e88708df6273-metrics-certs\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:11:13.966667 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:11:13.966635    4964 generic.go:356] "Generic (PLEG): container finished" podID="a5a09bbd-0dcf-4a46-a38b-e88708df6273" containerID="6e078db67a49e27f7f26bab843c97128bee2d93d10282111f5f6f9f7b4969c1f" exitCode=0
Oct 31 01:11:13.967072 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:11:13.966679    4964 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6889b8b577-pb2t7"
Oct 31 01:11:13.967072 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:11:13.966723    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6889b8b577-pb2t7" event={"ID":"a5a09bbd-0dcf-4a46-a38b-e88708df6273","Type":"ContainerDied","Data":"6e078db67a49e27f7f26bab843c97128bee2d93d10282111f5f6f9f7b4969c1f"}
Oct 31 01:11:13.967072 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:11:13.966776    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6889b8b577-pb2t7" event={"ID":"a5a09bbd-0dcf-4a46-a38b-e88708df6273","Type":"ContainerDied","Data":"b9bfecf2221048f192e9053d2808d3c17df62811b5832b696ebf89361b412299"}
Oct 31 01:11:13.967072 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:11:13.966797    4964 scope.go:122] "RemoveContainer" containerID="6e078db67a49e27f7f26bab843c97128bee2d93d10282111f5f6f9f7b4969c1f"
Oct 31 01:11:13.976086 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:11:13.976067    4964 scope.go:122] "RemoveContainer" containerID="6e078db67a49e27f7f26bab843c97128bee2d93d10282111f5f6f9f7b4969c1f"
Oct 31 01:11:13.976367 ip-10-0-133-197 kubenswrapper[4964]: E1031 01:11:13.976346    4964 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6e078db67a49e27f7f26bab843c97128bee2d93d10282111f5f6f9f7b4969c1f\": container with ID starting with 6e078db67a49e27f7f26bab843c97128bee2d93d10282111f5f6f9f7b4969c1f not found: ID does not exist" containerID="6e078db67a49e27f7f26bab843c97128bee2d93d10282111f5f6f9f7b4969c1f"
Oct 31 01:11:13.976416 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:11:13.976379    4964 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6e078db67a49e27f7f26bab843c97128bee2d93d10282111f5f6f9f7b4969c1f"} err="failed to get container status \"6e078db67a49e27f7f26bab843c97128bee2d93d10282111f5f6f9f7b4969c1f\": rpc error: code = NotFound desc = could not find container \"6e078db67a49e27f7f26bab843c97128bee2d93d10282111f5f6f9f7b4969c1f\": container with ID starting with 6e078db67a49e27f7f26bab843c97128bee2d93d10282111f5f6f9f7b4969c1f not found: ID does not exist"
Oct 31 01:11:14.129731 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:11:14.129646    4964 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-6889b8b577-pb2t7"]
Oct 31 01:11:14.179248 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:11:14.179214    4964 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-6889b8b577-pb2t7"]
Oct 31 01:11:15.775358 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:11:15.775308    4964 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a5a09bbd-0dcf-4a46-a38b-e88708df6273" path="/var/lib/kubelet/pods/a5a09bbd-0dcf-4a46-a38b-e88708df6273/volumes"
Oct 31 01:11:41.238052 ip-10-0-133-197 kubenswrapper[4964]: E1031 01:11:41.238016    4964 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q56x5" containerName="extractor"
Oct 31 01:11:43.451096 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:11:43.451055    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:11:53.453982 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:11:53.453940    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:11:59.148760 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:11:59.148714    4964 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/ai-gateway-controller-7d85675f98-cn7gj"]
Oct 31 01:11:59.149341 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:11:59.149303    4964 cpu_manager.go:414] "containerMap: removing container" podUID="a5a09bbd-0dcf-4a46-a38b-e88708df6273" containerName="maas-api"
Oct 31 01:11:59.149447 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:11:59.149349    4964 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a5a09bbd-0dcf-4a46-a38b-e88708df6273" containerName="maas-api"
Oct 31 01:11:59.149507 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:11:59.149467    4964 memory_manager.go:365] "RemoveStaleState removing state" podUID="a5a09bbd-0dcf-4a46-a38b-e88708df6273" containerName="maas-api"
Oct 31 01:11:59.153695 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:11:59.153670    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-7d85675f98-cn7gj"
Oct 31 01:11:59.187502 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:11:59.187468    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dlp2x\" (UniqueName: \"kubernetes.io/projected/5463b137-25cd-449a-b791-ea4cba60e1e1-kube-api-access-dlp2x\") pod \"ai-gateway-controller-7d85675f98-cn7gj\" (UID: \"5463b137-25cd-449a-b791-ea4cba60e1e1\") " pod="opendatahub/ai-gateway-controller-7d85675f98-cn7gj"
Oct 31 01:11:59.187691 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:11:59.187567    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5463b137-25cd-449a-b791-ea4cba60e1e1-metrics-certs\") pod \"ai-gateway-controller-7d85675f98-cn7gj\" (UID: \"5463b137-25cd-449a-b791-ea4cba60e1e1\") " pod="opendatahub/ai-gateway-controller-7d85675f98-cn7gj"
Oct 31 01:11:59.201263 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:11:59.201233    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:11:59.209834 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:11:59.209801    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-7d85675f98-cn7gj"]
Oct 31 01:11:59.256666 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:11:59.256636    4964 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"ai-gateway-controller-metrics-tls\""
Oct 31 01:11:59.256875 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:11:59.256681    4964 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"ai-gateway-controller-dockercfg-7jkz9\""
Oct 31 01:11:59.256875 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:11:59.256679    4964 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"ai-gateway-controller-parameters\""
Oct 31 01:11:59.288510 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:11:59.288471    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dlp2x\" (UniqueName: \"kubernetes.io/projected/5463b137-25cd-449a-b791-ea4cba60e1e1-kube-api-access-dlp2x\") pod \"ai-gateway-controller-7d85675f98-cn7gj\" (UID: \"5463b137-25cd-449a-b791-ea4cba60e1e1\") " pod="opendatahub/ai-gateway-controller-7d85675f98-cn7gj"
Oct 31 01:11:59.288705 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:11:59.288532    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5463b137-25cd-449a-b791-ea4cba60e1e1-metrics-certs\") pod \"ai-gateway-controller-7d85675f98-cn7gj\" (UID: \"5463b137-25cd-449a-b791-ea4cba60e1e1\") " pod="opendatahub/ai-gateway-controller-7d85675f98-cn7gj"
Oct 31 01:11:59.291262 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:11:59.291235    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5463b137-25cd-449a-b791-ea4cba60e1e1-metrics-certs\") pod \"ai-gateway-controller-7d85675f98-cn7gj\" (UID: \"5463b137-25cd-449a-b791-ea4cba60e1e1\") " pod="opendatahub/ai-gateway-controller-7d85675f98-cn7gj"
Oct 31 01:11:59.345715 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:11:59.345683    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dlp2x\" (UniqueName: \"kubernetes.io/projected/5463b137-25cd-449a-b791-ea4cba60e1e1-kube-api-access-dlp2x\") pod \"ai-gateway-controller-7d85675f98-cn7gj\" (UID: \"5463b137-25cd-449a-b791-ea4cba60e1e1\") " pod="opendatahub/ai-gateway-controller-7d85675f98-cn7gj"
Oct 31 01:11:59.409630 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:11:59.409531    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:11:59.467006 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:11:59.466968    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-7d85675f98-cn7gj"
Oct 31 01:11:59.839552 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:11:59.839519    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-7d85675f98-cn7gj"]
Oct 31 01:11:59.842790 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:11:59.842762    4964 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5463b137_25cd_449a_b791_ea4cba60e1e1.slice/crio-6cd2377e8fb52aa89deb02ba8c4e58dec015d7b515939b733f958ccfaced9ba6 WatchSource:0}: Error finding container 6cd2377e8fb52aa89deb02ba8c4e58dec015d7b515939b733f958ccfaced9ba6: Status 404 returned error can't find the container with id 6cd2377e8fb52aa89deb02ba8c4e58dec015d7b515939b733f958ccfaced9ba6
Oct 31 01:12:00.195710 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:12:00.195600    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-7d85675f98-cn7gj" event={"ID":"5463b137-25cd-449a-b791-ea4cba60e1e1","Type":"ContainerStarted","Data":"6cd2377e8fb52aa89deb02ba8c4e58dec015d7b515939b733f958ccfaced9ba6"}
Oct 31 01:12:03.213452 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:12:03.213419    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-7d85675f98-cn7gj" event={"ID":"5463b137-25cd-449a-b791-ea4cba60e1e1","Type":"ContainerStarted","Data":"68b95ae476d3d7571a62bb999e679317732372f6198554f709d16a819cc3fc08"}
Oct 31 01:12:03.213879 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:12:03.213539    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/ai-gateway-controller-7d85675f98-cn7gj"
Oct 31 01:12:03.328699 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:12:03.328645    4964 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/ai-gateway-controller-7d85675f98-cn7gj" podStartSLOduration=2.0021357 podStartE2EDuration="4.328629748s" totalImagesPullingTime="2.326494048s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:11:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:12:03.326973735 +0000 UTC m=+491.803540107" watchObservedRunningTime="2026-10-31 01:12:03.328629748 +0000 UTC m=+491.805196127"
Oct 31 01:12:06.104780 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:12:06.104735    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:12:06.333939 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:12:06.333900    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:12:06.632108 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:12:06.632069    4964 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-55fbbbf9dc-tlb5m"]
Oct 31 01:12:06.643843 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:12:06.643807    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-55fbbbf9dc-tlb5m"
Oct 31 01:12:06.650273 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:12:06.650235    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zvkms\" (UniqueName: \"kubernetes.io/projected/c8461fdb-12e5-4691-8b95-139d5a901f0d-kube-api-access-zvkms\") pod \"payload-processing-55fbbbf9dc-tlb5m\" (UID: \"c8461fdb-12e5-4691-8b95-139d5a901f0d\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-tlb5m"
Oct 31 01:12:06.650443 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:12:06.650410    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c8461fdb-12e5-4691-8b95-139d5a901f0d-plugins-config-volume\") pod \"payload-processing-55fbbbf9dc-tlb5m\" (UID: \"c8461fdb-12e5-4691-8b95-139d5a901f0d\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-tlb5m"
Oct 31 01:12:06.693063 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:12:06.693025    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-55fbbbf9dc-tlb5m"]
Oct 31 01:12:06.739500 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:12:06.739463    4964 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\""
Oct 31 01:12:06.739500 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:12:06.739497    4964 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-xbgfw\""
Oct 31 01:12:06.751517 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:12:06.751491    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zvkms\" (UniqueName: \"kubernetes.io/projected/c8461fdb-12e5-4691-8b95-139d5a901f0d-kube-api-access-zvkms\") pod \"payload-processing-55fbbbf9dc-tlb5m\" (UID: \"c8461fdb-12e5-4691-8b95-139d5a901f0d\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-tlb5m"
Oct 31 01:12:06.751663 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:12:06.751548    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c8461fdb-12e5-4691-8b95-139d5a901f0d-plugins-config-volume\") pod \"payload-processing-55fbbbf9dc-tlb5m\" (UID: \"c8461fdb-12e5-4691-8b95-139d5a901f0d\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-tlb5m"
Oct 31 01:12:06.752276 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:12:06.752255    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c8461fdb-12e5-4691-8b95-139d5a901f0d-plugins-config-volume\") pod \"payload-processing-55fbbbf9dc-tlb5m\" (UID: \"c8461fdb-12e5-4691-8b95-139d5a901f0d\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-tlb5m"
Oct 31 01:12:06.808531 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:12:06.808494    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zvkms\" (UniqueName: \"kubernetes.io/projected/c8461fdb-12e5-4691-8b95-139d5a901f0d-kube-api-access-zvkms\") pod \"payload-processing-55fbbbf9dc-tlb5m\" (UID: \"c8461fdb-12e5-4691-8b95-139d5a901f0d\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-tlb5m"
Oct 31 01:12:06.956841 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:12:06.956746    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-55fbbbf9dc-tlb5m"
Oct 31 01:12:07.329616 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:12:07.329545    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-55fbbbf9dc-tlb5m"]
Oct 31 01:12:07.332143 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:12:07.332113    4964 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc8461fdb_12e5_4691_8b95_139d5a901f0d.slice/crio-36acc84ad9c03c47ff9440aa3ef9df72c59bc3f14f4c521a793bd37cfe9f5723 WatchSource:0}: Error finding container 36acc84ad9c03c47ff9440aa3ef9df72c59bc3f14f4c521a793bd37cfe9f5723: Status 404 returned error can't find the container with id 36acc84ad9c03c47ff9440aa3ef9df72c59bc3f14f4c521a793bd37cfe9f5723
Oct 31 01:12:08.242715 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:12:08.242673    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-55fbbbf9dc-tlb5m" event={"ID":"c8461fdb-12e5-4691-8b95-139d5a901f0d","Type":"ContainerStarted","Data":"36acc84ad9c03c47ff9440aa3ef9df72c59bc3f14f4c521a793bd37cfe9f5723"}
Oct 31 01:12:10.254510 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:12:10.254465    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-55fbbbf9dc-tlb5m" event={"ID":"c8461fdb-12e5-4691-8b95-139d5a901f0d","Type":"ContainerStarted","Data":"c32095534131d93f5e385aa3136606eb54077a0eda982a4c3d88b127e02a0f10"}
Oct 31 01:12:10.254950 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:12:10.254587    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-55fbbbf9dc-tlb5m"
Oct 31 01:12:10.369522 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:12:10.369471    4964 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-55fbbbf9dc-tlb5m" podStartSLOduration=1.8544671959999999 podStartE2EDuration="4.369456693s" totalImagesPullingTime="2.514989497s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:12:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:12:10.367392332 +0000 UTC m=+498.843958714" watchObservedRunningTime="2026-10-31 01:12:10.369456693 +0000 UTC m=+498.846023073"
Oct 31 01:12:11.237675 ip-10-0-133-197 kubenswrapper[4964]: E1031 01:12:11.237628    4964 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q56x5" containerName="extractor"
Oct 31 01:12:12.743092 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:12:12.743058    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:12:14.221575 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:12:14.221544    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/ai-gateway-controller-7d85675f98-cn7gj"
Oct 31 01:12:14.602068 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:12:14.601991    4964 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/ai-gateway-controller-58f5dc44b9-c98xl"]
Oct 31 01:12:14.605719 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:12:14.605700    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-58f5dc44b9-c98xl"
Oct 31 01:12:14.630633 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:12:14.630599    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fbsxt\" (UniqueName: \"kubernetes.io/projected/cf528aec-95b1-46dd-8f6c-da3ac6cc85d9-kube-api-access-fbsxt\") pod \"ai-gateway-controller-58f5dc44b9-c98xl\" (UID: \"cf528aec-95b1-46dd-8f6c-da3ac6cc85d9\") " pod="opendatahub/ai-gateway-controller-58f5dc44b9-c98xl"
Oct 31 01:12:14.630827 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:12:14.630656    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cf528aec-95b1-46dd-8f6c-da3ac6cc85d9-metrics-certs\") pod \"ai-gateway-controller-58f5dc44b9-c98xl\" (UID: \"cf528aec-95b1-46dd-8f6c-da3ac6cc85d9\") " pod="opendatahub/ai-gateway-controller-58f5dc44b9-c98xl"
Oct 31 01:12:14.663739 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:12:14.663704    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-58f5dc44b9-c98xl"]
Oct 31 01:12:14.731531 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:12:14.731488    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fbsxt\" (UniqueName: \"kubernetes.io/projected/cf528aec-95b1-46dd-8f6c-da3ac6cc85d9-kube-api-access-fbsxt\") pod \"ai-gateway-controller-58f5dc44b9-c98xl\" (UID: \"cf528aec-95b1-46dd-8f6c-da3ac6cc85d9\") " pod="opendatahub/ai-gateway-controller-58f5dc44b9-c98xl"
Oct 31 01:12:14.731714 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:12:14.731548    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cf528aec-95b1-46dd-8f6c-da3ac6cc85d9-metrics-certs\") pod \"ai-gateway-controller-58f5dc44b9-c98xl\" (UID: \"cf528aec-95b1-46dd-8f6c-da3ac6cc85d9\") " pod="opendatahub/ai-gateway-controller-58f5dc44b9-c98xl"
Oct 31 01:12:14.734157 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:12:14.734131    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cf528aec-95b1-46dd-8f6c-da3ac6cc85d9-metrics-certs\") pod \"ai-gateway-controller-58f5dc44b9-c98xl\" (UID: \"cf528aec-95b1-46dd-8f6c-da3ac6cc85d9\") " pod="opendatahub/ai-gateway-controller-58f5dc44b9-c98xl"
Oct 31 01:12:14.788469 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:12:14.788433    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fbsxt\" (UniqueName: \"kubernetes.io/projected/cf528aec-95b1-46dd-8f6c-da3ac6cc85d9-kube-api-access-fbsxt\") pod \"ai-gateway-controller-58f5dc44b9-c98xl\" (UID: \"cf528aec-95b1-46dd-8f6c-da3ac6cc85d9\") " pod="opendatahub/ai-gateway-controller-58f5dc44b9-c98xl"
Oct 31 01:12:14.918895 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:12:14.918793    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-58f5dc44b9-c98xl"
Oct 31 01:12:15.294679 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:12:15.294651    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-58f5dc44b9-c98xl"]
Oct 31 01:12:15.297379 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:12:15.297340    4964 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf528aec_95b1_46dd_8f6c_da3ac6cc85d9.slice/crio-04991948ec7214f45e4c63fd81cf4a0f60b0db5c5ba4fe89905012f0e716aee0 WatchSource:0}: Error finding container 04991948ec7214f45e4c63fd81cf4a0f60b0db5c5ba4fe89905012f0e716aee0: Status 404 returned error can't find the container with id 04991948ec7214f45e4c63fd81cf4a0f60b0db5c5ba4fe89905012f0e716aee0
Oct 31 01:12:15.443158 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:12:15.443118    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:12:16.291565 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:12:16.291530    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-58f5dc44b9-c98xl" event={"ID":"cf528aec-95b1-46dd-8f6c-da3ac6cc85d9","Type":"ContainerStarted","Data":"6941748f0989c82a47228b97f4efbe708f8400cf212bb5e92fe9113c5c272fb1"}
Oct 31 01:12:16.291758 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:12:16.291585    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-58f5dc44b9-c98xl" event={"ID":"cf528aec-95b1-46dd-8f6c-da3ac6cc85d9","Type":"ContainerStarted","Data":"04991948ec7214f45e4c63fd81cf4a0f60b0db5c5ba4fe89905012f0e716aee0"}
Oct 31 01:12:16.291758 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:12:16.291726    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/ai-gateway-controller-58f5dc44b9-c98xl"
Oct 31 01:12:16.408352 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:12:16.408215    4964 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/ai-gateway-controller-58f5dc44b9-c98xl" podStartSLOduration=1.583917204 podStartE2EDuration="2.408199467s" totalImagesPullingTime="824.282263ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:12:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:12:16.407557926 +0000 UTC m=+504.884124307" watchObservedRunningTime="2026-10-31 01:12:16.408199467 +0000 UTC m=+504.884765848"
Oct 31 01:12:21.263489 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:12:21.263457    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-55fbbbf9dc-tlb5m"
Oct 31 01:12:22.534626 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:12:22.534579    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:12:27.298476 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:12:27.298446    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/ai-gateway-controller-58f5dc44b9-c98xl"
Oct 31 01:12:27.471555 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:12:27.471525    4964 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/ai-gateway-controller-7d85675f98-cn7gj"]
Oct 31 01:12:27.471814 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:12:27.471761    4964 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/ai-gateway-controller-7d85675f98-cn7gj" podUID="5463b137-25cd-449a-b791-ea4cba60e1e1" containerName="manager" containerID="cri-o://68b95ae476d3d7571a62bb999e679317732372f6198554f709d16a819cc3fc08" gracePeriod=30
Oct 31 01:12:27.718522 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:12:27.718495    4964 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-7d85675f98-cn7gj"
Oct 31 01:12:27.850826 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:12:27.850740    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5463b137-25cd-449a-b791-ea4cba60e1e1-metrics-certs\" (UniqueName: \"kubernetes.io/secret/5463b137-25cd-449a-b791-ea4cba60e1e1-metrics-certs\") pod \"5463b137-25cd-449a-b791-ea4cba60e1e1\" (UID: \"5463b137-25cd-449a-b791-ea4cba60e1e1\") "
Oct 31 01:12:27.850993 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:12:27.850845    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5463b137-25cd-449a-b791-ea4cba60e1e1-kube-api-access-dlp2x\" (UniqueName: \"kubernetes.io/projected/5463b137-25cd-449a-b791-ea4cba60e1e1-kube-api-access-dlp2x\") pod \"5463b137-25cd-449a-b791-ea4cba60e1e1\" (UID: \"5463b137-25cd-449a-b791-ea4cba60e1e1\") "
Oct 31 01:12:27.853298 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:12:27.853270    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5463b137-25cd-449a-b791-ea4cba60e1e1-metrics-certs" pod "5463b137-25cd-449a-b791-ea4cba60e1e1" (UID: "5463b137-25cd-449a-b791-ea4cba60e1e1"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:12:27.853454 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:12:27.853427    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5463b137-25cd-449a-b791-ea4cba60e1e1-kube-api-access-dlp2x" pod "5463b137-25cd-449a-b791-ea4cba60e1e1" (UID: "5463b137-25cd-449a-b791-ea4cba60e1e1"). InnerVolumeSpecName "kube-api-access-dlp2x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:12:27.951742 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:12:27.951708    4964 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5463b137-25cd-449a-b791-ea4cba60e1e1-metrics-certs\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:12:27.951742 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:12:27.951741    4964 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dlp2x\" (UniqueName: \"kubernetes.io/projected/5463b137-25cd-449a-b791-ea4cba60e1e1-kube-api-access-dlp2x\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:12:28.359008 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:12:28.358973    4964 generic.go:356] "Generic (PLEG): container finished" podID="5463b137-25cd-449a-b791-ea4cba60e1e1" containerID="68b95ae476d3d7571a62bb999e679317732372f6198554f709d16a819cc3fc08" exitCode=0
Oct 31 01:12:28.359474 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:12:28.359021    4964 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-7d85675f98-cn7gj"
Oct 31 01:12:28.359474 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:12:28.359058    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-7d85675f98-cn7gj" event={"ID":"5463b137-25cd-449a-b791-ea4cba60e1e1","Type":"ContainerDied","Data":"68b95ae476d3d7571a62bb999e679317732372f6198554f709d16a819cc3fc08"}
Oct 31 01:12:28.359474 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:12:28.359114    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-7d85675f98-cn7gj" event={"ID":"5463b137-25cd-449a-b791-ea4cba60e1e1","Type":"ContainerDied","Data":"6cd2377e8fb52aa89deb02ba8c4e58dec015d7b515939b733f958ccfaced9ba6"}
Oct 31 01:12:28.359474 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:12:28.359137    4964 scope.go:122] "RemoveContainer" containerID="68b95ae476d3d7571a62bb999e679317732372f6198554f709d16a819cc3fc08"
Oct 31 01:12:28.368176 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:12:28.368162    4964 scope.go:122] "RemoveContainer" containerID="68b95ae476d3d7571a62bb999e679317732372f6198554f709d16a819cc3fc08"
Oct 31 01:12:28.368456 ip-10-0-133-197 kubenswrapper[4964]: E1031 01:12:28.368436    4964 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"68b95ae476d3d7571a62bb999e679317732372f6198554f709d16a819cc3fc08\": container with ID starting with 68b95ae476d3d7571a62bb999e679317732372f6198554f709d16a819cc3fc08 not found: ID does not exist" containerID="68b95ae476d3d7571a62bb999e679317732372f6198554f709d16a819cc3fc08"
Oct 31 01:12:28.368503 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:12:28.368465    4964 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"68b95ae476d3d7571a62bb999e679317732372f6198554f709d16a819cc3fc08"} err="failed to get container status \"68b95ae476d3d7571a62bb999e679317732372f6198554f709d16a819cc3fc08\": rpc error: code = NotFound desc = could not find container \"68b95ae476d3d7571a62bb999e679317732372f6198554f709d16a819cc3fc08\": container with ID starting with 68b95ae476d3d7571a62bb999e679317732372f6198554f709d16a819cc3fc08 not found: ID does not exist"
Oct 31 01:12:28.522350 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:12:28.522303    4964 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/ai-gateway-controller-7d85675f98-cn7gj"]
Oct 31 01:12:28.571829 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:12:28.571797    4964 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/ai-gateway-controller-7d85675f98-cn7gj"]
Oct 31 01:12:29.775058 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:12:29.775026    4964 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5463b137-25cd-449a-b791-ea4cba60e1e1" path="/var/lib/kubelet/pods/5463b137-25cd-449a-b791-ea4cba60e1e1/volumes"
Oct 31 01:12:33.945981 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:12:33.945943    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:12:39.223757 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:12:39.223717    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:12:39.734849 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:12:39.734814    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:12:41.093088 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:12:41.093045    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:12:41.237638 ip-10-0-133-197 kubenswrapper[4964]: E1031 01:12:41.237602    4964 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q56x5" containerName="extractor"
Oct 31 01:12:41.345000 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:12:41.344916    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:12:43.767182 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:12:43.767148    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:12:46.919392 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:12:46.919346    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:12:47.111069 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:12:47.111025    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:12:51.051913 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:12:51.051872    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:12:53.782659 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:12:53.782619    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:12:58.376648 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:12:58.376608    4964 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-dls47"]
Oct 31 01:12:58.377052 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:12:58.377013    4964 cpu_manager.go:414] "containerMap: removing container" podUID="5463b137-25cd-449a-b791-ea4cba60e1e1" containerName="manager"
Oct 31 01:12:58.377052 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:12:58.377024    4964 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5463b137-25cd-449a-b791-ea4cba60e1e1" containerName="manager"
Oct 31 01:12:58.377134 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:12:58.377123    4964 memory_manager.go:365] "RemoveStaleState removing state" podUID="5463b137-25cd-449a-b791-ea4cba60e1e1" containerName="manager"
Oct 31 01:12:58.384280 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:12:58.384254    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-dls47"
Oct 31 01:12:58.415547 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:12:58.415519    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a90b1b97-ef44-4b3c-9c5f-b1b8a79dd356-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-dls47\" (UID: \"a90b1b97-ef44-4b3c-9c5f-b1b8a79dd356\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-dls47"
Oct 31 01:12:58.415547 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:12:58.415549    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7kb4m\" (UniqueName: \"kubernetes.io/projected/a90b1b97-ef44-4b3c-9c5f-b1b8a79dd356-kube-api-access-7kb4m\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-dls47\" (UID: \"a90b1b97-ef44-4b3c-9c5f-b1b8a79dd356\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-dls47"
Oct 31 01:12:58.415747 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:12:58.415571    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a90b1b97-ef44-4b3c-9c5f-b1b8a79dd356-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-dls47\" (UID: \"a90b1b97-ef44-4b3c-9c5f-b1b8a79dd356\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-dls47"
Oct 31 01:12:58.415747 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:12:58.415595    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a90b1b97-ef44-4b3c-9c5f-b1b8a79dd356-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-dls47\" (UID: \"a90b1b97-ef44-4b3c-9c5f-b1b8a79dd356\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-dls47"
Oct 31 01:12:58.415747 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:12:58.415733    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a90b1b97-ef44-4b3c-9c5f-b1b8a79dd356-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-dls47\" (UID: \"a90b1b97-ef44-4b3c-9c5f-b1b8a79dd356\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-dls47"
Oct 31 01:12:58.415870 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:12:58.415819    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a90b1b97-ef44-4b3c-9c5f-b1b8a79dd356-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-dls47\" (UID: \"a90b1b97-ef44-4b3c-9c5f-b1b8a79dd356\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-dls47"
Oct 31 01:12:58.439138 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:12:58.439104    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-dls47"]
Oct 31 01:12:58.485307 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:12:58.485272    4964 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Oct 31 01:12:58.485307 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:12:58.485287    4964 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Oct 31 01:12:58.485576 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:12:58.485417    4964 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\""
Oct 31 01:12:58.485576 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:12:58.485467    4964 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-btdsg\""
Oct 31 01:12:58.509564 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:12:58.509530    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:12:58.516717 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:12:58.516690    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a90b1b97-ef44-4b3c-9c5f-b1b8a79dd356-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-dls47\" (UID: \"a90b1b97-ef44-4b3c-9c5f-b1b8a79dd356\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-dls47"
Oct 31 01:12:58.516876 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:12:58.516745    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a90b1b97-ef44-4b3c-9c5f-b1b8a79dd356-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-dls47\" (UID: \"a90b1b97-ef44-4b3c-9c5f-b1b8a79dd356\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-dls47"
Oct 31 01:12:58.516876 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:12:58.516769    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7kb4m\" (UniqueName: \"kubernetes.io/projected/a90b1b97-ef44-4b3c-9c5f-b1b8a79dd356-kube-api-access-7kb4m\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-dls47\" (UID: \"a90b1b97-ef44-4b3c-9c5f-b1b8a79dd356\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-dls47"
Oct 31 01:12:58.516876 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:12:58.516790    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a90b1b97-ef44-4b3c-9c5f-b1b8a79dd356-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-dls47\" (UID: \"a90b1b97-ef44-4b3c-9c5f-b1b8a79dd356\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-dls47"
Oct 31 01:12:58.516876 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:12:58.516819    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a90b1b97-ef44-4b3c-9c5f-b1b8a79dd356-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-dls47\" (UID: \"a90b1b97-ef44-4b3c-9c5f-b1b8a79dd356\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-dls47"
Oct 31 01:12:58.517095 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:12:58.516887    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a90b1b97-ef44-4b3c-9c5f-b1b8a79dd356-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-dls47\" (UID: \"a90b1b97-ef44-4b3c-9c5f-b1b8a79dd356\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-dls47"
Oct 31 01:12:58.517207 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:12:58.517180    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a90b1b97-ef44-4b3c-9c5f-b1b8a79dd356-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-dls47\" (UID: \"a90b1b97-ef44-4b3c-9c5f-b1b8a79dd356\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-dls47"
Oct 31 01:12:58.517417 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:12:58.517263    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a90b1b97-ef44-4b3c-9c5f-b1b8a79dd356-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-dls47\" (UID: \"a90b1b97-ef44-4b3c-9c5f-b1b8a79dd356\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-dls47"
Oct 31 01:12:58.517553 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:12:58.517461    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a90b1b97-ef44-4b3c-9c5f-b1b8a79dd356-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-dls47\" (UID: \"a90b1b97-ef44-4b3c-9c5f-b1b8a79dd356\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-dls47"
Oct 31 01:12:58.519875 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:12:58.519850    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a90b1b97-ef44-4b3c-9c5f-b1b8a79dd356-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-dls47\" (UID: \"a90b1b97-ef44-4b3c-9c5f-b1b8a79dd356\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-dls47"
Oct 31 01:12:58.520132 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:12:58.519937    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a90b1b97-ef44-4b3c-9c5f-b1b8a79dd356-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-dls47\" (UID: \"a90b1b97-ef44-4b3c-9c5f-b1b8a79dd356\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-dls47"
Oct 31 01:12:58.573536 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:12:58.573500    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7kb4m\" (UniqueName: \"kubernetes.io/projected/a90b1b97-ef44-4b3c-9c5f-b1b8a79dd356-kube-api-access-7kb4m\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-dls47\" (UID: \"a90b1b97-ef44-4b3c-9c5f-b1b8a79dd356\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-dls47"
Oct 31 01:12:58.697798 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:12:58.697717    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-dls47"
Oct 31 01:12:59.076879 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:12:59.076850    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-dls47"]
Oct 31 01:12:59.079462 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:12:59.079428    4964 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda90b1b97_ef44_4b3c_9c5f_b1b8a79dd356.slice/crio-78025978ccacefb6ac39d5a131afb51444f27f2af07ffe1e803fc7f6acad5b89 WatchSource:0}: Error finding container 78025978ccacefb6ac39d5a131afb51444f27f2af07ffe1e803fc7f6acad5b89: Status 404 returned error can't find the container with id 78025978ccacefb6ac39d5a131afb51444f27f2af07ffe1e803fc7f6acad5b89
Oct 31 01:12:59.515539 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:12:59.515505    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-dls47" event={"ID":"a90b1b97-ef44-4b3c-9c5f-b1b8a79dd356","Type":"ContainerStarted","Data":"78025978ccacefb6ac39d5a131afb51444f27f2af07ffe1e803fc7f6acad5b89"}
Oct 31 01:13:01.532564 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:13:01.532518    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-dls47" event={"ID":"a90b1b97-ef44-4b3c-9c5f-b1b8a79dd356","Type":"ContainerStarted","Data":"9228850b0adb2b6b978a3ed30dc95a0c127d3abe9cfd01524637da7801e690ee"}
Oct 31 01:13:01.647974 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:13:01.647920    4964 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-dls47" podStartSLOduration=2.181796677 podStartE2EDuration="3.647903811s" totalImagesPullingTime="1.466107134s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:12:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:13:01.646771585 +0000 UTC m=+550.123337966" watchObservedRunningTime="2026-10-31 01:13:01.647903811 +0000 UTC m=+550.124470191"
Oct 31 01:13:01.697890 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:13:01.697852    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-dls47"
Oct 31 01:13:01.738494 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:13:01.738444    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:13:04.595651 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:13:04.595608    4964 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-x8krt"]
Oct 31 01:13:04.601045 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:13:04.601016    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-x8krt"
Oct 31 01:13:04.657892 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:13:04.657857    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-x8krt"]
Oct 31 01:13:04.677815 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:13:04.677784    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/59416eef-55d7-4df2-8f28-a64f69ea4568-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-x8krt\" (UID: \"59416eef-55d7-4df2-8f28-a64f69ea4568\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-x8krt"
Oct 31 01:13:04.677981 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:13:04.677831    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-chvc8\" (UniqueName: \"kubernetes.io/projected/59416eef-55d7-4df2-8f28-a64f69ea4568-kube-api-access-chvc8\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-x8krt\" (UID: \"59416eef-55d7-4df2-8f28-a64f69ea4568\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-x8krt"
Oct 31 01:13:04.677981 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:13:04.677937    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/59416eef-55d7-4df2-8f28-a64f69ea4568-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-x8krt\" (UID: \"59416eef-55d7-4df2-8f28-a64f69ea4568\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-x8krt"
Oct 31 01:13:04.678090 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:13:04.678069    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/59416eef-55d7-4df2-8f28-a64f69ea4568-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-x8krt\" (UID: \"59416eef-55d7-4df2-8f28-a64f69ea4568\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-x8krt"
Oct 31 01:13:04.678153 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:13:04.678140    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/59416eef-55d7-4df2-8f28-a64f69ea4568-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-x8krt\" (UID: \"59416eef-55d7-4df2-8f28-a64f69ea4568\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-x8krt"
Oct 31 01:13:04.678190 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:13:04.678162    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/59416eef-55d7-4df2-8f28-a64f69ea4568-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-x8krt\" (UID: \"59416eef-55d7-4df2-8f28-a64f69ea4568\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-x8krt"
Oct 31 01:13:04.703701 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:13:04.703674    4964 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\""
Oct 31 01:13:04.735973 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:13:04.735939    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:13:04.778920 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:13:04.778888    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-chvc8\" (UniqueName: \"kubernetes.io/projected/59416eef-55d7-4df2-8f28-a64f69ea4568-kube-api-access-chvc8\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-x8krt\" (UID: \"59416eef-55d7-4df2-8f28-a64f69ea4568\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-x8krt"
Oct 31 01:13:04.779099 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:13:04.778933    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/59416eef-55d7-4df2-8f28-a64f69ea4568-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-x8krt\" (UID: \"59416eef-55d7-4df2-8f28-a64f69ea4568\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-x8krt"
Oct 31 01:13:04.779099 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:13:04.778983    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/59416eef-55d7-4df2-8f28-a64f69ea4568-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-x8krt\" (UID: \"59416eef-55d7-4df2-8f28-a64f69ea4568\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-x8krt"
Oct 31 01:13:04.779099 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:13:04.779015    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/59416eef-55d7-4df2-8f28-a64f69ea4568-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-x8krt\" (UID: \"59416eef-55d7-4df2-8f28-a64f69ea4568\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-x8krt"
Oct 31 01:13:04.779099 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:13:04.779030    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/59416eef-55d7-4df2-8f28-a64f69ea4568-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-x8krt\" (UID: \"59416eef-55d7-4df2-8f28-a64f69ea4568\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-x8krt"
Oct 31 01:13:04.779099 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:13:04.779048    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/59416eef-55d7-4df2-8f28-a64f69ea4568-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-x8krt\" (UID: \"59416eef-55d7-4df2-8f28-a64f69ea4568\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-x8krt"
Oct 31 01:13:04.779464 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:13:04.779443    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/59416eef-55d7-4df2-8f28-a64f69ea4568-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-x8krt\" (UID: \"59416eef-55d7-4df2-8f28-a64f69ea4568\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-x8krt"
Oct 31 01:13:04.779506 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:13:04.779451    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/59416eef-55d7-4df2-8f28-a64f69ea4568-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-x8krt\" (UID: \"59416eef-55d7-4df2-8f28-a64f69ea4568\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-x8krt"
Oct 31 01:13:04.779568 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:13:04.779533    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/59416eef-55d7-4df2-8f28-a64f69ea4568-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-x8krt\" (UID: \"59416eef-55d7-4df2-8f28-a64f69ea4568\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-x8krt"
Oct 31 01:13:04.782256 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:13:04.782232    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/59416eef-55d7-4df2-8f28-a64f69ea4568-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-x8krt\" (UID: \"59416eef-55d7-4df2-8f28-a64f69ea4568\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-x8krt"
Oct 31 01:13:04.782664 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:13:04.782642    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/59416eef-55d7-4df2-8f28-a64f69ea4568-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-x8krt\" (UID: \"59416eef-55d7-4df2-8f28-a64f69ea4568\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-x8krt"
Oct 31 01:13:04.837641 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:13:04.837613    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-chvc8\" (UniqueName: \"kubernetes.io/projected/59416eef-55d7-4df2-8f28-a64f69ea4568-kube-api-access-chvc8\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-x8krt\" (UID: \"59416eef-55d7-4df2-8f28-a64f69ea4568\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-x8krt"
Oct 31 01:13:04.917676 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:13:04.917598    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-x8krt"
Oct 31 01:13:05.301297 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:13:05.301242    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-x8krt"]
Oct 31 01:13:05.303691 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:13:05.303665    4964 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod59416eef_55d7_4df2_8f28_a64f69ea4568.slice/crio-928c7b2925c93eac31a720512cffe8679912977a0307825e13b161b726497eda WatchSource:0}: Error finding container 928c7b2925c93eac31a720512cffe8679912977a0307825e13b161b726497eda: Status 404 returned error can't find the container with id 928c7b2925c93eac31a720512cffe8679912977a0307825e13b161b726497eda
Oct 31 01:13:05.543446 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:13:05.543412    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:13:05.554767 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:13:05.554687    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-x8krt" event={"ID":"59416eef-55d7-4df2-8f28-a64f69ea4568","Type":"ContainerStarted","Data":"928c7b2925c93eac31a720512cffe8679912977a0307825e13b161b726497eda"}
Oct 31 01:13:06.561619 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:13:06.561583    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-x8krt" event={"ID":"59416eef-55d7-4df2-8f28-a64f69ea4568","Type":"ContainerStarted","Data":"5ca01b26cab1b01aee251c0798b43b7081334f85b7a8f6474ebd116428376721"}
Oct 31 01:13:06.678479 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:13:06.678425    4964 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-x8krt" podStartSLOduration=2.10265524 podStartE2EDuration="2.678411878s" totalImagesPullingTime="575.756638ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:13:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:13:06.676118986 +0000 UTC m=+555.152685378" watchObservedRunningTime="2026-10-31 01:13:06.678411878 +0000 UTC m=+555.154978258"
Oct 31 01:13:06.918545 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:13:06.918464    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-x8krt"
Oct 31 01:13:08.698249 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:13:08.698205    4964 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-dls47"
Oct 31 01:13:08.712190 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:13:08.712156    4964 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-dls47"
Oct 31 01:13:09.377250 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:13:09.377217    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:13:09.588512 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:13:09.588485    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-dls47"
Oct 31 01:13:11.238292 ip-10-0-133-197 kubenswrapper[4964]: E1031 01:13:11.238254    4964 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q56x5" containerName="extractor"
Oct 31 01:13:13.080185 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:13:13.080145    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:13:13.791528 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:13:13.791501    4964 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-2p64m"]
Oct 31 01:13:13.795260 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:13:13.795240    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/echo-simulator-kserve-64b6f8f96c-2p64m"
Oct 31 01:13:13.870747 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:13:13.870696    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b2dfcad6-e2bb-4e79-aa45-9f53c989349d-home\") pod \"echo-simulator-kserve-64b6f8f96c-2p64m\" (UID: \"b2dfcad6-e2bb-4e79-aa45-9f53c989349d\") " pod="llm/echo-simulator-kserve-64b6f8f96c-2p64m"
Oct 31 01:13:13.870927 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:13:13.870800    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b2dfcad6-e2bb-4e79-aa45-9f53c989349d-tmp-dir\") pod \"echo-simulator-kserve-64b6f8f96c-2p64m\" (UID: \"b2dfcad6-e2bb-4e79-aa45-9f53c989349d\") " pod="llm/echo-simulator-kserve-64b6f8f96c-2p64m"
Oct 31 01:13:13.870927 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:13:13.870895    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-598q6\" (UniqueName: \"kubernetes.io/projected/b2dfcad6-e2bb-4e79-aa45-9f53c989349d-kube-api-access-598q6\") pod \"echo-simulator-kserve-64b6f8f96c-2p64m\" (UID: \"b2dfcad6-e2bb-4e79-aa45-9f53c989349d\") " pod="llm/echo-simulator-kserve-64b6f8f96c-2p64m"
Oct 31 01:13:13.871023 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:13:13.870935    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b2dfcad6-e2bb-4e79-aa45-9f53c989349d-dshm\") pod \"echo-simulator-kserve-64b6f8f96c-2p64m\" (UID: \"b2dfcad6-e2bb-4e79-aa45-9f53c989349d\") " pod="llm/echo-simulator-kserve-64b6f8f96c-2p64m"
Oct 31 01:13:13.871023 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:13:13.870956    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b2dfcad6-e2bb-4e79-aa45-9f53c989349d-model-cache\") pod \"echo-simulator-kserve-64b6f8f96c-2p64m\" (UID: \"b2dfcad6-e2bb-4e79-aa45-9f53c989349d\") " pod="llm/echo-simulator-kserve-64b6f8f96c-2p64m"
Oct 31 01:13:13.871095 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:13:13.871026    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b2dfcad6-e2bb-4e79-aa45-9f53c989349d-tls-certs\") pod \"echo-simulator-kserve-64b6f8f96c-2p64m\" (UID: \"b2dfcad6-e2bb-4e79-aa45-9f53c989349d\") " pod="llm/echo-simulator-kserve-64b6f8f96c-2p64m"
Oct 31 01:13:13.897480 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:13:13.897449    4964 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"echo-simulator-kserve-self-signed-certs\""
Oct 31 01:13:13.899538 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:13:13.899518    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-2p64m"]
Oct 31 01:13:13.972428 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:13:13.972394    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b2dfcad6-e2bb-4e79-aa45-9f53c989349d-home\") pod \"echo-simulator-kserve-64b6f8f96c-2p64m\" (UID: \"b2dfcad6-e2bb-4e79-aa45-9f53c989349d\") " pod="llm/echo-simulator-kserve-64b6f8f96c-2p64m"
Oct 31 01:13:13.972609 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:13:13.972491    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b2dfcad6-e2bb-4e79-aa45-9f53c989349d-tmp-dir\") pod \"echo-simulator-kserve-64b6f8f96c-2p64m\" (UID: \"b2dfcad6-e2bb-4e79-aa45-9f53c989349d\") " pod="llm/echo-simulator-kserve-64b6f8f96c-2p64m"
Oct 31 01:13:13.972609 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:13:13.972545    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-598q6\" (UniqueName: \"kubernetes.io/projected/b2dfcad6-e2bb-4e79-aa45-9f53c989349d-kube-api-access-598q6\") pod \"echo-simulator-kserve-64b6f8f96c-2p64m\" (UID: \"b2dfcad6-e2bb-4e79-aa45-9f53c989349d\") " pod="llm/echo-simulator-kserve-64b6f8f96c-2p64m"
Oct 31 01:13:13.972609 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:13:13.972579    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b2dfcad6-e2bb-4e79-aa45-9f53c989349d-dshm\") pod \"echo-simulator-kserve-64b6f8f96c-2p64m\" (UID: \"b2dfcad6-e2bb-4e79-aa45-9f53c989349d\") " pod="llm/echo-simulator-kserve-64b6f8f96c-2p64m"
Oct 31 01:13:13.972765 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:13:13.972608    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b2dfcad6-e2bb-4e79-aa45-9f53c989349d-model-cache\") pod \"echo-simulator-kserve-64b6f8f96c-2p64m\" (UID: \"b2dfcad6-e2bb-4e79-aa45-9f53c989349d\") " pod="llm/echo-simulator-kserve-64b6f8f96c-2p64m"
Oct 31 01:13:13.972765 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:13:13.972710    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b2dfcad6-e2bb-4e79-aa45-9f53c989349d-tls-certs\") pod \"echo-simulator-kserve-64b6f8f96c-2p64m\" (UID: \"b2dfcad6-e2bb-4e79-aa45-9f53c989349d\") " pod="llm/echo-simulator-kserve-64b6f8f96c-2p64m"
Oct 31 01:13:13.972909 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:13:13.972883    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b2dfcad6-e2bb-4e79-aa45-9f53c989349d-home\") pod \"echo-simulator-kserve-64b6f8f96c-2p64m\" (UID: \"b2dfcad6-e2bb-4e79-aa45-9f53c989349d\") " pod="llm/echo-simulator-kserve-64b6f8f96c-2p64m"
Oct 31 01:13:13.972975 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:13:13.972930    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b2dfcad6-e2bb-4e79-aa45-9f53c989349d-tmp-dir\") pod \"echo-simulator-kserve-64b6f8f96c-2p64m\" (UID: \"b2dfcad6-e2bb-4e79-aa45-9f53c989349d\") " pod="llm/echo-simulator-kserve-64b6f8f96c-2p64m"
Oct 31 01:13:13.973033 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:13:13.972988    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b2dfcad6-e2bb-4e79-aa45-9f53c989349d-model-cache\") pod \"echo-simulator-kserve-64b6f8f96c-2p64m\" (UID: \"b2dfcad6-e2bb-4e79-aa45-9f53c989349d\") " pod="llm/echo-simulator-kserve-64b6f8f96c-2p64m"
Oct 31 01:13:13.975003 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:13:13.974980    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b2dfcad6-e2bb-4e79-aa45-9f53c989349d-dshm\") pod \"echo-simulator-kserve-64b6f8f96c-2p64m\" (UID: \"b2dfcad6-e2bb-4e79-aa45-9f53c989349d\") " pod="llm/echo-simulator-kserve-64b6f8f96c-2p64m"
Oct 31 01:13:13.975412 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:13:13.975396    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b2dfcad6-e2bb-4e79-aa45-9f53c989349d-tls-certs\") pod \"echo-simulator-kserve-64b6f8f96c-2p64m\" (UID: \"b2dfcad6-e2bb-4e79-aa45-9f53c989349d\") " pod="llm/echo-simulator-kserve-64b6f8f96c-2p64m"
Oct 31 01:13:14.030251 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:13:14.030220    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-598q6\" (UniqueName: \"kubernetes.io/projected/b2dfcad6-e2bb-4e79-aa45-9f53c989349d-kube-api-access-598q6\") pod \"echo-simulator-kserve-64b6f8f96c-2p64m\" (UID: \"b2dfcad6-e2bb-4e79-aa45-9f53c989349d\") " pod="llm/echo-simulator-kserve-64b6f8f96c-2p64m"
Oct 31 01:13:14.108884 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:13:14.108785    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/echo-simulator-kserve-64b6f8f96c-2p64m"
Oct 31 01:13:14.490423 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:13:14.490397    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-2p64m"]
Oct 31 01:13:14.492962 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:13:14.492939    4964 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2dfcad6_e2bb_4e79_aa45_9f53c989349d.slice/crio-6fb12babadbdc868b47d6ed872176a11f3b1635896ed212d6feb8cda568f38a4 WatchSource:0}: Error finding container 6fb12babadbdc868b47d6ed872176a11f3b1635896ed212d6feb8cda568f38a4: Status 404 returned error can't find the container with id 6fb12babadbdc868b47d6ed872176a11f3b1635896ed212d6feb8cda568f38a4
Oct 31 01:13:14.567866 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:13:14.567832    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:13:14.619648 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:13:14.619601    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/echo-simulator-kserve-64b6f8f96c-2p64m" event={"ID":"b2dfcad6-e2bb-4e79-aa45-9f53c989349d","Type":"ContainerStarted","Data":"6fb12babadbdc868b47d6ed872176a11f3b1635896ed212d6feb8cda568f38a4"}
Oct 31 01:13:14.800857 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:13:14.800776    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:13:14.917902 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:13:14.917870    4964 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-x8krt"
Oct 31 01:13:14.931010 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:13:14.930985    4964 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-x8krt"
Oct 31 01:13:15.636441 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:13:15.636411    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-x8krt"
Oct 31 01:13:17.282113 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:13:17.282075    4964 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-76tzr"]
Oct 31 01:13:17.285917 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:13:17.285897    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-76tzr"
Oct 31 01:13:17.341217 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:13:17.341180    4964 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\""
Oct 31 01:13:17.389713 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:13:17.389629    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-76tzr"]
Oct 31 01:13:17.405405 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:13:17.405357    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a10752ec-71ed-44a2-a3d0-e6f5ad807ce1-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-76tzr\" (UID: \"a10752ec-71ed-44a2-a3d0-e6f5ad807ce1\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-76tzr"
Oct 31 01:13:17.405587 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:13:17.405430    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a10752ec-71ed-44a2-a3d0-e6f5ad807ce1-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-76tzr\" (UID: \"a10752ec-71ed-44a2-a3d0-e6f5ad807ce1\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-76tzr"
Oct 31 01:13:17.405587 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:13:17.405533    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bdn54\" (UniqueName: \"kubernetes.io/projected/a10752ec-71ed-44a2-a3d0-e6f5ad807ce1-kube-api-access-bdn54\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-76tzr\" (UID: \"a10752ec-71ed-44a2-a3d0-e6f5ad807ce1\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-76tzr"
Oct 31 01:13:17.405676 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:13:17.405634    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a10752ec-71ed-44a2-a3d0-e6f5ad807ce1-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-76tzr\" (UID: \"a10752ec-71ed-44a2-a3d0-e6f5ad807ce1\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-76tzr"
Oct 31 01:13:17.405676 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:13:17.405658    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a10752ec-71ed-44a2-a3d0-e6f5ad807ce1-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-76tzr\" (UID: \"a10752ec-71ed-44a2-a3d0-e6f5ad807ce1\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-76tzr"
Oct 31 01:13:17.405748 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:13:17.405677    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a10752ec-71ed-44a2-a3d0-e6f5ad807ce1-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-76tzr\" (UID: \"a10752ec-71ed-44a2-a3d0-e6f5ad807ce1\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-76tzr"
Oct 31 01:13:17.433663 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:13:17.433617    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:13:17.506840 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:13:17.506804    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a10752ec-71ed-44a2-a3d0-e6f5ad807ce1-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-76tzr\" (UID: \"a10752ec-71ed-44a2-a3d0-e6f5ad807ce1\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-76tzr"
Oct 31 01:13:17.507033 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:13:17.506855    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a10752ec-71ed-44a2-a3d0-e6f5ad807ce1-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-76tzr\" (UID: \"a10752ec-71ed-44a2-a3d0-e6f5ad807ce1\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-76tzr"
Oct 31 01:13:17.507033 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:13:17.506889    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a10752ec-71ed-44a2-a3d0-e6f5ad807ce1-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-76tzr\" (UID: \"a10752ec-71ed-44a2-a3d0-e6f5ad807ce1\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-76tzr"
Oct 31 01:13:17.507033 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:13:17.506938    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a10752ec-71ed-44a2-a3d0-e6f5ad807ce1-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-76tzr\" (UID: \"a10752ec-71ed-44a2-a3d0-e6f5ad807ce1\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-76tzr"
Oct 31 01:13:17.507226 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:13:17.507137    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a10752ec-71ed-44a2-a3d0-e6f5ad807ce1-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-76tzr\" (UID: \"a10752ec-71ed-44a2-a3d0-e6f5ad807ce1\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-76tzr"
Oct 31 01:13:17.507226 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:13:17.507203    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bdn54\" (UniqueName: \"kubernetes.io/projected/a10752ec-71ed-44a2-a3d0-e6f5ad807ce1-kube-api-access-bdn54\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-76tzr\" (UID: \"a10752ec-71ed-44a2-a3d0-e6f5ad807ce1\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-76tzr"
Oct 31 01:13:17.507415 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:13:17.507389    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a10752ec-71ed-44a2-a3d0-e6f5ad807ce1-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-76tzr\" (UID: \"a10752ec-71ed-44a2-a3d0-e6f5ad807ce1\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-76tzr"
Oct 31 01:13:17.507499 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:13:17.507421    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a10752ec-71ed-44a2-a3d0-e6f5ad807ce1-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-76tzr\" (UID: \"a10752ec-71ed-44a2-a3d0-e6f5ad807ce1\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-76tzr"
Oct 31 01:13:17.507670 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:13:17.507634    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a10752ec-71ed-44a2-a3d0-e6f5ad807ce1-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-76tzr\" (UID: \"a10752ec-71ed-44a2-a3d0-e6f5ad807ce1\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-76tzr"
Oct 31 01:13:17.509662 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:13:17.509634    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a10752ec-71ed-44a2-a3d0-e6f5ad807ce1-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-76tzr\" (UID: \"a10752ec-71ed-44a2-a3d0-e6f5ad807ce1\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-76tzr"
Oct 31 01:13:17.510192 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:13:17.510169    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a10752ec-71ed-44a2-a3d0-e6f5ad807ce1-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-76tzr\" (UID: \"a10752ec-71ed-44a2-a3d0-e6f5ad807ce1\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-76tzr"
Oct 31 01:13:17.564634 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:13:17.564597    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bdn54\" (UniqueName: \"kubernetes.io/projected/a10752ec-71ed-44a2-a3d0-e6f5ad807ce1-kube-api-access-bdn54\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-76tzr\" (UID: \"a10752ec-71ed-44a2-a3d0-e6f5ad807ce1\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-76tzr"
Oct 31 01:13:17.599928 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:13:17.599883    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-76tzr"
Oct 31 01:13:17.644013 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:13:17.643926    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/echo-simulator-kserve-64b6f8f96c-2p64m" event={"ID":"b2dfcad6-e2bb-4e79-aa45-9f53c989349d","Type":"ContainerStarted","Data":"01cd190ebfa96f644db1f561397d6c0dd7790a217a58ba943b47f33d4743fd6f"}
Oct 31 01:13:17.762789 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:13:17.762724    4964 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/echo-simulator-kserve-64b6f8f96c-2p64m" podStartSLOduration=2.142973983 podStartE2EDuration="4.762706786s" totalImagesPullingTime="2.619732803s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:13:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:13:17.761225039 +0000 UTC m=+566.237791444" watchObservedRunningTime="2026-10-31 01:13:17.762706786 +0000 UTC m=+566.239273168"
Oct 31 01:13:17.983731 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:13:17.983702    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-76tzr"]
Oct 31 01:13:17.985934 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:13:17.985904    4964 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda10752ec_71ed_44a2_a3d0_e6f5ad807ce1.slice/crio-6d737fef227a5fa42fe40f89fcfd65345b68783c401cc087b95571c3c18ec21e WatchSource:0}: Error finding container 6d737fef227a5fa42fe40f89fcfd65345b68783c401cc087b95571c3c18ec21e: Status 404 returned error can't find the container with id 6d737fef227a5fa42fe40f89fcfd65345b68783c401cc087b95571c3c18ec21e
Oct 31 01:13:18.110231 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:13:18.110195    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/echo-simulator-kserve-64b6f8f96c-2p64m"
Oct 31 01:13:18.650117 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:13:18.650085    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-76tzr" event={"ID":"a10752ec-71ed-44a2-a3d0-e6f5ad807ce1","Type":"ContainerStarted","Data":"6d737fef227a5fa42fe40f89fcfd65345b68783c401cc087b95571c3c18ec21e"}
Oct 31 01:13:19.656669 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:13:19.656621    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-76tzr" event={"ID":"a10752ec-71ed-44a2-a3d0-e6f5ad807ce1","Type":"ContainerStarted","Data":"f78d18a98694355eed53205b27a1aa6779af9b4872901ce053c6b7d92922e458"}
Oct 31 01:13:19.774164 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:13:19.774111    4964 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-76tzr" podStartSLOduration=2.172042645 podStartE2EDuration="2.774095913s" totalImagesPullingTime="602.053268ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:13:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:13:19.771877472 +0000 UTC m=+568.248443866" watchObservedRunningTime="2026-10-31 01:13:19.774095913 +0000 UTC m=+568.250662292"
Oct 31 01:13:20.582734 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:13:20.582671    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:13:20.600850 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:13:20.600811    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-76tzr"
Oct 31 01:13:24.109729 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:13:24.109685    4964 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/echo-simulator-kserve-64b6f8f96c-2p64m"
Oct 31 01:13:24.122748 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:13:24.122720    4964 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/echo-simulator-kserve-64b6f8f96c-2p64m"
Oct 31 01:13:24.694696 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:13:24.694670    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/echo-simulator-kserve-64b6f8f96c-2p64m"
Oct 31 01:13:27.600289 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:13:27.600255    4964 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-76tzr"
Oct 31 01:13:27.613468 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:13:27.613436    4964 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-76tzr"
Oct 31 01:13:27.711527 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:13:27.711499    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-76tzr"
Oct 31 01:13:34.623727 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:13:34.623596    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:13:41.237612 ip-10-0-133-197 kubenswrapper[4964]: E1031 01:13:41.237574    4964 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q56x5" containerName="extractor"
Oct 31 01:13:44.113331 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:13:44.113279    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:13:49.359733 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:13:49.359688    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:13:49.553379 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:13:49.553310    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:13:50.967610 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:13:50.967566    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:13:51.228774 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:13:51.228730    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:13:51.830489 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:13:51.830463    4964 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-9257n_f0c0ada7-7688-4dc2-97fc-813e4d0c4ef3/console-operator/2.log"
Oct 31 01:13:51.831569 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:13:51.831540    4964 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-9257n_f0c0ada7-7688-4dc2-97fc-813e4d0c4ef3/console-operator/2.log"
Oct 31 01:13:54.178714 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:13:54.178678    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:13:56.372220 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:13:56.372177    4964 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-jd7bn"]
Oct 31 01:13:56.377539 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:13:56.377513    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-jd7bn"
Oct 31 01:13:56.443721 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:13:56.443682    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8qtd9\" (UniqueName: \"kubernetes.io/projected/2bf44719-ff6b-4ada-9fe7-a6f466992512-kube-api-access-8qtd9\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-jd7bn\" (UID: \"2bf44719-ff6b-4ada-9fe7-a6f466992512\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-jd7bn"
Oct 31 01:13:56.443904 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:13:56.443727    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2bf44719-ff6b-4ada-9fe7-a6f466992512-istio-data\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-jd7bn\" (UID: \"2bf44719-ff6b-4ada-9fe7-a6f466992512\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-jd7bn"
Oct 31 01:13:56.443904 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:13:56.443798    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2bf44719-ff6b-4ada-9fe7-a6f466992512-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-jd7bn\" (UID: \"2bf44719-ff6b-4ada-9fe7-a6f466992512\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-jd7bn"
Oct 31 01:13:56.443904 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:13:56.443859    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2bf44719-ff6b-4ada-9fe7-a6f466992512-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-jd7bn\" (UID: \"2bf44719-ff6b-4ada-9fe7-a6f466992512\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-jd7bn"
Oct 31 01:13:56.444046 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:13:56.443910    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2bf44719-ff6b-4ada-9fe7-a6f466992512-workload-certs\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-jd7bn\" (UID: \"2bf44719-ff6b-4ada-9fe7-a6f466992512\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-jd7bn"
Oct 31 01:13:56.444046 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:13:56.443965    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2bf44719-ff6b-4ada-9fe7-a6f466992512-workload-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-jd7bn\" (UID: \"2bf44719-ff6b-4ada-9fe7-a6f466992512\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-jd7bn"
Oct 31 01:13:56.444046 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:13:56.444033    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2bf44719-ff6b-4ada-9fe7-a6f466992512-istio-envoy\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-jd7bn\" (UID: \"2bf44719-ff6b-4ada-9fe7-a6f466992512\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-jd7bn"
Oct 31 01:13:56.444137 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:13:56.444052    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2bf44719-ff6b-4ada-9fe7-a6f466992512-istio-token\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-jd7bn\" (UID: \"2bf44719-ff6b-4ada-9fe7-a6f466992512\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-jd7bn"
Oct 31 01:13:56.444137 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:13:56.444077    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2bf44719-ff6b-4ada-9fe7-a6f466992512-credential-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-jd7bn\" (UID: \"2bf44719-ff6b-4ada-9fe7-a6f466992512\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-jd7bn"
Oct 31 01:13:56.481479 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:13:56.481447    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-jd7bn"]
Oct 31 01:13:56.544644 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:13:56.544607    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8qtd9\" (UniqueName: \"kubernetes.io/projected/2bf44719-ff6b-4ada-9fe7-a6f466992512-kube-api-access-8qtd9\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-jd7bn\" (UID: \"2bf44719-ff6b-4ada-9fe7-a6f466992512\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-jd7bn"
Oct 31 01:13:56.544644 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:13:56.544648    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2bf44719-ff6b-4ada-9fe7-a6f466992512-istio-data\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-jd7bn\" (UID: \"2bf44719-ff6b-4ada-9fe7-a6f466992512\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-jd7bn"
Oct 31 01:13:56.544860 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:13:56.544672    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2bf44719-ff6b-4ada-9fe7-a6f466992512-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-jd7bn\" (UID: \"2bf44719-ff6b-4ada-9fe7-a6f466992512\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-jd7bn"
Oct 31 01:13:56.544860 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:13:56.544798    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2bf44719-ff6b-4ada-9fe7-a6f466992512-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-jd7bn\" (UID: \"2bf44719-ff6b-4ada-9fe7-a6f466992512\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-jd7bn"
Oct 31 01:13:56.544952 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:13:56.544880    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2bf44719-ff6b-4ada-9fe7-a6f466992512-workload-certs\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-jd7bn\" (UID: \"2bf44719-ff6b-4ada-9fe7-a6f466992512\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-jd7bn"
Oct 31 01:13:56.545007 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:13:56.544994    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2bf44719-ff6b-4ada-9fe7-a6f466992512-workload-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-jd7bn\" (UID: \"2bf44719-ff6b-4ada-9fe7-a6f466992512\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-jd7bn"
Oct 31 01:13:56.545057 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:13:56.545028    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2bf44719-ff6b-4ada-9fe7-a6f466992512-istio-envoy\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-jd7bn\" (UID: \"2bf44719-ff6b-4ada-9fe7-a6f466992512\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-jd7bn"
Oct 31 01:13:56.545116 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:13:56.545054    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2bf44719-ff6b-4ada-9fe7-a6f466992512-istio-token\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-jd7bn\" (UID: \"2bf44719-ff6b-4ada-9fe7-a6f466992512\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-jd7bn"
Oct 31 01:13:56.545116 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:13:56.545085    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2bf44719-ff6b-4ada-9fe7-a6f466992512-credential-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-jd7bn\" (UID: \"2bf44719-ff6b-4ada-9fe7-a6f466992512\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-jd7bn"
Oct 31 01:13:56.545116 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:13:56.545098    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2bf44719-ff6b-4ada-9fe7-a6f466992512-istio-data\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-jd7bn\" (UID: \"2bf44719-ff6b-4ada-9fe7-a6f466992512\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-jd7bn"
Oct 31 01:13:56.545278 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:13:56.545222    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2bf44719-ff6b-4ada-9fe7-a6f466992512-workload-certs\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-jd7bn\" (UID: \"2bf44719-ff6b-4ada-9fe7-a6f466992512\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-jd7bn"
Oct 31 01:13:56.545513 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:13:56.545472    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2bf44719-ff6b-4ada-9fe7-a6f466992512-credential-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-jd7bn\" (UID: \"2bf44719-ff6b-4ada-9fe7-a6f466992512\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-jd7bn"
Oct 31 01:13:56.545741 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:13:56.545725    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2bf44719-ff6b-4ada-9fe7-a6f466992512-workload-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-jd7bn\" (UID: \"2bf44719-ff6b-4ada-9fe7-a6f466992512\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-jd7bn"
Oct 31 01:13:56.545789 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:13:56.545723    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2bf44719-ff6b-4ada-9fe7-a6f466992512-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-jd7bn\" (UID: \"2bf44719-ff6b-4ada-9fe7-a6f466992512\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-jd7bn"
Oct 31 01:13:56.547334 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:13:56.547302    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2bf44719-ff6b-4ada-9fe7-a6f466992512-istio-envoy\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-jd7bn\" (UID: \"2bf44719-ff6b-4ada-9fe7-a6f466992512\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-jd7bn"
Oct 31 01:13:56.547430 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:13:56.547388    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2bf44719-ff6b-4ada-9fe7-a6f466992512-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-jd7bn\" (UID: \"2bf44719-ff6b-4ada-9fe7-a6f466992512\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-jd7bn"
Oct 31 01:13:56.602741 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:13:56.602705    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2bf44719-ff6b-4ada-9fe7-a6f466992512-istio-token\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-jd7bn\" (UID: \"2bf44719-ff6b-4ada-9fe7-a6f466992512\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-jd7bn"
Oct 31 01:13:56.602907 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:13:56.602884    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8qtd9\" (UniqueName: \"kubernetes.io/projected/2bf44719-ff6b-4ada-9fe7-a6f466992512-kube-api-access-8qtd9\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-jd7bn\" (UID: \"2bf44719-ff6b-4ada-9fe7-a6f466992512\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-jd7bn"
Oct 31 01:13:56.690716 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:13:56.690624    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-jd7bn"
Oct 31 01:13:57.118382 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:13:57.118357    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-jd7bn"]
Oct 31 01:13:57.120272 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:13:57.120247    4964 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2bf44719_ff6b_4ada_9fe7_a6f466992512.slice/crio-9ac92f8e8af1ae59ddc7d88c96d1deab689c1d28c5c9bc9fc83ee4bac76f843b WatchSource:0}: Error finding container 9ac92f8e8af1ae59ddc7d88c96d1deab689c1d28c5c9bc9fc83ee4bac76f843b: Status 404 returned error can't find the container with id 9ac92f8e8af1ae59ddc7d88c96d1deab689c1d28c5c9bc9fc83ee4bac76f843b
Oct 31 01:13:57.122376 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:13:57.122347    4964 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 01:13:57.122441 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:13:57.122424    4964 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 01:13:57.122479 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:13:57.122470    4964 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 01:13:57.874951 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:13:57.874912    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-jd7bn" event={"ID":"2bf44719-ff6b-4ada-9fe7-a6f466992512","Type":"ContainerStarted","Data":"5d9493ca37bee237a56b1d5009b41f338585064db868ef1f43c6a72b70aa582a"}
Oct 31 01:13:57.875404 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:13:57.874976    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-jd7bn" event={"ID":"2bf44719-ff6b-4ada-9fe7-a6f466992512","Type":"ContainerStarted","Data":"9ac92f8e8af1ae59ddc7d88c96d1deab689c1d28c5c9bc9fc83ee4bac76f843b"}
Oct 31 01:13:58.049932 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:13:58.049875    4964 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-jd7bn" podStartSLOduration=2.049859421 podStartE2EDuration="2.049859421s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:13:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:13:58.040247242 +0000 UTC m=+606.516813634" watchObservedRunningTime="2026-10-31 01:13:58.049859421 +0000 UTC m=+606.526425801"
Oct 31 01:13:58.704342 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:13:58.703748    4964 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-jd7bn"
Oct 31 01:13:58.708341 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:13:58.706757    4964 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-jd7bn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" start-of-body=
Oct 31 01:13:58.708341 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:13:58.706813    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-jd7bn" podUID="2bf44719-ff6b-4ada-9fe7-a6f466992512" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused"
Oct 31 01:13:58.730592 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:13:58.730556    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:13:58.916016 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:13:58.915979    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:13:59.695718 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:13:59.695681    4964 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-jd7bn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" start-of-body=
Oct 31 01:13:59.695997 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:13:59.695971    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-jd7bn" podUID="2bf44719-ff6b-4ada-9fe7-a6f466992512" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused"
Oct 31 01:14:00.702345 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:14:00.694718    4964 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-jd7bn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" start-of-body=
Oct 31 01:14:00.702345 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:14:00.694787    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-jd7bn" podUID="2bf44719-ff6b-4ada-9fe7-a6f466992512" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused"
Oct 31 01:14:01.705354 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:14:01.703944    4964 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-jd7bn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" start-of-body=
Oct 31 01:14:01.705354 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:14:01.704007    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-jd7bn" podUID="2bf44719-ff6b-4ada-9fe7-a6f466992512" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused"
Oct 31 01:14:02.706345 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:14:02.699989    4964 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-jd7bn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" start-of-body=
Oct 31 01:14:02.717438 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:14:02.717388    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-jd7bn" podUID="2bf44719-ff6b-4ada-9fe7-a6f466992512" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused"
Oct 31 01:14:03.691263 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:14:03.691225    4964 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-jd7bn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" start-of-body=
Oct 31 01:14:03.691565 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:14:03.691285    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-jd7bn" podUID="2bf44719-ff6b-4ada-9fe7-a6f466992512" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused"
Oct 31 01:14:05.453007 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:14:05.452957    4964 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-jd7bn"
Oct 31 01:14:05.453515 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:14:05.453376    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-jd7bn"
Oct 31 01:14:05.454289 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:14:05.454266    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-jd7bn"
Oct 31 01:14:11.237848 ip-10-0-133-197 kubenswrapper[4964]: E1031 01:14:11.237807    4964 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q56x5" containerName="extractor"
Oct 31 01:14:23.428477 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:14:23.428436    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:14:40.090036 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:14:40.089999    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:14:40.338029 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:14:40.337995    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:14:40.618693 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:14:40.618653    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:14:40.991623 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:14:40.991586    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:14:41.247014 ip-10-0-133-197 kubenswrapper[4964]: E1031 01:14:41.246915    4964 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q56x5" containerName="extractor"
Oct 31 01:14:41.279182 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:14:41.279141    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:14:41.442990 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:14:41.442956    4964 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-migrate-a0c7a371-gw-openshift-default-7857fb8585-vnnng"]
Oct 31 01:14:41.446601 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:14:41.446577    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-a0c7a371-gw-openshift-default-7857fb8585-vnnng"
Oct 31 01:14:41.498871 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:14:41.498776    4964 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-a0c7a371-gw-openshift-default-dockercfg-8pvrs\""
Oct 31 01:14:41.518213 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:14:41.518180    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/41125f0c-f8ce-416b-92b0-4d6535255a5e-workload-certs\") pod \"e2e-migrate-a0c7a371-gw-openshift-default-7857fb8585-vnnng\" (UID: \"41125f0c-f8ce-416b-92b0-4d6535255a5e\") " pod="openshift-ingress/e2e-migrate-a0c7a371-gw-openshift-default-7857fb8585-vnnng"
Oct 31 01:14:41.518425 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:14:41.518219    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xcx5h\" (UniqueName: \"kubernetes.io/projected/41125f0c-f8ce-416b-92b0-4d6535255a5e-kube-api-access-xcx5h\") pod \"e2e-migrate-a0c7a371-gw-openshift-default-7857fb8585-vnnng\" (UID: \"41125f0c-f8ce-416b-92b0-4d6535255a5e\") " pod="openshift-ingress/e2e-migrate-a0c7a371-gw-openshift-default-7857fb8585-vnnng"
Oct 31 01:14:41.518425 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:14:41.518253    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/41125f0c-f8ce-416b-92b0-4d6535255a5e-istio-token\") pod \"e2e-migrate-a0c7a371-gw-openshift-default-7857fb8585-vnnng\" (UID: \"41125f0c-f8ce-416b-92b0-4d6535255a5e\") " pod="openshift-ingress/e2e-migrate-a0c7a371-gw-openshift-default-7857fb8585-vnnng"
Oct 31 01:14:41.518425 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:14:41.518343    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/41125f0c-f8ce-416b-92b0-4d6535255a5e-istio-data\") pod \"e2e-migrate-a0c7a371-gw-openshift-default-7857fb8585-vnnng\" (UID: \"41125f0c-f8ce-416b-92b0-4d6535255a5e\") " pod="openshift-ingress/e2e-migrate-a0c7a371-gw-openshift-default-7857fb8585-vnnng"
Oct 31 01:14:41.518425 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:14:41.518412    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/41125f0c-f8ce-416b-92b0-4d6535255a5e-credential-socket\") pod \"e2e-migrate-a0c7a371-gw-openshift-default-7857fb8585-vnnng\" (UID: \"41125f0c-f8ce-416b-92b0-4d6535255a5e\") " pod="openshift-ingress/e2e-migrate-a0c7a371-gw-openshift-default-7857fb8585-vnnng"
Oct 31 01:14:41.518634 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:14:41.518455    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/41125f0c-f8ce-416b-92b0-4d6535255a5e-workload-socket\") pod \"e2e-migrate-a0c7a371-gw-openshift-default-7857fb8585-vnnng\" (UID: \"41125f0c-f8ce-416b-92b0-4d6535255a5e\") " pod="openshift-ingress/e2e-migrate-a0c7a371-gw-openshift-default-7857fb8585-vnnng"
Oct 31 01:14:41.518634 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:14:41.518507    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/41125f0c-f8ce-416b-92b0-4d6535255a5e-istio-envoy\") pod \"e2e-migrate-a0c7a371-gw-openshift-default-7857fb8585-vnnng\" (UID: \"41125f0c-f8ce-416b-92b0-4d6535255a5e\") " pod="openshift-ingress/e2e-migrate-a0c7a371-gw-openshift-default-7857fb8585-vnnng"
Oct 31 01:14:41.518634 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:14:41.518527    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/41125f0c-f8ce-416b-92b0-4d6535255a5e-istiod-ca-cert\") pod \"e2e-migrate-a0c7a371-gw-openshift-default-7857fb8585-vnnng\" (UID: \"41125f0c-f8ce-416b-92b0-4d6535255a5e\") " pod="openshift-ingress/e2e-migrate-a0c7a371-gw-openshift-default-7857fb8585-vnnng"
Oct 31 01:14:41.518634 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:14:41.518615    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/41125f0c-f8ce-416b-92b0-4d6535255a5e-istio-podinfo\") pod \"e2e-migrate-a0c7a371-gw-openshift-default-7857fb8585-vnnng\" (UID: \"41125f0c-f8ce-416b-92b0-4d6535255a5e\") " pod="openshift-ingress/e2e-migrate-a0c7a371-gw-openshift-default-7857fb8585-vnnng"
Oct 31 01:14:41.550552 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:14:41.550517    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-a0c7a371-gw-openshift-default-7857fb8585-vnnng"]
Oct 31 01:14:41.584194 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:14:41.584158    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:14:41.619938 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:14:41.619898    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/41125f0c-f8ce-416b-92b0-4d6535255a5e-workload-certs\") pod \"e2e-migrate-a0c7a371-gw-openshift-default-7857fb8585-vnnng\" (UID: \"41125f0c-f8ce-416b-92b0-4d6535255a5e\") " pod="openshift-ingress/e2e-migrate-a0c7a371-gw-openshift-default-7857fb8585-vnnng"
Oct 31 01:14:41.619938 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:14:41.619944    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xcx5h\" (UniqueName: \"kubernetes.io/projected/41125f0c-f8ce-416b-92b0-4d6535255a5e-kube-api-access-xcx5h\") pod \"e2e-migrate-a0c7a371-gw-openshift-default-7857fb8585-vnnng\" (UID: \"41125f0c-f8ce-416b-92b0-4d6535255a5e\") " pod="openshift-ingress/e2e-migrate-a0c7a371-gw-openshift-default-7857fb8585-vnnng"
Oct 31 01:14:41.620201 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:14:41.619978    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/41125f0c-f8ce-416b-92b0-4d6535255a5e-istio-token\") pod \"e2e-migrate-a0c7a371-gw-openshift-default-7857fb8585-vnnng\" (UID: \"41125f0c-f8ce-416b-92b0-4d6535255a5e\") " pod="openshift-ingress/e2e-migrate-a0c7a371-gw-openshift-default-7857fb8585-vnnng"
Oct 31 01:14:41.620201 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:14:41.619995    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/41125f0c-f8ce-416b-92b0-4d6535255a5e-istio-data\") pod \"e2e-migrate-a0c7a371-gw-openshift-default-7857fb8585-vnnng\" (UID: \"41125f0c-f8ce-416b-92b0-4d6535255a5e\") " pod="openshift-ingress/e2e-migrate-a0c7a371-gw-openshift-default-7857fb8585-vnnng"
Oct 31 01:14:41.620201 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:14:41.620018    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/41125f0c-f8ce-416b-92b0-4d6535255a5e-credential-socket\") pod \"e2e-migrate-a0c7a371-gw-openshift-default-7857fb8585-vnnng\" (UID: \"41125f0c-f8ce-416b-92b0-4d6535255a5e\") " pod="openshift-ingress/e2e-migrate-a0c7a371-gw-openshift-default-7857fb8585-vnnng"
Oct 31 01:14:41.620201 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:14:41.620052    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/41125f0c-f8ce-416b-92b0-4d6535255a5e-workload-socket\") pod \"e2e-migrate-a0c7a371-gw-openshift-default-7857fb8585-vnnng\" (UID: \"41125f0c-f8ce-416b-92b0-4d6535255a5e\") " pod="openshift-ingress/e2e-migrate-a0c7a371-gw-openshift-default-7857fb8585-vnnng"
Oct 31 01:14:41.620201 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:14:41.620077    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/41125f0c-f8ce-416b-92b0-4d6535255a5e-istio-envoy\") pod \"e2e-migrate-a0c7a371-gw-openshift-default-7857fb8585-vnnng\" (UID: \"41125f0c-f8ce-416b-92b0-4d6535255a5e\") " pod="openshift-ingress/e2e-migrate-a0c7a371-gw-openshift-default-7857fb8585-vnnng"
Oct 31 01:14:41.620201 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:14:41.620100    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/41125f0c-f8ce-416b-92b0-4d6535255a5e-istiod-ca-cert\") pod \"e2e-migrate-a0c7a371-gw-openshift-default-7857fb8585-vnnng\" (UID: \"41125f0c-f8ce-416b-92b0-4d6535255a5e\") " pod="openshift-ingress/e2e-migrate-a0c7a371-gw-openshift-default-7857fb8585-vnnng"
Oct 31 01:14:41.620567 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:14:41.620254    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/41125f0c-f8ce-416b-92b0-4d6535255a5e-istio-podinfo\") pod \"e2e-migrate-a0c7a371-gw-openshift-default-7857fb8585-vnnng\" (UID: \"41125f0c-f8ce-416b-92b0-4d6535255a5e\") " pod="openshift-ingress/e2e-migrate-a0c7a371-gw-openshift-default-7857fb8585-vnnng"
Oct 31 01:14:41.620567 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:14:41.620505    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/41125f0c-f8ce-416b-92b0-4d6535255a5e-workload-certs\") pod \"e2e-migrate-a0c7a371-gw-openshift-default-7857fb8585-vnnng\" (UID: \"41125f0c-f8ce-416b-92b0-4d6535255a5e\") " pod="openshift-ingress/e2e-migrate-a0c7a371-gw-openshift-default-7857fb8585-vnnng"
Oct 31 01:14:41.620671 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:14:41.620566    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/41125f0c-f8ce-416b-92b0-4d6535255a5e-credential-socket\") pod \"e2e-migrate-a0c7a371-gw-openshift-default-7857fb8585-vnnng\" (UID: \"41125f0c-f8ce-416b-92b0-4d6535255a5e\") " pod="openshift-ingress/e2e-migrate-a0c7a371-gw-openshift-default-7857fb8585-vnnng"
Oct 31 01:14:41.620727 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:14:41.620703    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/41125f0c-f8ce-416b-92b0-4d6535255a5e-istio-data\") pod \"e2e-migrate-a0c7a371-gw-openshift-default-7857fb8585-vnnng\" (UID: \"41125f0c-f8ce-416b-92b0-4d6535255a5e\") " pod="openshift-ingress/e2e-migrate-a0c7a371-gw-openshift-default-7857fb8585-vnnng"
Oct 31 01:14:41.620832 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:14:41.620810    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/41125f0c-f8ce-416b-92b0-4d6535255a5e-workload-socket\") pod \"e2e-migrate-a0c7a371-gw-openshift-default-7857fb8585-vnnng\" (UID: \"41125f0c-f8ce-416b-92b0-4d6535255a5e\") " pod="openshift-ingress/e2e-migrate-a0c7a371-gw-openshift-default-7857fb8585-vnnng"
Oct 31 01:14:41.621036 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:14:41.621016    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/41125f0c-f8ce-416b-92b0-4d6535255a5e-istiod-ca-cert\") pod \"e2e-migrate-a0c7a371-gw-openshift-default-7857fb8585-vnnng\" (UID: \"41125f0c-f8ce-416b-92b0-4d6535255a5e\") " pod="openshift-ingress/e2e-migrate-a0c7a371-gw-openshift-default-7857fb8585-vnnng"
Oct 31 01:14:41.622761 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:14:41.622739    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/41125f0c-f8ce-416b-92b0-4d6535255a5e-istio-envoy\") pod \"e2e-migrate-a0c7a371-gw-openshift-default-7857fb8585-vnnng\" (UID: \"41125f0c-f8ce-416b-92b0-4d6535255a5e\") " pod="openshift-ingress/e2e-migrate-a0c7a371-gw-openshift-default-7857fb8585-vnnng"
Oct 31 01:14:41.622965 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:14:41.622933    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/41125f0c-f8ce-416b-92b0-4d6535255a5e-istio-podinfo\") pod \"e2e-migrate-a0c7a371-gw-openshift-default-7857fb8585-vnnng\" (UID: \"41125f0c-f8ce-416b-92b0-4d6535255a5e\") " pod="openshift-ingress/e2e-migrate-a0c7a371-gw-openshift-default-7857fb8585-vnnng"
Oct 31 01:14:41.676966 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:14:41.676936    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/41125f0c-f8ce-416b-92b0-4d6535255a5e-istio-token\") pod \"e2e-migrate-a0c7a371-gw-openshift-default-7857fb8585-vnnng\" (UID: \"41125f0c-f8ce-416b-92b0-4d6535255a5e\") " pod="openshift-ingress/e2e-migrate-a0c7a371-gw-openshift-default-7857fb8585-vnnng"
Oct 31 01:14:41.677543 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:14:41.677522    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xcx5h\" (UniqueName: \"kubernetes.io/projected/41125f0c-f8ce-416b-92b0-4d6535255a5e-kube-api-access-xcx5h\") pod \"e2e-migrate-a0c7a371-gw-openshift-default-7857fb8585-vnnng\" (UID: \"41125f0c-f8ce-416b-92b0-4d6535255a5e\") " pod="openshift-ingress/e2e-migrate-a0c7a371-gw-openshift-default-7857fb8585-vnnng"
Oct 31 01:14:41.761017 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:14:41.760926    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-a0c7a371-gw-openshift-default-7857fb8585-vnnng"
Oct 31 01:14:41.775899 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:14:41.775865    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:14:42.050707 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:14:42.050600    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:14:42.195241 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:14:42.195217    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-a0c7a371-gw-openshift-default-7857fb8585-vnnng"]
Oct 31 01:14:42.197390 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:14:42.197358    4964 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41125f0c_f8ce_416b_92b0_4d6535255a5e.slice/crio-bc34b6c8524cf0689b48aa4496d44d0767fffcfb11969d052644855c5fc2d4a8 WatchSource:0}: Error finding container bc34b6c8524cf0689b48aa4496d44d0767fffcfb11969d052644855c5fc2d4a8: Status 404 returned error can't find the container with id bc34b6c8524cf0689b48aa4496d44d0767fffcfb11969d052644855c5fc2d4a8
Oct 31 01:14:42.199514 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:14:42.199483    4964 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 01:14:42.199602 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:14:42.199549    4964 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 01:14:42.199602 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:14:42.199580    4964 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 01:14:42.800561 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:14:42.800529    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:14:43.040287 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:14:43.040246    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:14:43.156872 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:14:43.156774    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-a0c7a371-gw-openshift-default-7857fb8585-vnnng" event={"ID":"41125f0c-f8ce-416b-92b0-4d6535255a5e","Type":"ContainerStarted","Data":"d9f691765b032b8ac0927bcead1177abb538a4eb46b02af7d90f649c077e5c31"}
Oct 31 01:14:43.156872 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:14:43.156832    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-a0c7a371-gw-openshift-default-7857fb8585-vnnng" event={"ID":"41125f0c-f8ce-416b-92b0-4d6535255a5e","Type":"ContainerStarted","Data":"bc34b6c8524cf0689b48aa4496d44d0767fffcfb11969d052644855c5fc2d4a8"}
Oct 31 01:14:43.325379 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:14:43.325296    4964 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-migrate-a0c7a371-gw-openshift-default-7857fb8585-vnnng" podStartSLOduration=2.32527918 podStartE2EDuration="2.32527918s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:14:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:14:43.323429407 +0000 UTC m=+651.799995797" watchObservedRunningTime="2026-10-31 01:14:43.32527918 +0000 UTC m=+651.801845559"
Oct 31 01:14:43.761110 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:14:43.761082    4964 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-a0c7a371-gw-openshift-default-7857fb8585-vnnng"
Oct 31 01:14:43.765806 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:14:43.765782    4964 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-a0c7a371-gw-openshift-default-7857fb8585-vnnng"
Oct 31 01:14:44.162086 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:14:44.162055    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-a0c7a371-gw-openshift-default-7857fb8585-vnnng"
Oct 31 01:14:44.163245 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:14:44.163228    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-a0c7a371-gw-openshift-default-7857fb8585-vnnng"
Oct 31 01:14:44.443755 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:14:44.443659    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:14:44.906116 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:14:44.906080    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:14:45.117747 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:14:45.117705    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:14:45.316591 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:14:45.316557    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:14:45.941160 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:14:45.941121    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:14:46.130620 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:14:46.130584    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:14:46.972012 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:14:46.971971    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:14:47.161277 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:14:47.161243    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:14:47.371627 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:14:47.371539    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:14:47.775577 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:14:47.775551    4964 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-b9656f65-openshift-default-c6f497c7-r4pr4"]
Oct 31 01:14:47.779104 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:14:47.779087    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-b9656f65-openshift-default-c6f497c7-r4pr4"
Oct 31 01:14:47.834281 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:14:47.834238    4964 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-b9656f65-openshift-default-dockercfg-pg6vj\""
Oct 31 01:14:47.880190 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:14:47.880146    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b23eb8c-e963-42d6-a1f2-61e474aaa829-workload-socket\") pod \"e2e-derive-b9656f65-openshift-default-c6f497c7-r4pr4\" (UID: \"9b23eb8c-e963-42d6-a1f2-61e474aaa829\") " pod="openshift-ingress/e2e-derive-b9656f65-openshift-default-c6f497c7-r4pr4"
Oct 31 01:14:47.880419 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:14:47.880271    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9b23eb8c-e963-42d6-a1f2-61e474aaa829-istio-data\") pod \"e2e-derive-b9656f65-openshift-default-c6f497c7-r4pr4\" (UID: \"9b23eb8c-e963-42d6-a1f2-61e474aaa829\") " pod="openshift-ingress/e2e-derive-b9656f65-openshift-default-c6f497c7-r4pr4"
Oct 31 01:14:47.880419 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:14:47.880311    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9b23eb8c-e963-42d6-a1f2-61e474aaa829-istio-token\") pod \"e2e-derive-b9656f65-openshift-default-c6f497c7-r4pr4\" (UID: \"9b23eb8c-e963-42d6-a1f2-61e474aaa829\") " pod="openshift-ingress/e2e-derive-b9656f65-openshift-default-c6f497c7-r4pr4"
Oct 31 01:14:47.880419 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:14:47.880408    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9b23eb8c-e963-42d6-a1f2-61e474aaa829-istio-envoy\") pod \"e2e-derive-b9656f65-openshift-default-c6f497c7-r4pr4\" (UID: \"9b23eb8c-e963-42d6-a1f2-61e474aaa829\") " pod="openshift-ingress/e2e-derive-b9656f65-openshift-default-c6f497c7-r4pr4"
Oct 31 01:14:47.880559 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:14:47.880433    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rqb5p\" (UniqueName: \"kubernetes.io/projected/9b23eb8c-e963-42d6-a1f2-61e474aaa829-kube-api-access-rqb5p\") pod \"e2e-derive-b9656f65-openshift-default-c6f497c7-r4pr4\" (UID: \"9b23eb8c-e963-42d6-a1f2-61e474aaa829\") " pod="openshift-ingress/e2e-derive-b9656f65-openshift-default-c6f497c7-r4pr4"
Oct 31 01:14:47.880559 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:14:47.880474    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9b23eb8c-e963-42d6-a1f2-61e474aaa829-istiod-ca-cert\") pod \"e2e-derive-b9656f65-openshift-default-c6f497c7-r4pr4\" (UID: \"9b23eb8c-e963-42d6-a1f2-61e474aaa829\") " pod="openshift-ingress/e2e-derive-b9656f65-openshift-default-c6f497c7-r4pr4"
Oct 31 01:14:47.880559 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:14:47.880533    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9b23eb8c-e963-42d6-a1f2-61e474aaa829-workload-certs\") pod \"e2e-derive-b9656f65-openshift-default-c6f497c7-r4pr4\" (UID: \"9b23eb8c-e963-42d6-a1f2-61e474aaa829\") " pod="openshift-ingress/e2e-derive-b9656f65-openshift-default-c6f497c7-r4pr4"
Oct 31 01:14:47.880676 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:14:47.880659    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b23eb8c-e963-42d6-a1f2-61e474aaa829-credential-socket\") pod \"e2e-derive-b9656f65-openshift-default-c6f497c7-r4pr4\" (UID: \"9b23eb8c-e963-42d6-a1f2-61e474aaa829\") " pod="openshift-ingress/e2e-derive-b9656f65-openshift-default-c6f497c7-r4pr4"
Oct 31 01:14:47.880902 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:14:47.880870    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9b23eb8c-e963-42d6-a1f2-61e474aaa829-istio-podinfo\") pod \"e2e-derive-b9656f65-openshift-default-c6f497c7-r4pr4\" (UID: \"9b23eb8c-e963-42d6-a1f2-61e474aaa829\") " pod="openshift-ingress/e2e-derive-b9656f65-openshift-default-c6f497c7-r4pr4"
Oct 31 01:14:47.887519 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:14:47.887494    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-b9656f65-openshift-default-c6f497c7-r4pr4"]
Oct 31 01:14:47.961415 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:14:47.961368    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:14:47.982492 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:14:47.982457    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9b23eb8c-e963-42d6-a1f2-61e474aaa829-istio-podinfo\") pod \"e2e-derive-b9656f65-openshift-default-c6f497c7-r4pr4\" (UID: \"9b23eb8c-e963-42d6-a1f2-61e474aaa829\") " pod="openshift-ingress/e2e-derive-b9656f65-openshift-default-c6f497c7-r4pr4"
Oct 31 01:14:47.982968 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:14:47.982502    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b23eb8c-e963-42d6-a1f2-61e474aaa829-workload-socket\") pod \"e2e-derive-b9656f65-openshift-default-c6f497c7-r4pr4\" (UID: \"9b23eb8c-e963-42d6-a1f2-61e474aaa829\") " pod="openshift-ingress/e2e-derive-b9656f65-openshift-default-c6f497c7-r4pr4"
Oct 31 01:14:47.982968 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:14:47.982530    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9b23eb8c-e963-42d6-a1f2-61e474aaa829-istio-data\") pod \"e2e-derive-b9656f65-openshift-default-c6f497c7-r4pr4\" (UID: \"9b23eb8c-e963-42d6-a1f2-61e474aaa829\") " pod="openshift-ingress/e2e-derive-b9656f65-openshift-default-c6f497c7-r4pr4"
Oct 31 01:14:47.982968 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:14:47.982552    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9b23eb8c-e963-42d6-a1f2-61e474aaa829-istio-token\") pod \"e2e-derive-b9656f65-openshift-default-c6f497c7-r4pr4\" (UID: \"9b23eb8c-e963-42d6-a1f2-61e474aaa829\") " pod="openshift-ingress/e2e-derive-b9656f65-openshift-default-c6f497c7-r4pr4"
Oct 31 01:14:47.982968 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:14:47.982571    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9b23eb8c-e963-42d6-a1f2-61e474aaa829-istio-envoy\") pod \"e2e-derive-b9656f65-openshift-default-c6f497c7-r4pr4\" (UID: \"9b23eb8c-e963-42d6-a1f2-61e474aaa829\") " pod="openshift-ingress/e2e-derive-b9656f65-openshift-default-c6f497c7-r4pr4"
Oct 31 01:14:47.982968 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:14:47.982586    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rqb5p\" (UniqueName: \"kubernetes.io/projected/9b23eb8c-e963-42d6-a1f2-61e474aaa829-kube-api-access-rqb5p\") pod \"e2e-derive-b9656f65-openshift-default-c6f497c7-r4pr4\" (UID: \"9b23eb8c-e963-42d6-a1f2-61e474aaa829\") " pod="openshift-ingress/e2e-derive-b9656f65-openshift-default-c6f497c7-r4pr4"
Oct 31 01:14:47.982968 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:14:47.982779    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9b23eb8c-e963-42d6-a1f2-61e474aaa829-istiod-ca-cert\") pod \"e2e-derive-b9656f65-openshift-default-c6f497c7-r4pr4\" (UID: \"9b23eb8c-e963-42d6-a1f2-61e474aaa829\") " pod="openshift-ingress/e2e-derive-b9656f65-openshift-default-c6f497c7-r4pr4"
Oct 31 01:14:47.983231 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:14:47.983042    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9b23eb8c-e963-42d6-a1f2-61e474aaa829-istio-data\") pod \"e2e-derive-b9656f65-openshift-default-c6f497c7-r4pr4\" (UID: \"9b23eb8c-e963-42d6-a1f2-61e474aaa829\") " pod="openshift-ingress/e2e-derive-b9656f65-openshift-default-c6f497c7-r4pr4"
Oct 31 01:14:47.983231 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:14:47.983060    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9b23eb8c-e963-42d6-a1f2-61e474aaa829-workload-certs\") pod \"e2e-derive-b9656f65-openshift-default-c6f497c7-r4pr4\" (UID: \"9b23eb8c-e963-42d6-a1f2-61e474aaa829\") " pod="openshift-ingress/e2e-derive-b9656f65-openshift-default-c6f497c7-r4pr4"
Oct 31 01:14:47.983231 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:14:47.983091    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b23eb8c-e963-42d6-a1f2-61e474aaa829-workload-socket\") pod \"e2e-derive-b9656f65-openshift-default-c6f497c7-r4pr4\" (UID: \"9b23eb8c-e963-42d6-a1f2-61e474aaa829\") " pod="openshift-ingress/e2e-derive-b9656f65-openshift-default-c6f497c7-r4pr4"
Oct 31 01:14:47.983231 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:14:47.983166    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b23eb8c-e963-42d6-a1f2-61e474aaa829-credential-socket\") pod \"e2e-derive-b9656f65-openshift-default-c6f497c7-r4pr4\" (UID: \"9b23eb8c-e963-42d6-a1f2-61e474aaa829\") " pod="openshift-ingress/e2e-derive-b9656f65-openshift-default-c6f497c7-r4pr4"
Oct 31 01:14:47.983482 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:14:47.983390    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9b23eb8c-e963-42d6-a1f2-61e474aaa829-workload-certs\") pod \"e2e-derive-b9656f65-openshift-default-c6f497c7-r4pr4\" (UID: \"9b23eb8c-e963-42d6-a1f2-61e474aaa829\") " pod="openshift-ingress/e2e-derive-b9656f65-openshift-default-c6f497c7-r4pr4"
Oct 31 01:14:47.983578 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:14:47.983557    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b23eb8c-e963-42d6-a1f2-61e474aaa829-credential-socket\") pod \"e2e-derive-b9656f65-openshift-default-c6f497c7-r4pr4\" (UID: \"9b23eb8c-e963-42d6-a1f2-61e474aaa829\") " pod="openshift-ingress/e2e-derive-b9656f65-openshift-default-c6f497c7-r4pr4"
Oct 31 01:14:47.983770 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:14:47.983746    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9b23eb8c-e963-42d6-a1f2-61e474aaa829-istiod-ca-cert\") pod \"e2e-derive-b9656f65-openshift-default-c6f497c7-r4pr4\" (UID: \"9b23eb8c-e963-42d6-a1f2-61e474aaa829\") " pod="openshift-ingress/e2e-derive-b9656f65-openshift-default-c6f497c7-r4pr4"
Oct 31 01:14:47.985011 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:14:47.984982    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9b23eb8c-e963-42d6-a1f2-61e474aaa829-istio-envoy\") pod \"e2e-derive-b9656f65-openshift-default-c6f497c7-r4pr4\" (UID: \"9b23eb8c-e963-42d6-a1f2-61e474aaa829\") " pod="openshift-ingress/e2e-derive-b9656f65-openshift-default-c6f497c7-r4pr4"
Oct 31 01:14:47.985405 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:14:47.985385    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9b23eb8c-e963-42d6-a1f2-61e474aaa829-istio-podinfo\") pod \"e2e-derive-b9656f65-openshift-default-c6f497c7-r4pr4\" (UID: \"9b23eb8c-e963-42d6-a1f2-61e474aaa829\") " pod="openshift-ingress/e2e-derive-b9656f65-openshift-default-c6f497c7-r4pr4"
Oct 31 01:14:48.040547 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:14:48.040462    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9b23eb8c-e963-42d6-a1f2-61e474aaa829-istio-token\") pod \"e2e-derive-b9656f65-openshift-default-c6f497c7-r4pr4\" (UID: \"9b23eb8c-e963-42d6-a1f2-61e474aaa829\") " pod="openshift-ingress/e2e-derive-b9656f65-openshift-default-c6f497c7-r4pr4"
Oct 31 01:14:48.040735 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:14:48.040641    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rqb5p\" (UniqueName: \"kubernetes.io/projected/9b23eb8c-e963-42d6-a1f2-61e474aaa829-kube-api-access-rqb5p\") pod \"e2e-derive-b9656f65-openshift-default-c6f497c7-r4pr4\" (UID: \"9b23eb8c-e963-42d6-a1f2-61e474aaa829\") " pod="openshift-ingress/e2e-derive-b9656f65-openshift-default-c6f497c7-r4pr4"
Oct 31 01:14:48.092342 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:14:48.092289    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-b9656f65-openshift-default-c6f497c7-r4pr4"
Oct 31 01:14:48.200865 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:14:48.200828    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:14:48.405175 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:14:48.405078    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:14:48.528418 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:14:48.528393    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-b9656f65-openshift-default-c6f497c7-r4pr4"]
Oct 31 01:14:48.531862 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:14:48.531829    4964 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b23eb8c_e963_42d6_a1f2_61e474aaa829.slice/crio-43c627ee2d2b3d3a77b8821b66f03804fc3505ad854f3ddd82f56654ca7fa68f WatchSource:0}: Error finding container 43c627ee2d2b3d3a77b8821b66f03804fc3505ad854f3ddd82f56654ca7fa68f: Status 404 returned error can't find the container with id 43c627ee2d2b3d3a77b8821b66f03804fc3505ad854f3ddd82f56654ca7fa68f
Oct 31 01:14:48.534185 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:14:48.534151    4964 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 01:14:48.534288 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:14:48.534232    4964 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 01:14:48.534288 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:14:48.534278    4964 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 01:14:48.591546 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:14:48.591516    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:14:48.923286 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:14:48.923252    4964 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-a0c7a371-7d55468d95-tjb8l"]
Oct 31 01:14:48.927114 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:14:48.927089    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-a0c7a371-7d55468d95-tjb8l"
Oct 31 01:14:48.981938 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:14:48.981891    4964 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-a0c7a371\""
Oct 31 01:14:48.982225 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:14:48.982001    4964 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-a0c7a371-dockercfg-vz4zp\""
Oct 31 01:14:48.985309 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:14:48.985242    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-a0c7a371-7d55468d95-tjb8l"]
Oct 31 01:14:48.994789 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:14:48.994755    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b537e3cd-aa63-4840-8ec5-4447d616d56f-plugins-config-volume\") pod \"payload-processing-e2e-migrate-a0c7a371-7d55468d95-tjb8l\" (UID: \"b537e3cd-aa63-4840-8ec5-4447d616d56f\") " pod="openshift-ingress/payload-processing-e2e-migrate-a0c7a371-7d55468d95-tjb8l"
Oct 31 01:14:48.994931 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:14:48.994899    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zgfnm\" (UniqueName: \"kubernetes.io/projected/b537e3cd-aa63-4840-8ec5-4447d616d56f-kube-api-access-zgfnm\") pod \"payload-processing-e2e-migrate-a0c7a371-7d55468d95-tjb8l\" (UID: \"b537e3cd-aa63-4840-8ec5-4447d616d56f\") " pod="openshift-ingress/payload-processing-e2e-migrate-a0c7a371-7d55468d95-tjb8l"
Oct 31 01:14:49.096490 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:14:49.096454    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zgfnm\" (UniqueName: \"kubernetes.io/projected/b537e3cd-aa63-4840-8ec5-4447d616d56f-kube-api-access-zgfnm\") pod \"payload-processing-e2e-migrate-a0c7a371-7d55468d95-tjb8l\" (UID: \"b537e3cd-aa63-4840-8ec5-4447d616d56f\") " pod="openshift-ingress/payload-processing-e2e-migrate-a0c7a371-7d55468d95-tjb8l"
Oct 31 01:14:49.096670 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:14:49.096552    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b537e3cd-aa63-4840-8ec5-4447d616d56f-plugins-config-volume\") pod \"payload-processing-e2e-migrate-a0c7a371-7d55468d95-tjb8l\" (UID: \"b537e3cd-aa63-4840-8ec5-4447d616d56f\") " pod="openshift-ingress/payload-processing-e2e-migrate-a0c7a371-7d55468d95-tjb8l"
Oct 31 01:14:49.097255 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:14:49.097237    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b537e3cd-aa63-4840-8ec5-4447d616d56f-plugins-config-volume\") pod \"payload-processing-e2e-migrate-a0c7a371-7d55468d95-tjb8l\" (UID: \"b537e3cd-aa63-4840-8ec5-4447d616d56f\") " pod="openshift-ingress/payload-processing-e2e-migrate-a0c7a371-7d55468d95-tjb8l"
Oct 31 01:14:49.143151 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:14:49.143111    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:14:49.153949 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:14:49.153910    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zgfnm\" (UniqueName: \"kubernetes.io/projected/b537e3cd-aa63-4840-8ec5-4447d616d56f-kube-api-access-zgfnm\") pod \"payload-processing-e2e-migrate-a0c7a371-7d55468d95-tjb8l\" (UID: \"b537e3cd-aa63-4840-8ec5-4447d616d56f\") " pod="openshift-ingress/payload-processing-e2e-migrate-a0c7a371-7d55468d95-tjb8l"
Oct 31 01:14:49.193065 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:14:49.192974    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-b9656f65-openshift-default-c6f497c7-r4pr4" event={"ID":"9b23eb8c-e963-42d6-a1f2-61e474aaa829","Type":"ContainerStarted","Data":"c33cd5188f07608e3088a7c08784ba8e02b959608c559196365e86d4e5589a7d"}
Oct 31 01:14:49.193065 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:14:49.193037    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-b9656f65-openshift-default-c6f497c7-r4pr4" event={"ID":"9b23eb8c-e963-42d6-a1f2-61e474aaa829","Type":"ContainerStarted","Data":"43c627ee2d2b3d3a77b8821b66f03804fc3505ad854f3ddd82f56654ca7fa68f"}
Oct 31 01:14:49.242653 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:14:49.242615    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-a0c7a371-7d55468d95-tjb8l"
Oct 31 01:14:49.315481 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:14:49.315427    4964 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-b9656f65-openshift-default-c6f497c7-r4pr4" podStartSLOduration=2.315396473 podStartE2EDuration="2.315396473s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:14:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:14:49.314496052 +0000 UTC m=+657.791062433" watchObservedRunningTime="2026-10-31 01:14:49.315396473 +0000 UTC m=+657.791962907"
Oct 31 01:14:49.407893 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:14:49.407859    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:14:49.622671 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:14:49.622640    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-a0c7a371-7d55468d95-tjb8l"]
Oct 31 01:14:49.624522 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:14:49.624493    4964 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb537e3cd_aa63_4840_8ec5_4447d616d56f.slice/crio-8576da08365ae53cda5435a08d8bd3d03ba0fd95ca9908030bdc777a4801467f WatchSource:0}: Error finding container 8576da08365ae53cda5435a08d8bd3d03ba0fd95ca9908030bdc777a4801467f: Status 404 returned error can't find the container with id 8576da08365ae53cda5435a08d8bd3d03ba0fd95ca9908030bdc777a4801467f
Oct 31 01:14:50.093229 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:14:50.093196    4964 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-b9656f65-openshift-default-c6f497c7-r4pr4"
Oct 31 01:14:50.098839 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:14:50.098809    4964 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-b9656f65-openshift-default-c6f497c7-r4pr4"
Oct 31 01:14:50.200103 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:14:50.200068    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-a0c7a371-7d55468d95-tjb8l" event={"ID":"b537e3cd-aa63-4840-8ec5-4447d616d56f","Type":"ContainerStarted","Data":"0a6ffaf259248353a39cb4870af766d3ebf0283c77d5e1d0294e9c0d6327c6a6"}
Oct 31 01:14:50.200267 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:14:50.200124    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-a0c7a371-7d55468d95-tjb8l" event={"ID":"b537e3cd-aa63-4840-8ec5-4447d616d56f","Type":"ContainerStarted","Data":"8576da08365ae53cda5435a08d8bd3d03ba0fd95ca9908030bdc777a4801467f"}
Oct 31 01:14:50.200382 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:14:50.200348    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-b9656f65-openshift-default-c6f497c7-r4pr4"
Oct 31 01:14:50.200445 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:14:50.200399    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-migrate-a0c7a371-7d55468d95-tjb8l"
Oct 31 01:14:50.201419 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:14:50.201397    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-b9656f65-openshift-default-c6f497c7-r4pr4"
Oct 31 01:14:50.330534 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:14:50.330479    4964 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-migrate-a0c7a371-7d55468d95-tjb8l" podStartSLOduration=2.330466366 podStartE2EDuration="2.330466366s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:14:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:14:50.327841037 +0000 UTC m=+658.804407428" watchObservedRunningTime="2026-10-31 01:14:50.330466366 +0000 UTC m=+658.807032754"
Oct 31 01:14:51.146194 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:14:51.146162    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:14:51.255185 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:14:51.255147    4964 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-a0c7a371-76657b6bb8-nfjxd"]
Oct 31 01:14:51.259070 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:14:51.259048    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-a0c7a371-76657b6bb8-nfjxd"
Oct 31 01:14:51.310927 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:14:51.310888    4964 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-a0c7a371-serving-cert\""
Oct 31 01:14:51.322798 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:14:51.322757    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b972c\" (UniqueName: \"kubernetes.io/projected/9dc20490-124e-40b4-9502-10949d62d531-kube-api-access-b972c\") pod \"maas-api-e2e-migrate-a0c7a371-76657b6bb8-nfjxd\" (UID: \"9dc20490-124e-40b4-9502-10949d62d531\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-a0c7a371-76657b6bb8-nfjxd"
Oct 31 01:14:51.322991 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:14:51.322820    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9dc20490-124e-40b4-9502-10949d62d531-openshift-service-ca\") pod \"maas-api-e2e-migrate-a0c7a371-76657b6bb8-nfjxd\" (UID: \"9dc20490-124e-40b4-9502-10949d62d531\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-a0c7a371-76657b6bb8-nfjxd"
Oct 31 01:14:51.322991 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:14:51.322846    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9dc20490-124e-40b4-9502-10949d62d531-metrics-certs\") pod \"maas-api-e2e-migrate-a0c7a371-76657b6bb8-nfjxd\" (UID: \"9dc20490-124e-40b4-9502-10949d62d531\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-a0c7a371-76657b6bb8-nfjxd"
Oct 31 01:14:51.322991 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:14:51.322893    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9dc20490-124e-40b4-9502-10949d62d531-maas-api-tls\") pod \"maas-api-e2e-migrate-a0c7a371-76657b6bb8-nfjxd\" (UID: \"9dc20490-124e-40b4-9502-10949d62d531\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-a0c7a371-76657b6bb8-nfjxd"
Oct 31 01:14:51.344938 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:14:51.344910    4964 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 01:14:51.360281 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:14:51.360244    4964 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Oct 31 01:14:51.360281 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:14:51.360250    4964 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 01:14:51.360533 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:14:51.360412    4964 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-x26rq\""
Oct 31 01:14:51.362729 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:14:51.362708    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-a0c7a371-76657b6bb8-nfjxd"]
Oct 31 01:14:51.400561 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:14:51.400468    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:14:51.424364 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:14:51.424331    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b972c\" (UniqueName: \"kubernetes.io/projected/9dc20490-124e-40b4-9502-10949d62d531-kube-api-access-b972c\") pod \"maas-api-e2e-migrate-a0c7a371-76657b6bb8-nfjxd\" (UID: \"9dc20490-124e-40b4-9502-10949d62d531\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-a0c7a371-76657b6bb8-nfjxd"
Oct 31 01:14:51.424562 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:14:51.424381    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9dc20490-124e-40b4-9502-10949d62d531-openshift-service-ca\") pod \"maas-api-e2e-migrate-a0c7a371-76657b6bb8-nfjxd\" (UID: \"9dc20490-124e-40b4-9502-10949d62d531\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-a0c7a371-76657b6bb8-nfjxd"
Oct 31 01:14:51.424562 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:14:51.424401    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9dc20490-124e-40b4-9502-10949d62d531-metrics-certs\") pod \"maas-api-e2e-migrate-a0c7a371-76657b6bb8-nfjxd\" (UID: \"9dc20490-124e-40b4-9502-10949d62d531\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-a0c7a371-76657b6bb8-nfjxd"
Oct 31 01:14:51.424562 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:14:51.424435    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9dc20490-124e-40b4-9502-10949d62d531-maas-api-tls\") pod \"maas-api-e2e-migrate-a0c7a371-76657b6bb8-nfjxd\" (UID: \"9dc20490-124e-40b4-9502-10949d62d531\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-a0c7a371-76657b6bb8-nfjxd"
Oct 31 01:14:51.424742 ip-10-0-133-197 kubenswrapper[4964]: E1031 01:14:51.424629    4964 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-migrate-a0c7a371-serving-cert: secret "maas-api-e2e-migrate-a0c7a371-serving-cert" not found
Oct 31 01:14:51.424742 ip-10-0-133-197 kubenswrapper[4964]: E1031 01:14:51.424716    4964 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9dc20490-124e-40b4-9502-10949d62d531-maas-api-tls podName:9dc20490-124e-40b4-9502-10949d62d531 nodeName:}" failed. No retries permitted until 2026-10-31 01:14:51.924694241 +0000 UTC m=+660.401260605 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/9dc20490-124e-40b4-9502-10949d62d531-maas-api-tls") pod "maas-api-e2e-migrate-a0c7a371-76657b6bb8-nfjxd" (UID: "9dc20490-124e-40b4-9502-10949d62d531") : secret "maas-api-e2e-migrate-a0c7a371-serving-cert" not found
Oct 31 01:14:51.425147 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:14:51.425120    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9dc20490-124e-40b4-9502-10949d62d531-openshift-service-ca\") pod \"maas-api-e2e-migrate-a0c7a371-76657b6bb8-nfjxd\" (UID: \"9dc20490-124e-40b4-9502-10949d62d531\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-a0c7a371-76657b6bb8-nfjxd"
Oct 31 01:14:51.427143 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:14:51.427120    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9dc20490-124e-40b4-9502-10949d62d531-metrics-certs\") pod \"maas-api-e2e-migrate-a0c7a371-76657b6bb8-nfjxd\" (UID: \"9dc20490-124e-40b4-9502-10949d62d531\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-a0c7a371-76657b6bb8-nfjxd"
Oct 31 01:14:51.487627 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:14:51.487590    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b972c\" (UniqueName: \"kubernetes.io/projected/9dc20490-124e-40b4-9502-10949d62d531-kube-api-access-b972c\") pod \"maas-api-e2e-migrate-a0c7a371-76657b6bb8-nfjxd\" (UID: \"9dc20490-124e-40b4-9502-10949d62d531\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-a0c7a371-76657b6bb8-nfjxd"
Oct 31 01:14:51.583056 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:14:51.583003    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:14:51.767570 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:14:51.767537    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:14:51.929103 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:14:51.929070    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9dc20490-124e-40b4-9502-10949d62d531-maas-api-tls\") pod \"maas-api-e2e-migrate-a0c7a371-76657b6bb8-nfjxd\" (UID: \"9dc20490-124e-40b4-9502-10949d62d531\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-a0c7a371-76657b6bb8-nfjxd"
Oct 31 01:14:51.931659 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:14:51.931629    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9dc20490-124e-40b4-9502-10949d62d531-maas-api-tls\") pod \"maas-api-e2e-migrate-a0c7a371-76657b6bb8-nfjxd\" (UID: \"9dc20490-124e-40b4-9502-10949d62d531\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-a0c7a371-76657b6bb8-nfjxd"
Oct 31 01:14:51.947391 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:14:51.947365    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:14:52.224215 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:14:52.224188    4964 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-x26rq\""
Oct 31 01:14:52.233386 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:14:52.233365    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-a0c7a371-76657b6bb8-nfjxd"
Oct 31 01:14:52.610059 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:14:52.610035    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-a0c7a371-76657b6bb8-nfjxd"]
Oct 31 01:14:52.612106 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:14:52.612083    4964 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9dc20490_124e_40b4_9502_10949d62d531.slice/crio-e5cc2ae7b0ccb3b2c0af893c39ad7d9b379aa61f5aabb4234a5ea91139b61264 WatchSource:0}: Error finding container e5cc2ae7b0ccb3b2c0af893c39ad7d9b379aa61f5aabb4234a5ea91139b61264: Status 404 returned error can't find the container with id e5cc2ae7b0ccb3b2c0af893c39ad7d9b379aa61f5aabb4234a5ea91139b61264
Oct 31 01:14:52.613607 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:14:52.613585    4964 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 31 01:14:53.165776 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:14:53.165743    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:14:53.223846 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:14:53.223775    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-a0c7a371-76657b6bb8-nfjxd" event={"ID":"9dc20490-124e-40b4-9502-10949d62d531","Type":"ContainerStarted","Data":"e5cc2ae7b0ccb3b2c0af893c39ad7d9b379aa61f5aabb4234a5ea91139b61264"}
Oct 31 01:14:53.343156 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:14:53.343113    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:14:54.097599 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:14:54.097547    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:14:54.232960 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:14:54.232918    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-a0c7a371-76657b6bb8-nfjxd" event={"ID":"9dc20490-124e-40b4-9502-10949d62d531","Type":"ContainerStarted","Data":"2be436bec730505e5a0f715ee14233e7871912d291c7276e13559a82582fb81f"}
Oct 31 01:14:54.233160 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:14:54.233100    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-a0c7a371-76657b6bb8-nfjxd"
Oct 31 01:14:54.320105 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:14:54.320052    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:14:54.360276 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:14:54.360179    4964 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-a0c7a371-76657b6bb8-nfjxd" podStartSLOduration=2.427592861 podStartE2EDuration="3.360164731s" totalImagesPullingTime="932.57187ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:14:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:14:54.357832872 +0000 UTC m=+662.834399252" watchObservedRunningTime="2026-10-31 01:14:54.360164731 +0000 UTC m=+662.836731111"
Oct 31 01:14:55.622352 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:14:55.622292    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:14:55.804498 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:14:55.804467    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:14:55.995331 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:14:55.995291    4964 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-7164dd00-6dcd9d9bc9-24xtq"]
Oct 31 01:14:55.999192 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:14:55.999168    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-7164dd00-6dcd9d9bc9-24xtq"
Oct 31 01:14:56.057335 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:14:56.057288    4964 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-7164dd00\""
Oct 31 01:14:56.069703 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:14:56.069673    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e044e488-b8db-46d9-afe1-b2a39b9a10c1-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-7164dd00-6dcd9d9bc9-24xtq\" (UID: \"e044e488-b8db-46d9-afe1-b2a39b9a10c1\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-7164dd00-6dcd9d9bc9-24xtq"
Oct 31 01:14:56.069703 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:14:56.069707    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l4pcc\" (UniqueName: \"kubernetes.io/projected/e044e488-b8db-46d9-afe1-b2a39b9a10c1-kube-api-access-l4pcc\") pod \"payload-pre-processing-e2e-ait-7164dd00-6dcd9d9bc9-24xtq\" (UID: \"e044e488-b8db-46d9-afe1-b2a39b9a10c1\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-7164dd00-6dcd9d9bc9-24xtq"
Oct 31 01:14:56.079860 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:14:56.079828    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-7164dd00-6dcd9d9bc9-24xtq"]
Oct 31 01:14:56.118517 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:14:56.118487    4964 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-7164dd00-dockercfg-4tgdb\""
Oct 31 01:14:56.170569 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:14:56.170533    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e044e488-b8db-46d9-afe1-b2a39b9a10c1-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-7164dd00-6dcd9d9bc9-24xtq\" (UID: \"e044e488-b8db-46d9-afe1-b2a39b9a10c1\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-7164dd00-6dcd9d9bc9-24xtq"
Oct 31 01:14:56.170744 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:14:56.170578    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l4pcc\" (UniqueName: \"kubernetes.io/projected/e044e488-b8db-46d9-afe1-b2a39b9a10c1-kube-api-access-l4pcc\") pod \"payload-pre-processing-e2e-ait-7164dd00-6dcd9d9bc9-24xtq\" (UID: \"e044e488-b8db-46d9-afe1-b2a39b9a10c1\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-7164dd00-6dcd9d9bc9-24xtq"
Oct 31 01:14:56.171311 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:14:56.171292    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e044e488-b8db-46d9-afe1-b2a39b9a10c1-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-7164dd00-6dcd9d9bc9-24xtq\" (UID: \"e044e488-b8db-46d9-afe1-b2a39b9a10c1\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-7164dd00-6dcd9d9bc9-24xtq"
Oct 31 01:14:56.217547 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:14:56.217515    4964 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-f94871a1-6f988c57d9-t52mh"]
Oct 31 01:14:56.221270 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:14:56.221253    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f94871a1-6f988c57d9-t52mh"
Oct 31 01:14:56.234505 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:14:56.234471    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l4pcc\" (UniqueName: \"kubernetes.io/projected/e044e488-b8db-46d9-afe1-b2a39b9a10c1-kube-api-access-l4pcc\") pod \"payload-pre-processing-e2e-ait-7164dd00-6dcd9d9bc9-24xtq\" (UID: \"e044e488-b8db-46d9-afe1-b2a39b9a10c1\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-7164dd00-6dcd9d9bc9-24xtq"
Oct 31 01:14:56.271523 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:14:56.271433    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/101c024d-b61f-47e5-a863-330ef7b7069d-metrics-certs\") pod \"maas-api-e2e-ait-f94871a1-6f988c57d9-t52mh\" (UID: \"101c024d-b61f-47e5-a863-330ef7b7069d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f94871a1-6f988c57d9-t52mh"
Oct 31 01:14:56.271678 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:14:56.271594    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8xmx4\" (UniqueName: \"kubernetes.io/projected/101c024d-b61f-47e5-a863-330ef7b7069d-kube-api-access-8xmx4\") pod \"maas-api-e2e-ait-f94871a1-6f988c57d9-t52mh\" (UID: \"101c024d-b61f-47e5-a863-330ef7b7069d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f94871a1-6f988c57d9-t52mh"
Oct 31 01:14:56.271678 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:14:56.271642    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/101c024d-b61f-47e5-a863-330ef7b7069d-maas-api-tls\") pod \"maas-api-e2e-ait-f94871a1-6f988c57d9-t52mh\" (UID: \"101c024d-b61f-47e5-a863-330ef7b7069d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f94871a1-6f988c57d9-t52mh"
Oct 31 01:14:56.271678 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:14:56.271660    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/101c024d-b61f-47e5-a863-330ef7b7069d-openshift-service-ca\") pod \"maas-api-e2e-ait-f94871a1-6f988c57d9-t52mh\" (UID: \"101c024d-b61f-47e5-a863-330ef7b7069d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f94871a1-6f988c57d9-t52mh"
Oct 31 01:14:56.284750 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:14:56.284722    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-f94871a1-6f988c57d9-t52mh"]
Oct 31 01:14:56.312591 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:14:56.312558    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-7164dd00-6dcd9d9bc9-24xtq"
Oct 31 01:14:56.323820 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:14:56.323788    4964 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-f94871a1-serving-cert\""
Oct 31 01:14:56.373761 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:14:56.373549    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8xmx4\" (UniqueName: \"kubernetes.io/projected/101c024d-b61f-47e5-a863-330ef7b7069d-kube-api-access-8xmx4\") pod \"maas-api-e2e-ait-f94871a1-6f988c57d9-t52mh\" (UID: \"101c024d-b61f-47e5-a863-330ef7b7069d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f94871a1-6f988c57d9-t52mh"
Oct 31 01:14:56.374163 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:14:56.374139    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/101c024d-b61f-47e5-a863-330ef7b7069d-maas-api-tls\") pod \"maas-api-e2e-ait-f94871a1-6f988c57d9-t52mh\" (UID: \"101c024d-b61f-47e5-a863-330ef7b7069d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f94871a1-6f988c57d9-t52mh"
Oct 31 01:14:56.374277 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:14:56.374185    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/101c024d-b61f-47e5-a863-330ef7b7069d-openshift-service-ca\") pod \"maas-api-e2e-ait-f94871a1-6f988c57d9-t52mh\" (UID: \"101c024d-b61f-47e5-a863-330ef7b7069d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f94871a1-6f988c57d9-t52mh"
Oct 31 01:14:56.374376 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:14:56.374288    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/101c024d-b61f-47e5-a863-330ef7b7069d-metrics-certs\") pod \"maas-api-e2e-ait-f94871a1-6f988c57d9-t52mh\" (UID: \"101c024d-b61f-47e5-a863-330ef7b7069d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f94871a1-6f988c57d9-t52mh"
Oct 31 01:14:56.374660 ip-10-0-133-197 kubenswrapper[4964]: E1031 01:14:56.374647    4964 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-f94871a1-serving-cert: secret "maas-api-e2e-ait-f94871a1-serving-cert" not found
Oct 31 01:14:56.374806 ip-10-0-133-197 kubenswrapper[4964]: E1031 01:14:56.374793    4964 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/101c024d-b61f-47e5-a863-330ef7b7069d-maas-api-tls podName:101c024d-b61f-47e5-a863-330ef7b7069d nodeName:}" failed. No retries permitted until 2026-10-31 01:14:56.874758466 +0000 UTC m=+665.351324838 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/101c024d-b61f-47e5-a863-330ef7b7069d-maas-api-tls") pod "maas-api-e2e-ait-f94871a1-6f988c57d9-t52mh" (UID: "101c024d-b61f-47e5-a863-330ef7b7069d") : secret "maas-api-e2e-ait-f94871a1-serving-cert" not found
Oct 31 01:14:56.375876 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:14:56.375850    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/101c024d-b61f-47e5-a863-330ef7b7069d-openshift-service-ca\") pod \"maas-api-e2e-ait-f94871a1-6f988c57d9-t52mh\" (UID: \"101c024d-b61f-47e5-a863-330ef7b7069d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f94871a1-6f988c57d9-t52mh"
Oct 31 01:14:56.377667 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:14:56.377645    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/101c024d-b61f-47e5-a863-330ef7b7069d-metrics-certs\") pod \"maas-api-e2e-ait-f94871a1-6f988c57d9-t52mh\" (UID: \"101c024d-b61f-47e5-a863-330ef7b7069d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f94871a1-6f988c57d9-t52mh"
Oct 31 01:14:56.423547 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:14:56.423501    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:14:56.431882 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:14:56.431854    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8xmx4\" (UniqueName: \"kubernetes.io/projected/101c024d-b61f-47e5-a863-330ef7b7069d-kube-api-access-8xmx4\") pod \"maas-api-e2e-ait-f94871a1-6f988c57d9-t52mh\" (UID: \"101c024d-b61f-47e5-a863-330ef7b7069d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f94871a1-6f988c57d9-t52mh"
Oct 31 01:14:56.608545 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:14:56.608458    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:14:56.705272 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:14:56.704868    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-7164dd00-6dcd9d9bc9-24xtq"]
Oct 31 01:14:56.707377 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:14:56.707346    4964 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode044e488_b8db_46d9_afe1_b2a39b9a10c1.slice/crio-393e6bb4cc1397d88fa5e449d6c331ebd839eca2c7d730c317bd4b31dbd5035d WatchSource:0}: Error finding container 393e6bb4cc1397d88fa5e449d6c331ebd839eca2c7d730c317bd4b31dbd5035d: Status 404 returned error can't find the container with id 393e6bb4cc1397d88fa5e449d6c331ebd839eca2c7d730c317bd4b31dbd5035d
Oct 31 01:14:56.804907 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:14:56.804872    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:14:56.880937 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:14:56.880852    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/101c024d-b61f-47e5-a863-330ef7b7069d-maas-api-tls\") pod \"maas-api-e2e-ait-f94871a1-6f988c57d9-t52mh\" (UID: \"101c024d-b61f-47e5-a863-330ef7b7069d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f94871a1-6f988c57d9-t52mh"
Oct 31 01:14:56.883481 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:14:56.883459    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/101c024d-b61f-47e5-a863-330ef7b7069d-maas-api-tls\") pod \"maas-api-e2e-ait-f94871a1-6f988c57d9-t52mh\" (UID: \"101c024d-b61f-47e5-a863-330ef7b7069d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f94871a1-6f988c57d9-t52mh"
Oct 31 01:14:57.135913 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:14:57.135822    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f94871a1-6f988c57d9-t52mh"
Oct 31 01:14:57.253127 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:14:57.253098    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-7164dd00-6dcd9d9bc9-24xtq" event={"ID":"e044e488-b8db-46d9-afe1-b2a39b9a10c1","Type":"ContainerStarted","Data":"c844b0d7595c33cf569c97428c7829957e1df65a3701bea72a5f801822f4535d"}
Oct 31 01:14:57.253333 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:14:57.253154    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-7164dd00-6dcd9d9bc9-24xtq" event={"ID":"e044e488-b8db-46d9-afe1-b2a39b9a10c1","Type":"ContainerStarted","Data":"393e6bb4cc1397d88fa5e449d6c331ebd839eca2c7d730c317bd4b31dbd5035d"}
Oct 31 01:14:57.253333 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:14:57.253230    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-7164dd00-6dcd9d9bc9-24xtq"
Oct 31 01:14:57.369145 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:14:57.369090    4964 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-7164dd00-6dcd9d9bc9-24xtq" podStartSLOduration=2.369074737 podStartE2EDuration="2.369074737s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:14:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:14:57.367198062 +0000 UTC m=+665.843764449" watchObservedRunningTime="2026-10-31 01:14:57.369074737 +0000 UTC m=+665.845641116"
Oct 31 01:14:57.513176 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:14:57.513144    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-f94871a1-6f988c57d9-t52mh"]
Oct 31 01:14:57.515234 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:14:57.515205    4964 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod101c024d_b61f_47e5_a863_330ef7b7069d.slice/crio-0f7683f743c0db2072ad5bc188692cc2159290d668917afb04e69152fef455e2 WatchSource:0}: Error finding container 0f7683f743c0db2072ad5bc188692cc2159290d668917afb04e69152fef455e2: Status 404 returned error can't find the container with id 0f7683f743c0db2072ad5bc188692cc2159290d668917afb04e69152fef455e2
Oct 31 01:14:58.260673 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:14:58.260643    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f94871a1-6f988c57d9-t52mh" event={"ID":"101c024d-b61f-47e5-a863-330ef7b7069d","Type":"ContainerStarted","Data":"0f7683f743c0db2072ad5bc188692cc2159290d668917afb04e69152fef455e2"}
Oct 31 01:14:58.357910 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:14:58.357875    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:14:58.554722 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:14:58.554685    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:14:58.737400 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:14:58.737359    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:14:59.267198 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:14:59.267162    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f94871a1-6f988c57d9-t52mh" event={"ID":"101c024d-b61f-47e5-a863-330ef7b7069d","Type":"ContainerStarted","Data":"81ab776d077584ef5bbafcffb33cbed319d98b968a758b10bd77e3e89203eb75"}
Oct 31 01:14:59.267648 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:14:59.267356    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f94871a1-6f988c57d9-t52mh"
Oct 31 01:14:59.384951 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:14:59.384897    4964 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f94871a1-6f988c57d9-t52mh" podStartSLOduration=2.4637773210000002 podStartE2EDuration="3.384882471s" totalImagesPullingTime="921.10515ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:14:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:14:59.382016839 +0000 UTC m=+667.858583218" watchObservedRunningTime="2026-10-31 01:14:59.384882471 +0000 UTC m=+667.861448880"
Oct 31 01:15:00.245001 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:15:00.244921    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-a0c7a371-76657b6bb8-nfjxd"
Oct 31 01:15:01.208829 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:15:01.208799    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-migrate-a0c7a371-7d55468d95-tjb8l"
Oct 31 01:15:01.713854 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:15:01.713813    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:15:02.870008 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:15:02.869972    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:15:03.094447 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:15:03.094415    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:15:03.482670 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:15:03.482637    4964 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-b9656f65-755b48cd75-lpd6d"]
Oct 31 01:15:03.486390 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:15:03.486368    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-b9656f65-755b48cd75-lpd6d"
Oct 31 01:15:03.538275 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:15:03.538244    4964 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-b9656f65-dockercfg-nssj5\""
Oct 31 01:15:03.551524 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:15:03.551483    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/40d9e5d2-76da-4a1f-9ac9-78ab76bc653e-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-b9656f65-755b48cd75-lpd6d\" (UID: \"40d9e5d2-76da-4a1f-9ac9-78ab76bc653e\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-b9656f65-755b48cd75-lpd6d"
Oct 31 01:15:03.551744 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:15:03.551722    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-99682\" (UniqueName: \"kubernetes.io/projected/40d9e5d2-76da-4a1f-9ac9-78ab76bc653e-kube-api-access-99682\") pod \"payload-pre-processing-e2e-derive-b9656f65-755b48cd75-lpd6d\" (UID: \"40d9e5d2-76da-4a1f-9ac9-78ab76bc653e\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-b9656f65-755b48cd75-lpd6d"
Oct 31 01:15:03.587889 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:15:03.587860    4964 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-b9656f65\""
Oct 31 01:15:03.590722 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:15:03.590697    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-b9656f65-755b48cd75-lpd6d"]
Oct 31 01:15:03.656952 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:15:03.656918    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/40d9e5d2-76da-4a1f-9ac9-78ab76bc653e-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-b9656f65-755b48cd75-lpd6d\" (UID: \"40d9e5d2-76da-4a1f-9ac9-78ab76bc653e\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-b9656f65-755b48cd75-lpd6d"
Oct 31 01:15:03.657141 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:15:03.657052    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-99682\" (UniqueName: \"kubernetes.io/projected/40d9e5d2-76da-4a1f-9ac9-78ab76bc653e-kube-api-access-99682\") pod \"payload-pre-processing-e2e-derive-b9656f65-755b48cd75-lpd6d\" (UID: \"40d9e5d2-76da-4a1f-9ac9-78ab76bc653e\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-b9656f65-755b48cd75-lpd6d"
Oct 31 01:15:03.657779 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:15:03.657755    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/40d9e5d2-76da-4a1f-9ac9-78ab76bc653e-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-b9656f65-755b48cd75-lpd6d\" (UID: \"40d9e5d2-76da-4a1f-9ac9-78ab76bc653e\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-b9656f65-755b48cd75-lpd6d"
Oct 31 01:15:03.721911 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:15:03.721880    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-99682\" (UniqueName: \"kubernetes.io/projected/40d9e5d2-76da-4a1f-9ac9-78ab76bc653e-kube-api-access-99682\") pod \"payload-pre-processing-e2e-derive-b9656f65-755b48cd75-lpd6d\" (UID: \"40d9e5d2-76da-4a1f-9ac9-78ab76bc653e\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-b9656f65-755b48cd75-lpd6d"
Oct 31 01:15:03.810396 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:15:03.805524    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-b9656f65-755b48cd75-lpd6d"
Oct 31 01:15:04.222340 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:15:04.205755    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-b9656f65-755b48cd75-lpd6d"]
Oct 31 01:15:04.325799 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:15:04.325763    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-b9656f65-755b48cd75-lpd6d" event={"ID":"40d9e5d2-76da-4a1f-9ac9-78ab76bc653e","Type":"ContainerStarted","Data":"35795743f28a7a1ab43d93558c2d215ee362cb28997de39da8ed5aa85350baeb"}
Oct 31 01:15:05.279637 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:15:05.279604    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f94871a1-6f988c57d9-t52mh"
Oct 31 01:15:05.333650 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:15:05.333614    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-b9656f65-755b48cd75-lpd6d" event={"ID":"40d9e5d2-76da-4a1f-9ac9-78ab76bc653e","Type":"ContainerStarted","Data":"1983d52ece9992bcd1f2c5819ed17e18d03a3c20f37bcca587cf18a7713069e7"}
Oct 31 01:15:05.333836 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:15:05.333677    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-b9656f65-755b48cd75-lpd6d"
Oct 31 01:15:05.524802 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:15:05.524715    4964 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-derive-b9656f65-755b48cd75-lpd6d" podStartSLOduration=2.524699906 podStartE2EDuration="2.524699906s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:15:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:15:05.524232139 +0000 UTC m=+674.000798519" watchObservedRunningTime="2026-10-31 01:15:05.524699906 +0000 UTC m=+674.001266287"
Oct 31 01:15:06.516341 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:15:06.515601    4964 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-7164dd00-6c969c79f6-2g46h"]
Oct 31 01:15:06.520436 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:15:06.520407    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-7164dd00-6c969c79f6-2g46h"
Oct 31 01:15:06.573777 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:15:06.573750    4964 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-7164dd00-serving-cert\""
Oct 31 01:15:06.589283 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:15:06.589258    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8270a0a8-7261-45b3-849f-6c05a6b17f8e-maas-api-tls\") pod \"maas-api-e2e-ait-7164dd00-6c969c79f6-2g46h\" (UID: \"8270a0a8-7261-45b3-849f-6c05a6b17f8e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-7164dd00-6c969c79f6-2g46h"
Oct 31 01:15:06.589441 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:15:06.589302    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8270a0a8-7261-45b3-849f-6c05a6b17f8e-openshift-service-ca\") pod \"maas-api-e2e-ait-7164dd00-6c969c79f6-2g46h\" (UID: \"8270a0a8-7261-45b3-849f-6c05a6b17f8e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-7164dd00-6c969c79f6-2g46h"
Oct 31 01:15:06.589441 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:15:06.589411    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7p987\" (UniqueName: \"kubernetes.io/projected/8270a0a8-7261-45b3-849f-6c05a6b17f8e-kube-api-access-7p987\") pod \"maas-api-e2e-ait-7164dd00-6c969c79f6-2g46h\" (UID: \"8270a0a8-7261-45b3-849f-6c05a6b17f8e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-7164dd00-6c969c79f6-2g46h"
Oct 31 01:15:06.589518 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:15:06.589491    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8270a0a8-7261-45b3-849f-6c05a6b17f8e-metrics-certs\") pod \"maas-api-e2e-ait-7164dd00-6c969c79f6-2g46h\" (UID: \"8270a0a8-7261-45b3-849f-6c05a6b17f8e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-7164dd00-6c969c79f6-2g46h"
Oct 31 01:15:06.626948 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:15:06.626904    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-7164dd00-6c969c79f6-2g46h"]
Oct 31 01:15:06.719345 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:15:06.709936    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8270a0a8-7261-45b3-849f-6c05a6b17f8e-metrics-certs\") pod \"maas-api-e2e-ait-7164dd00-6c969c79f6-2g46h\" (UID: \"8270a0a8-7261-45b3-849f-6c05a6b17f8e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-7164dd00-6c969c79f6-2g46h"
Oct 31 01:15:06.719345 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:15:06.710095    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8270a0a8-7261-45b3-849f-6c05a6b17f8e-maas-api-tls\") pod \"maas-api-e2e-ait-7164dd00-6c969c79f6-2g46h\" (UID: \"8270a0a8-7261-45b3-849f-6c05a6b17f8e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-7164dd00-6c969c79f6-2g46h"
Oct 31 01:15:06.719345 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:15:06.710148    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8270a0a8-7261-45b3-849f-6c05a6b17f8e-openshift-service-ca\") pod \"maas-api-e2e-ait-7164dd00-6c969c79f6-2g46h\" (UID: \"8270a0a8-7261-45b3-849f-6c05a6b17f8e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-7164dd00-6c969c79f6-2g46h"
Oct 31 01:15:06.719345 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:15:06.710194    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7p987\" (UniqueName: \"kubernetes.io/projected/8270a0a8-7261-45b3-849f-6c05a6b17f8e-kube-api-access-7p987\") pod \"maas-api-e2e-ait-7164dd00-6c969c79f6-2g46h\" (UID: \"8270a0a8-7261-45b3-849f-6c05a6b17f8e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-7164dd00-6c969c79f6-2g46h"
Oct 31 01:15:06.720032 ip-10-0-133-197 kubenswrapper[4964]: E1031 01:15:06.720006    4964 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-7164dd00-serving-cert: secret "maas-api-e2e-ait-7164dd00-serving-cert" not found
Oct 31 01:15:06.720231 ip-10-0-133-197 kubenswrapper[4964]: E1031 01:15:06.720219    4964 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8270a0a8-7261-45b3-849f-6c05a6b17f8e-maas-api-tls podName:8270a0a8-7261-45b3-849f-6c05a6b17f8e nodeName:}" failed. No retries permitted until 2026-10-31 01:15:07.220194956 +0000 UTC m=+675.696761318 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/8270a0a8-7261-45b3-849f-6c05a6b17f8e-maas-api-tls") pod "maas-api-e2e-ait-7164dd00-6c969c79f6-2g46h" (UID: "8270a0a8-7261-45b3-849f-6c05a6b17f8e") : secret "maas-api-e2e-ait-7164dd00-serving-cert" not found
Oct 31 01:15:06.722031 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:15:06.722006    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8270a0a8-7261-45b3-849f-6c05a6b17f8e-openshift-service-ca\") pod \"maas-api-e2e-ait-7164dd00-6c969c79f6-2g46h\" (UID: \"8270a0a8-7261-45b3-849f-6c05a6b17f8e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-7164dd00-6c969c79f6-2g46h"
Oct 31 01:15:06.726932 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:15:06.726896    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8270a0a8-7261-45b3-849f-6c05a6b17f8e-metrics-certs\") pod \"maas-api-e2e-ait-7164dd00-6c969c79f6-2g46h\" (UID: \"8270a0a8-7261-45b3-849f-6c05a6b17f8e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-7164dd00-6c969c79f6-2g46h"
Oct 31 01:15:06.755234 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:15:06.755194    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:15:06.780682 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:15:06.780615    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7p987\" (UniqueName: \"kubernetes.io/projected/8270a0a8-7261-45b3-849f-6c05a6b17f8e-kube-api-access-7p987\") pod \"maas-api-e2e-ait-7164dd00-6c969c79f6-2g46h\" (UID: \"8270a0a8-7261-45b3-849f-6c05a6b17f8e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-7164dd00-6c969c79f6-2g46h"
Oct 31 01:15:06.927135 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:15:06.926620    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:15:07.318158 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:15:07.318123    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8270a0a8-7261-45b3-849f-6c05a6b17f8e-maas-api-tls\") pod \"maas-api-e2e-ait-7164dd00-6c969c79f6-2g46h\" (UID: \"8270a0a8-7261-45b3-849f-6c05a6b17f8e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-7164dd00-6c969c79f6-2g46h"
Oct 31 01:15:07.320761 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:15:07.320723    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8270a0a8-7261-45b3-849f-6c05a6b17f8e-maas-api-tls\") pod \"maas-api-e2e-ait-7164dd00-6c969c79f6-2g46h\" (UID: \"8270a0a8-7261-45b3-849f-6c05a6b17f8e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-7164dd00-6c969c79f6-2g46h"
Oct 31 01:15:07.434549 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:15:07.434519    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-7164dd00-6c969c79f6-2g46h"
Oct 31 01:15:07.718307 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:15:07.718172    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:15:07.834933 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:15:07.834907    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-7164dd00-6c969c79f6-2g46h"]
Oct 31 01:15:07.836492 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:15:07.836450    4964 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8270a0a8_7261_45b3_849f_6c05a6b17f8e.slice/crio-d715d222023a68bd40b2de940a837d31679030a65b0ed097adfdf843d898651b WatchSource:0}: Error finding container d715d222023a68bd40b2de940a837d31679030a65b0ed097adfdf843d898651b: Status 404 returned error can't find the container with id d715d222023a68bd40b2de940a837d31679030a65b0ed097adfdf843d898651b
Oct 31 01:15:07.916468 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:15:07.916432    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:15:08.264251 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:15:08.264220    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-7164dd00-6dcd9d9bc9-24xtq"
Oct 31 01:15:08.351048 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:15:08.351007    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-7164dd00-6c969c79f6-2g46h" event={"ID":"8270a0a8-7261-45b3-849f-6c05a6b17f8e","Type":"ContainerStarted","Data":"d715d222023a68bd40b2de940a837d31679030a65b0ed097adfdf843d898651b"}
Oct 31 01:15:09.358732 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:15:09.358700    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-7164dd00-6c969c79f6-2g46h" event={"ID":"8270a0a8-7261-45b3-849f-6c05a6b17f8e","Type":"ContainerStarted","Data":"c72aceac6d2ef75286f906229ff65c57423f8c6dd01cd6f2a45916f868269a00"}
Oct 31 01:15:09.359137 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:15:09.358815    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-7164dd00-6c969c79f6-2g46h"
Oct 31 01:15:09.478209 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:15:09.478093    4964 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-7164dd00-6c969c79f6-2g46h" podStartSLOduration=2.658121736 podStartE2EDuration="3.478071616s" totalImagesPullingTime="819.94988ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:15:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:15:09.473134997 +0000 UTC m=+677.949701407" watchObservedRunningTime="2026-10-31 01:15:09.478071616 +0000 UTC m=+677.954638004"
Oct 31 01:15:11.239134 ip-10-0-133-197 kubenswrapper[4964]: E1031 01:15:11.239100    4964 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q56x5" containerName="extractor"
Oct 31 01:15:11.847912 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:15:11.847873    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:15:12.084372 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:15:12.084311    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:15:13.385725 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:15:13.385694    4964 generic.go:356] "Generic (PLEG): container finished" podID="41125f0c-f8ce-416b-92b0-4d6535255a5e" containerID="d9f691765b032b8ac0927bcead1177abb538a4eb46b02af7d90f649c077e5c31" exitCode=0
Oct 31 01:15:13.386144 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:15:13.385763    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-a0c7a371-gw-openshift-default-7857fb8585-vnnng" event={"ID":"41125f0c-f8ce-416b-92b0-4d6535255a5e","Type":"ContainerDied","Data":"d9f691765b032b8ac0927bcead1177abb538a4eb46b02af7d90f649c077e5c31"}
Oct 31 01:15:13.386190 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:15:13.386165    4964 scope.go:122] "RemoveContainer" containerID="d9f691765b032b8ac0927bcead1177abb538a4eb46b02af7d90f649c077e5c31"
Oct 31 01:15:13.387523 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:15:13.387482    4964 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 01:15:13.387640 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:15:13.387565    4964 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 01:15:13.387640 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:15:13.387613    4964 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 01:15:13.422776 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:15:13.422743    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:15:13.863077 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:15:13.863024    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:15:14.162833 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:15:14.162754    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-a0c7a371-gw-openshift-default-7857fb8585-vnnng"
Oct 31 01:15:14.418109 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:15:14.418022    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-a0c7a371-gw-openshift-default-7857fb8585-vnnng" event={"ID":"41125f0c-f8ce-416b-92b0-4d6535255a5e","Type":"ContainerStarted","Data":"9eb973ebe3085655bb8f60dc0681c07b56fd7889921fd5fb00bd8d731e256d20"}
Oct 31 01:15:14.617056 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:15:14.617021    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:15:14.761148 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:15:14.761114    4964 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-a0c7a371-gw-openshift-default-7857fb8585-vnnng"
Oct 31 01:15:14.762502 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:15:14.762478    4964 patch_prober.go:29] interesting pod/e2e-migrate-a0c7a371-gw-openshift-default-7857fb8585-vnnng container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body=
Oct 31 01:15:14.762648 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:15:14.762525    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-a0c7a371-gw-openshift-default-7857fb8585-vnnng" podUID="41125f0c-f8ce-416b-92b0-4d6535255a5e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused"
Oct 31 01:15:15.371885 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:15:15.371855    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-7164dd00-6c969c79f6-2g46h"
Oct 31 01:15:15.762096 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:15:15.762062    4964 patch_prober.go:29] interesting pod/e2e-migrate-a0c7a371-gw-openshift-default-7857fb8585-vnnng container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body=
Oct 31 01:15:15.762549 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:15:15.762128    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-a0c7a371-gw-openshift-default-7857fb8585-vnnng" podUID="41125f0c-f8ce-416b-92b0-4d6535255a5e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused"
Oct 31 01:15:16.256805 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:15:16.256768    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:15:16.342866 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:15:16.342833    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-b9656f65-755b48cd75-lpd6d"
Oct 31 01:15:16.761951 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:15:16.761916    4964 patch_prober.go:29] interesting pod/e2e-migrate-a0c7a371-gw-openshift-default-7857fb8585-vnnng container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body=
Oct 31 01:15:16.762146 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:15:16.761984    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-a0c7a371-gw-openshift-default-7857fb8585-vnnng" podUID="41125f0c-f8ce-416b-92b0-4d6535255a5e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused"
Oct 31 01:15:16.915586 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:15:16.915544    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:15:17.762278 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:15:17.762242    4964 patch_prober.go:29] interesting pod/e2e-migrate-a0c7a371-gw-openshift-default-7857fb8585-vnnng container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body=
Oct 31 01:15:17.762816 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:15:17.762293    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-a0c7a371-gw-openshift-default-7857fb8585-vnnng" podUID="41125f0c-f8ce-416b-92b0-4d6535255a5e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused"
Oct 31 01:15:18.761796 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:15:18.761764    4964 patch_prober.go:29] interesting pod/e2e-migrate-a0c7a371-gw-openshift-default-7857fb8585-vnnng container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body=
Oct 31 01:15:18.761975 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:15:18.761815    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-a0c7a371-gw-openshift-default-7857fb8585-vnnng" podUID="41125f0c-f8ce-416b-92b0-4d6535255a5e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused"
Oct 31 01:15:19.766069 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:15:19.766041    4964 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-a0c7a371-gw-openshift-default-7857fb8585-vnnng"
Oct 31 01:15:19.766565 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:15:19.766548    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-a0c7a371-gw-openshift-default-7857fb8585-vnnng"
Oct 31 01:15:19.767255 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:15:19.767237    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-a0c7a371-gw-openshift-default-7857fb8585-vnnng"
Oct 31 01:15:21.218396 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:15:21.218363    4964 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-870eefeb-cc99bd9cb-g7445"]
Oct 31 01:15:21.222385 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:15:21.222369    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-870eefeb-cc99bd9cb-g7445"
Oct 31 01:15:21.276131 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:15:21.276106    4964 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-870eefeb-serving-cert\""
Oct 31 01:15:21.289611 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:15:21.289586    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xk9fd\" (UniqueName: \"kubernetes.io/projected/61037cfe-c14f-4293-b6f6-e49775dc2c94-kube-api-access-xk9fd\") pod \"maas-api-e2e-ait-870eefeb-cc99bd9cb-g7445\" (UID: \"61037cfe-c14f-4293-b6f6-e49775dc2c94\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-870eefeb-cc99bd9cb-g7445"
Oct 31 01:15:21.289756 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:15:21.289635    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/61037cfe-c14f-4293-b6f6-e49775dc2c94-metrics-certs\") pod \"maas-api-e2e-ait-870eefeb-cc99bd9cb-g7445\" (UID: \"61037cfe-c14f-4293-b6f6-e49775dc2c94\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-870eefeb-cc99bd9cb-g7445"
Oct 31 01:15:21.289756 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:15:21.289708    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/61037cfe-c14f-4293-b6f6-e49775dc2c94-openshift-service-ca\") pod \"maas-api-e2e-ait-870eefeb-cc99bd9cb-g7445\" (UID: \"61037cfe-c14f-4293-b6f6-e49775dc2c94\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-870eefeb-cc99bd9cb-g7445"
Oct 31 01:15:21.289846 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:15:21.289833    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/61037cfe-c14f-4293-b6f6-e49775dc2c94-maas-api-tls\") pod \"maas-api-e2e-ait-870eefeb-cc99bd9cb-g7445\" (UID: \"61037cfe-c14f-4293-b6f6-e49775dc2c94\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-870eefeb-cc99bd9cb-g7445"
Oct 31 01:15:21.327300 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:15:21.327269    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-870eefeb-cc99bd9cb-g7445"]
Oct 31 01:15:21.413344 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:15:21.390911    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/61037cfe-c14f-4293-b6f6-e49775dc2c94-maas-api-tls\") pod \"maas-api-e2e-ait-870eefeb-cc99bd9cb-g7445\" (UID: \"61037cfe-c14f-4293-b6f6-e49775dc2c94\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-870eefeb-cc99bd9cb-g7445"
Oct 31 01:15:21.413761 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:15:21.413733    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xk9fd\" (UniqueName: \"kubernetes.io/projected/61037cfe-c14f-4293-b6f6-e49775dc2c94-kube-api-access-xk9fd\") pod \"maas-api-e2e-ait-870eefeb-cc99bd9cb-g7445\" (UID: \"61037cfe-c14f-4293-b6f6-e49775dc2c94\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-870eefeb-cc99bd9cb-g7445"
Oct 31 01:15:21.414336 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:15:21.414290    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/61037cfe-c14f-4293-b6f6-e49775dc2c94-metrics-certs\") pod \"maas-api-e2e-ait-870eefeb-cc99bd9cb-g7445\" (UID: \"61037cfe-c14f-4293-b6f6-e49775dc2c94\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-870eefeb-cc99bd9cb-g7445"
Oct 31 01:15:21.414537 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:15:21.414518    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/61037cfe-c14f-4293-b6f6-e49775dc2c94-openshift-service-ca\") pod \"maas-api-e2e-ait-870eefeb-cc99bd9cb-g7445\" (UID: \"61037cfe-c14f-4293-b6f6-e49775dc2c94\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-870eefeb-cc99bd9cb-g7445"
Oct 31 01:15:21.415394 ip-10-0-133-197 kubenswrapper[4964]: E1031 01:15:21.403464    4964 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-870eefeb-serving-cert: secret "maas-api-e2e-ait-870eefeb-serving-cert" not found
Oct 31 01:15:21.415524 ip-10-0-133-197 kubenswrapper[4964]: E1031 01:15:21.415488    4964 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/61037cfe-c14f-4293-b6f6-e49775dc2c94-maas-api-tls podName:61037cfe-c14f-4293-b6f6-e49775dc2c94 nodeName:}" failed. No retries permitted until 2026-10-31 01:15:21.91546233 +0000 UTC m=+690.392028704 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/61037cfe-c14f-4293-b6f6-e49775dc2c94-maas-api-tls") pod "maas-api-e2e-ait-870eefeb-cc99bd9cb-g7445" (UID: "61037cfe-c14f-4293-b6f6-e49775dc2c94") : secret "maas-api-e2e-ait-870eefeb-serving-cert" not found
Oct 31 01:15:21.416348 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:15:21.416304    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/61037cfe-c14f-4293-b6f6-e49775dc2c94-openshift-service-ca\") pod \"maas-api-e2e-ait-870eefeb-cc99bd9cb-g7445\" (UID: \"61037cfe-c14f-4293-b6f6-e49775dc2c94\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-870eefeb-cc99bd9cb-g7445"
Oct 31 01:15:21.418681 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:15:21.418653    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/61037cfe-c14f-4293-b6f6-e49775dc2c94-metrics-certs\") pod \"maas-api-e2e-ait-870eefeb-cc99bd9cb-g7445\" (UID: \"61037cfe-c14f-4293-b6f6-e49775dc2c94\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-870eefeb-cc99bd9cb-g7445"
Oct 31 01:15:21.459369 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:15:21.459343    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:15:21.470613 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:15:21.470593    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xk9fd\" (UniqueName: \"kubernetes.io/projected/61037cfe-c14f-4293-b6f6-e49775dc2c94-kube-api-access-xk9fd\") pod \"maas-api-e2e-ait-870eefeb-cc99bd9cb-g7445\" (UID: \"61037cfe-c14f-4293-b6f6-e49775dc2c94\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-870eefeb-cc99bd9cb-g7445"
Oct 31 01:15:21.642722 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:15:21.642683    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:15:21.820880 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:15:21.820786    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:15:21.920985 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:15:21.920954    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/61037cfe-c14f-4293-b6f6-e49775dc2c94-maas-api-tls\") pod \"maas-api-e2e-ait-870eefeb-cc99bd9cb-g7445\" (UID: \"61037cfe-c14f-4293-b6f6-e49775dc2c94\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-870eefeb-cc99bd9cb-g7445"
Oct 31 01:15:21.923555 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:15:21.923526    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/61037cfe-c14f-4293-b6f6-e49775dc2c94-maas-api-tls\") pod \"maas-api-e2e-ait-870eefeb-cc99bd9cb-g7445\" (UID: \"61037cfe-c14f-4293-b6f6-e49775dc2c94\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-870eefeb-cc99bd9cb-g7445"
Oct 31 01:15:22.135805 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:15:22.135699    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-870eefeb-cc99bd9cb-g7445"
Oct 31 01:15:22.534609 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:15:22.534577    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-870eefeb-cc99bd9cb-g7445"]
Oct 31 01:15:22.538203 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:15:22.538177    4964 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61037cfe_c14f_4293_b6f6_e49775dc2c94.slice/crio-a87ca34da28947b69cf60e618575dc55ca2da07e0af4b1224ab554ac2075725e WatchSource:0}: Error finding container a87ca34da28947b69cf60e618575dc55ca2da07e0af4b1224ab554ac2075725e: Status 404 returned error can't find the container with id a87ca34da28947b69cf60e618575dc55ca2da07e0af4b1224ab554ac2075725e
Oct 31 01:15:23.477900 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:15:23.477867    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-870eefeb-cc99bd9cb-g7445" event={"ID":"61037cfe-c14f-4293-b6f6-e49775dc2c94","Type":"ContainerStarted","Data":"a87ca34da28947b69cf60e618575dc55ca2da07e0af4b1224ab554ac2075725e"}
Oct 31 01:15:23.813342 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:15:23.812586    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:15:24.526063 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:15:24.526026    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-870eefeb-cc99bd9cb-g7445" event={"ID":"61037cfe-c14f-4293-b6f6-e49775dc2c94","Type":"ContainerStarted","Data":"2e931c55a2bd29092f6d2d3f578aece147c2468de2bcbac0aa5411cc767e4c90"}
Oct 31 01:15:24.527390 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:15:24.527280    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-870eefeb-cc99bd9cb-g7445"
Oct 31 01:15:24.642022 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:15:24.641971    4964 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-870eefeb-cc99bd9cb-g7445" podStartSLOduration=2.803315702 podStartE2EDuration="3.641956959s" totalImagesPullingTime="838.641257ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:15:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:15:24.641916186 +0000 UTC m=+693.118482566" watchObservedRunningTime="2026-10-31 01:15:24.641956959 +0000 UTC m=+693.118523336"
Oct 31 01:15:26.854714 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:15:26.854668    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:15:29.557687 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:15:29.557661    4964 generic.go:356] "Generic (PLEG): container finished" podID="9b23eb8c-e963-42d6-a1f2-61e474aaa829" containerID="c33cd5188f07608e3088a7c08784ba8e02b959608c559196365e86d4e5589a7d" exitCode=0
Oct 31 01:15:29.558097 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:15:29.557740    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-b9656f65-openshift-default-c6f497c7-r4pr4" event={"ID":"9b23eb8c-e963-42d6-a1f2-61e474aaa829","Type":"ContainerDied","Data":"c33cd5188f07608e3088a7c08784ba8e02b959608c559196365e86d4e5589a7d"}
Oct 31 01:15:29.558143 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:15:29.558116    4964 scope.go:122] "RemoveContainer" containerID="c33cd5188f07608e3088a7c08784ba8e02b959608c559196365e86d4e5589a7d"
Oct 31 01:15:29.559498 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:15:29.559471    4964 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 01:15:29.559578 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:15:29.559540    4964 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 01:15:29.559578 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:15:29.559572    4964 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 01:15:30.565632 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:15:30.565592    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-b9656f65-openshift-default-c6f497c7-r4pr4" event={"ID":"9b23eb8c-e963-42d6-a1f2-61e474aaa829","Type":"ContainerStarted","Data":"49327c69dd44d704eb65dede3c38075873ae1c118a54578f04c659aba237ac9e"}
Oct 31 01:15:31.102344 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:15:31.095379    4964 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-b9656f65-openshift-default-c6f497c7-r4pr4"
Oct 31 01:15:31.110356 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:15:31.108475    4964 patch_prober.go:29] interesting pod/e2e-derive-b9656f65-openshift-default-c6f497c7-r4pr4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body=
Oct 31 01:15:31.110356 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:15:31.108546    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-b9656f65-openshift-default-c6f497c7-r4pr4" podUID="9b23eb8c-e963-42d6-a1f2-61e474aaa829" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused"
Oct 31 01:15:31.543807 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:15:31.543776    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-870eefeb-cc99bd9cb-g7445"
Oct 31 01:15:32.110835 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:15:32.110798    4964 patch_prober.go:29] interesting pod/e2e-derive-b9656f65-openshift-default-c6f497c7-r4pr4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body=
Oct 31 01:15:32.111500 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:15:32.111463    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-b9656f65-openshift-default-c6f497c7-r4pr4" podUID="9b23eb8c-e963-42d6-a1f2-61e474aaa829" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused"
Oct 31 01:15:32.117132 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:15:32.117099    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:15:33.101650 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:15:33.095477    4964 patch_prober.go:29] interesting pod/e2e-derive-b9656f65-openshift-default-c6f497c7-r4pr4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body=
Oct 31 01:15:33.101650 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:15:33.095537    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-b9656f65-openshift-default-c6f497c7-r4pr4" podUID="9b23eb8c-e963-42d6-a1f2-61e474aaa829" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused"
Oct 31 01:15:34.099346 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:15:34.095698    4964 patch_prober.go:29] interesting pod/e2e-derive-b9656f65-openshift-default-c6f497c7-r4pr4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body=
Oct 31 01:15:34.099346 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:15:34.095762    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-b9656f65-openshift-default-c6f497c7-r4pr4" podUID="9b23eb8c-e963-42d6-a1f2-61e474aaa829" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused"
Oct 31 01:15:35.112343 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:15:35.111812    4964 patch_prober.go:29] interesting pod/e2e-derive-b9656f65-openshift-default-c6f497c7-r4pr4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body=
Oct 31 01:15:35.112343 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:15:35.111888    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-b9656f65-openshift-default-c6f497c7-r4pr4" podUID="9b23eb8c-e963-42d6-a1f2-61e474aaa829" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused"
Oct 31 01:15:35.210342 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:15:35.205443    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-b9656f65-openshift-default-c6f497c7-r4pr4"
Oct 31 01:15:36.097559 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:15:36.097533    4964 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-b9656f65-openshift-default-c6f497c7-r4pr4"
Oct 31 01:15:36.099032 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:15:36.099005    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-b9656f65-openshift-default-c6f497c7-r4pr4"
Oct 31 01:15:37.318105 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:15:37.318071    4964 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-a0c7a371-7d55468d95-tjb8l"]
Oct 31 01:15:37.344553 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:15:37.344519    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:15:37.724675 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:15:37.724642    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:15:37.970909 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:15:37.970870    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:15:41.237586 ip-10-0-133-197 kubenswrapper[4964]: E1031 01:15:41.237540    4964 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q56x5" containerName="extractor"
Oct 31 01:15:42.319416 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:15:42.319372    4964 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-migrate-a0c7a371-7d55468d95-tjb8l" podUID="b537e3cd-aa63-4840-8ec5-4447d616d56f" containerName="payload-processing" containerID="cri-o://0a6ffaf259248353a39cb4870af766d3ebf0283c77d5e1d0294e9c0d6327c6a6" gracePeriod=25
Oct 31 01:15:42.585494 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:15:42.585435    4964 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-a0c7a371-7d55468d95-tjb8l"
Oct 31 01:15:42.652202 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:15:42.652171    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b537e3cd-aa63-4840-8ec5-4447d616d56f-kube-api-access-zgfnm\" (UniqueName: \"kubernetes.io/projected/b537e3cd-aa63-4840-8ec5-4447d616d56f-kube-api-access-zgfnm\") pod \"b537e3cd-aa63-4840-8ec5-4447d616d56f\" (UID: \"b537e3cd-aa63-4840-8ec5-4447d616d56f\") "
Oct 31 01:15:42.652434 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:15:42.652281    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b537e3cd-aa63-4840-8ec5-4447d616d56f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b537e3cd-aa63-4840-8ec5-4447d616d56f-plugins-config-volume\") pod \"b537e3cd-aa63-4840-8ec5-4447d616d56f\" (UID: \"b537e3cd-aa63-4840-8ec5-4447d616d56f\") "
Oct 31 01:15:42.652606 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:15:42.652579    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b537e3cd-aa63-4840-8ec5-4447d616d56f-plugins-config-volume" pod "b537e3cd-aa63-4840-8ec5-4447d616d56f" (UID: "b537e3cd-aa63-4840-8ec5-4447d616d56f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:15:42.652747 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:15:42.652731    4964 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b537e3cd-aa63-4840-8ec5-4447d616d56f-plugins-config-volume\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:15:42.654679 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:15:42.654655    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b537e3cd-aa63-4840-8ec5-4447d616d56f-kube-api-access-zgfnm" pod "b537e3cd-aa63-4840-8ec5-4447d616d56f" (UID: "b537e3cd-aa63-4840-8ec5-4447d616d56f"). InnerVolumeSpecName "kube-api-access-zgfnm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:15:42.669680 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:15:42.669652    4964 generic.go:356] "Generic (PLEG): container finished" podID="b537e3cd-aa63-4840-8ec5-4447d616d56f" containerID="0a6ffaf259248353a39cb4870af766d3ebf0283c77d5e1d0294e9c0d6327c6a6" exitCode=0
Oct 31 01:15:42.669800 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:15:42.669702    4964 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-a0c7a371-7d55468d95-tjb8l"
Oct 31 01:15:42.669800 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:15:42.669732    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-a0c7a371-7d55468d95-tjb8l" event={"ID":"b537e3cd-aa63-4840-8ec5-4447d616d56f","Type":"ContainerDied","Data":"0a6ffaf259248353a39cb4870af766d3ebf0283c77d5e1d0294e9c0d6327c6a6"}
Oct 31 01:15:42.669800 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:15:42.669787    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-a0c7a371-7d55468d95-tjb8l" event={"ID":"b537e3cd-aa63-4840-8ec5-4447d616d56f","Type":"ContainerDied","Data":"8576da08365ae53cda5435a08d8bd3d03ba0fd95ca9908030bdc777a4801467f"}
Oct 31 01:15:42.669967 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:15:42.669816    4964 scope.go:122] "RemoveContainer" containerID="0a6ffaf259248353a39cb4870af766d3ebf0283c77d5e1d0294e9c0d6327c6a6"
Oct 31 01:15:42.679882 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:15:42.679867    4964 scope.go:122] "RemoveContainer" containerID="0a6ffaf259248353a39cb4870af766d3ebf0283c77d5e1d0294e9c0d6327c6a6"
Oct 31 01:15:42.680136 ip-10-0-133-197 kubenswrapper[4964]: E1031 01:15:42.680118    4964 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0a6ffaf259248353a39cb4870af766d3ebf0283c77d5e1d0294e9c0d6327c6a6\": container with ID starting with 0a6ffaf259248353a39cb4870af766d3ebf0283c77d5e1d0294e9c0d6327c6a6 not found: ID does not exist" containerID="0a6ffaf259248353a39cb4870af766d3ebf0283c77d5e1d0294e9c0d6327c6a6"
Oct 31 01:15:42.680174 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:15:42.680145    4964 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0a6ffaf259248353a39cb4870af766d3ebf0283c77d5e1d0294e9c0d6327c6a6"} err="failed to get container status \"0a6ffaf259248353a39cb4870af766d3ebf0283c77d5e1d0294e9c0d6327c6a6\": rpc error: code = NotFound desc = could not find container \"0a6ffaf259248353a39cb4870af766d3ebf0283c77d5e1d0294e9c0d6327c6a6\": container with ID starting with 0a6ffaf259248353a39cb4870af766d3ebf0283c77d5e1d0294e9c0d6327c6a6 not found: ID does not exist"
Oct 31 01:15:42.735612 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:15:42.735580    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:15:42.753541 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:15:42.753511    4964 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zgfnm\" (UniqueName: \"kubernetes.io/projected/b537e3cd-aa63-4840-8ec5-4447d616d56f-kube-api-access-zgfnm\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:15:42.815793 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:15:42.815760    4964 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-a0c7a371-7d55468d95-tjb8l"]
Oct 31 01:15:42.818048 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:15:42.818019    4964 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-a0c7a371-7d55468d95-tjb8l"]
Oct 31 01:15:43.776160 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:15:43.776127    4964 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b537e3cd-aa63-4840-8ec5-4447d616d56f" path="/var/lib/kubelet/pods/b537e3cd-aa63-4840-8ec5-4447d616d56f/volumes"
Oct 31 01:15:44.841910 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:15:44.841871    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:15:47.153858 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:15:47.153822    4964 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-a0c7a371-76657b6bb8-nfjxd"]
Oct 31 01:15:47.155188 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:15:47.155154    4964 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-a0c7a371-76657b6bb8-nfjxd" podUID="9dc20490-124e-40b4-9502-10949d62d531" containerName="maas-api" containerID="cri-o://2be436bec730505e5a0f715ee14233e7871912d291c7276e13559a82582fb81f" gracePeriod=30
Oct 31 01:15:47.299275 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:15:47.299226    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:15:47.437434 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:15:47.437410    4964 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-a0c7a371-76657b6bb8-nfjxd"
Oct 31 01:15:47.611464 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:15:47.611430    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9dc20490-124e-40b4-9502-10949d62d531-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9dc20490-124e-40b4-9502-10949d62d531-openshift-service-ca\") pod \"9dc20490-124e-40b4-9502-10949d62d531\" (UID: \"9dc20490-124e-40b4-9502-10949d62d531\") "
Oct 31 01:15:47.611635 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:15:47.611551    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9dc20490-124e-40b4-9502-10949d62d531-kube-api-access-b972c\" (UniqueName: \"kubernetes.io/projected/9dc20490-124e-40b4-9502-10949d62d531-kube-api-access-b972c\") pod \"9dc20490-124e-40b4-9502-10949d62d531\" (UID: \"9dc20490-124e-40b4-9502-10949d62d531\") "
Oct 31 01:15:47.611635 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:15:47.611626    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9dc20490-124e-40b4-9502-10949d62d531-metrics-certs\" (UniqueName: \"kubernetes.io/secret/9dc20490-124e-40b4-9502-10949d62d531-metrics-certs\") pod \"9dc20490-124e-40b4-9502-10949d62d531\" (UID: \"9dc20490-124e-40b4-9502-10949d62d531\") "
Oct 31 01:15:47.611716 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:15:47.611658    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9dc20490-124e-40b4-9502-10949d62d531-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9dc20490-124e-40b4-9502-10949d62d531-maas-api-tls\") pod \"9dc20490-124e-40b4-9502-10949d62d531\" (UID: \"9dc20490-124e-40b4-9502-10949d62d531\") "
Oct 31 01:15:47.611808 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:15:47.611775    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9dc20490-124e-40b4-9502-10949d62d531-openshift-service-ca" pod "9dc20490-124e-40b4-9502-10949d62d531" (UID: "9dc20490-124e-40b4-9502-10949d62d531"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:15:47.612035 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:15:47.612017    4964 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9dc20490-124e-40b4-9502-10949d62d531-openshift-service-ca\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:15:47.613861 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:15:47.613830    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9dc20490-124e-40b4-9502-10949d62d531-metrics-certs" pod "9dc20490-124e-40b4-9502-10949d62d531" (UID: "9dc20490-124e-40b4-9502-10949d62d531"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:15:47.613953 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:15:47.613895    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9dc20490-124e-40b4-9502-10949d62d531-kube-api-access-b972c" pod "9dc20490-124e-40b4-9502-10949d62d531" (UID: "9dc20490-124e-40b4-9502-10949d62d531"). InnerVolumeSpecName "kube-api-access-b972c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:15:47.613953 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:15:47.613894    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9dc20490-124e-40b4-9502-10949d62d531-maas-api-tls" pod "9dc20490-124e-40b4-9502-10949d62d531" (UID: "9dc20490-124e-40b4-9502-10949d62d531"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:15:47.701393 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:15:47.701281    4964 generic.go:356] "Generic (PLEG): container finished" podID="9dc20490-124e-40b4-9502-10949d62d531" containerID="2be436bec730505e5a0f715ee14233e7871912d291c7276e13559a82582fb81f" exitCode=0
Oct 31 01:15:47.701393 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:15:47.701352    4964 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-a0c7a371-76657b6bb8-nfjxd"
Oct 31 01:15:47.701393 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:15:47.701374    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-a0c7a371-76657b6bb8-nfjxd" event={"ID":"9dc20490-124e-40b4-9502-10949d62d531","Type":"ContainerDied","Data":"2be436bec730505e5a0f715ee14233e7871912d291c7276e13559a82582fb81f"}
Oct 31 01:15:47.701680 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:15:47.701453    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-a0c7a371-76657b6bb8-nfjxd" event={"ID":"9dc20490-124e-40b4-9502-10949d62d531","Type":"ContainerDied","Data":"e5cc2ae7b0ccb3b2c0af893c39ad7d9b379aa61f5aabb4234a5ea91139b61264"}
Oct 31 01:15:47.701680 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:15:47.701483    4964 scope.go:122] "RemoveContainer" containerID="2be436bec730505e5a0f715ee14233e7871912d291c7276e13559a82582fb81f"
Oct 31 01:15:47.712144 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:15:47.712127    4964 scope.go:122] "RemoveContainer" containerID="2be436bec730505e5a0f715ee14233e7871912d291c7276e13559a82582fb81f"
Oct 31 01:15:47.712453 ip-10-0-133-197 kubenswrapper[4964]: E1031 01:15:47.712430    4964 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2be436bec730505e5a0f715ee14233e7871912d291c7276e13559a82582fb81f\": container with ID starting with 2be436bec730505e5a0f715ee14233e7871912d291c7276e13559a82582fb81f not found: ID does not exist" containerID="2be436bec730505e5a0f715ee14233e7871912d291c7276e13559a82582fb81f"
Oct 31 01:15:47.712535 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:15:47.712461    4964 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2be436bec730505e5a0f715ee14233e7871912d291c7276e13559a82582fb81f"} err="failed to get container status \"2be436bec730505e5a0f715ee14233e7871912d291c7276e13559a82582fb81f\": rpc error: code = NotFound desc = could not find container \"2be436bec730505e5a0f715ee14233e7871912d291c7276e13559a82582fb81f\": container with ID starting with 2be436bec730505e5a0f715ee14233e7871912d291c7276e13559a82582fb81f not found: ID does not exist"
Oct 31 01:15:47.712659 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:15:47.712643    4964 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b972c\" (UniqueName: \"kubernetes.io/projected/9dc20490-124e-40b4-9502-10949d62d531-kube-api-access-b972c\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:15:47.712702 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:15:47.712663    4964 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9dc20490-124e-40b4-9502-10949d62d531-metrics-certs\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:15:47.712702 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:15:47.712674    4964 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9dc20490-124e-40b4-9502-10949d62d531-maas-api-tls\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:15:47.863822 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:15:47.863785    4964 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-a0c7a371-76657b6bb8-nfjxd"]
Oct 31 01:15:47.867440 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:15:47.867416    4964 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-a0c7a371-76657b6bb8-nfjxd"]
Oct 31 01:15:49.086061 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:15:49.086022    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:15:49.292879 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:15:49.292828    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:15:49.478899 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:15:49.478863    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:15:49.747893 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:15:49.747815    4964 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-a0c7a371-gw-openshift-default-7857fb8585-vnnng"]
Oct 31 01:15:49.748062 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:15:49.748042    4964 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-migrate-a0c7a371-gw-openshift-default-7857fb8585-vnnng" podUID="41125f0c-f8ce-416b-92b0-4d6535255a5e" containerName="istio-proxy" containerID="cri-o://9eb973ebe3085655bb8f60dc0681c07b56fd7889921fd5fb00bd8d731e256d20" gracePeriod=30
Oct 31 01:15:49.768732 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:15:49.768706    4964 patch_prober.go:29] interesting pod/e2e-migrate-a0c7a371-gw-openshift-default-7857fb8585-vnnng container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 01:15:49.768859 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:15:49.768758    4964 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-migrate-a0c7a371-gw-openshift-default-7857fb8585-vnnng" podUID="41125f0c-f8ce-416b-92b0-4d6535255a5e" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 01:15:49.776623 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:15:49.776595    4964 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9dc20490-124e-40b4-9502-10949d62d531" path="/var/lib/kubelet/pods/9dc20490-124e-40b4-9502-10949d62d531/volumes"
Oct 31 01:15:49.792531 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:15:49.792499    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:15:50.618836 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:15:50.618796    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:15:51.220720 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:15:51.220686    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:15:51.422371 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:15:51.422338    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:15:52.021763 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:15:52.021729    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:15:52.249676 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:15:52.249642    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:15:53.576269 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:15:53.576234    4964 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-f94871a1-6f988c57d9-t52mh"]
Oct 31 01:15:53.576760 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:15:53.576490    4964 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f94871a1-6f988c57d9-t52mh" podUID="101c024d-b61f-47e5-a863-330ef7b7069d" containerName="maas-api" containerID="cri-o://81ab776d077584ef5bbafcffb33cbed319d98b968a758b10bd77e3e89203eb75" gracePeriod=30
Oct 31 01:15:53.721849 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:15:53.721807    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:15:53.741490 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:15:53.741456    4964 generic.go:356] "Generic (PLEG): container finished" podID="101c024d-b61f-47e5-a863-330ef7b7069d" containerID="81ab776d077584ef5bbafcffb33cbed319d98b968a758b10bd77e3e89203eb75" exitCode=0
Oct 31 01:15:53.741663 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:15:53.741527    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f94871a1-6f988c57d9-t52mh" event={"ID":"101c024d-b61f-47e5-a863-330ef7b7069d","Type":"ContainerDied","Data":"81ab776d077584ef5bbafcffb33cbed319d98b968a758b10bd77e3e89203eb75"}
Oct 31 01:15:53.851558 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:15:53.851529    4964 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f94871a1-6f988c57d9-t52mh"
Oct 31 01:15:53.885437 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:15:53.885404    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/101c024d-b61f-47e5-a863-330ef7b7069d-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/101c024d-b61f-47e5-a863-330ef7b7069d-maas-api-tls\") pod \"101c024d-b61f-47e5-a863-330ef7b7069d\" (UID: \"101c024d-b61f-47e5-a863-330ef7b7069d\") "
Oct 31 01:15:53.885612 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:15:53.885449    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/101c024d-b61f-47e5-a863-330ef7b7069d-metrics-certs\" (UniqueName: \"kubernetes.io/secret/101c024d-b61f-47e5-a863-330ef7b7069d-metrics-certs\") pod \"101c024d-b61f-47e5-a863-330ef7b7069d\" (UID: \"101c024d-b61f-47e5-a863-330ef7b7069d\") "
Oct 31 01:15:53.885612 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:15:53.885489    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/101c024d-b61f-47e5-a863-330ef7b7069d-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/101c024d-b61f-47e5-a863-330ef7b7069d-openshift-service-ca\") pod \"101c024d-b61f-47e5-a863-330ef7b7069d\" (UID: \"101c024d-b61f-47e5-a863-330ef7b7069d\") "
Oct 31 01:15:53.885612 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:15:53.885558    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/101c024d-b61f-47e5-a863-330ef7b7069d-kube-api-access-8xmx4\" (UniqueName: \"kubernetes.io/projected/101c024d-b61f-47e5-a863-330ef7b7069d-kube-api-access-8xmx4\") pod \"101c024d-b61f-47e5-a863-330ef7b7069d\" (UID: \"101c024d-b61f-47e5-a863-330ef7b7069d\") "
Oct 31 01:15:53.885934 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:15:53.885897    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/101c024d-b61f-47e5-a863-330ef7b7069d-openshift-service-ca" pod "101c024d-b61f-47e5-a863-330ef7b7069d" (UID: "101c024d-b61f-47e5-a863-330ef7b7069d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:15:53.887775 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:15:53.887741    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/101c024d-b61f-47e5-a863-330ef7b7069d-maas-api-tls" pod "101c024d-b61f-47e5-a863-330ef7b7069d" (UID: "101c024d-b61f-47e5-a863-330ef7b7069d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:15:53.887896 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:15:53.887847    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/101c024d-b61f-47e5-a863-330ef7b7069d-kube-api-access-8xmx4" pod "101c024d-b61f-47e5-a863-330ef7b7069d" (UID: "101c024d-b61f-47e5-a863-330ef7b7069d"). InnerVolumeSpecName "kube-api-access-8xmx4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:15:53.887969 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:15:53.887952    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/101c024d-b61f-47e5-a863-330ef7b7069d-metrics-certs" pod "101c024d-b61f-47e5-a863-330ef7b7069d" (UID: "101c024d-b61f-47e5-a863-330ef7b7069d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:15:53.986485 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:15:53.986453    4964 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8xmx4\" (UniqueName: \"kubernetes.io/projected/101c024d-b61f-47e5-a863-330ef7b7069d-kube-api-access-8xmx4\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:15:53.986485 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:15:53.986483    4964 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/101c024d-b61f-47e5-a863-330ef7b7069d-maas-api-tls\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:15:53.986485 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:15:53.986493    4964 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/101c024d-b61f-47e5-a863-330ef7b7069d-metrics-certs\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:15:53.986792 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:15:53.986502    4964 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/101c024d-b61f-47e5-a863-330ef7b7069d-openshift-service-ca\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:15:54.751014 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:15:54.750978    4964 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f94871a1-6f988c57d9-t52mh"
Oct 31 01:15:54.751574 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:15:54.750978    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f94871a1-6f988c57d9-t52mh" event={"ID":"101c024d-b61f-47e5-a863-330ef7b7069d","Type":"ContainerDied","Data":"0f7683f743c0db2072ad5bc188692cc2159290d668917afb04e69152fef455e2"}
Oct 31 01:15:54.751574 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:15:54.751122    4964 scope.go:122] "RemoveContainer" containerID="81ab776d077584ef5bbafcffb33cbed319d98b968a758b10bd77e3e89203eb75"
Oct 31 01:15:54.914786 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:15:54.914754    4964 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-f94871a1-6f988c57d9-t52mh"]
Oct 31 01:15:54.917825 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:15:54.917787    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:15:54.966772 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:15:54.966742    4964 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-f94871a1-6f988c57d9-t52mh"]
Oct 31 01:15:55.044289 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:15:55.044264    4964 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-a0c7a371-gw-openshift-default-7857fb8585-vnnng"
Oct 31 01:15:55.097740 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:15:55.097711    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/41125f0c-f8ce-416b-92b0-4d6535255a5e-kube-api-access-xcx5h\" (UniqueName: \"kubernetes.io/projected/41125f0c-f8ce-416b-92b0-4d6535255a5e-kube-api-access-xcx5h\") pod \"41125f0c-f8ce-416b-92b0-4d6535255a5e\" (UID: \"41125f0c-f8ce-416b-92b0-4d6535255a5e\") "
Oct 31 01:15:55.097913 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:15:55.097769    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/41125f0c-f8ce-416b-92b0-4d6535255a5e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/41125f0c-f8ce-416b-92b0-4d6535255a5e-workload-certs\") pod \"41125f0c-f8ce-416b-92b0-4d6535255a5e\" (UID: \"41125f0c-f8ce-416b-92b0-4d6535255a5e\") "
Oct 31 01:15:55.097913 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:15:55.097831    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/41125f0c-f8ce-416b-92b0-4d6535255a5e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/41125f0c-f8ce-416b-92b0-4d6535255a5e-workload-socket\") pod \"41125f0c-f8ce-416b-92b0-4d6535255a5e\" (UID: \"41125f0c-f8ce-416b-92b0-4d6535255a5e\") "
Oct 31 01:15:55.097913 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:15:55.097883    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/41125f0c-f8ce-416b-92b0-4d6535255a5e-istio-token\" (UniqueName: \"kubernetes.io/projected/41125f0c-f8ce-416b-92b0-4d6535255a5e-istio-token\") pod \"41125f0c-f8ce-416b-92b0-4d6535255a5e\" (UID: \"41125f0c-f8ce-416b-92b0-4d6535255a5e\") "
Oct 31 01:15:55.098104 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:15:55.097959    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/41125f0c-f8ce-416b-92b0-4d6535255a5e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/41125f0c-f8ce-416b-92b0-4d6535255a5e-istio-envoy\") pod \"41125f0c-f8ce-416b-92b0-4d6535255a5e\" (UID: \"41125f0c-f8ce-416b-92b0-4d6535255a5e\") "
Oct 31 01:15:55.098104 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:15:55.098005    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/41125f0c-f8ce-416b-92b0-4d6535255a5e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/41125f0c-f8ce-416b-92b0-4d6535255a5e-istio-data\") pod \"41125f0c-f8ce-416b-92b0-4d6535255a5e\" (UID: \"41125f0c-f8ce-416b-92b0-4d6535255a5e\") "
Oct 31 01:15:55.098104 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:15:55.098034    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/41125f0c-f8ce-416b-92b0-4d6535255a5e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/41125f0c-f8ce-416b-92b0-4d6535255a5e-istiod-ca-cert\") pod \"41125f0c-f8ce-416b-92b0-4d6535255a5e\" (UID: \"41125f0c-f8ce-416b-92b0-4d6535255a5e\") "
Oct 31 01:15:55.098104 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:15:55.098059    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/41125f0c-f8ce-416b-92b0-4d6535255a5e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/41125f0c-f8ce-416b-92b0-4d6535255a5e-istio-podinfo\") pod \"41125f0c-f8ce-416b-92b0-4d6535255a5e\" (UID: \"41125f0c-f8ce-416b-92b0-4d6535255a5e\") "
Oct 31 01:15:55.098104 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:15:55.098094    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/41125f0c-f8ce-416b-92b0-4d6535255a5e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/41125f0c-f8ce-416b-92b0-4d6535255a5e-credential-socket\") pod \"41125f0c-f8ce-416b-92b0-4d6535255a5e\" (UID: \"41125f0c-f8ce-416b-92b0-4d6535255a5e\") "
Oct 31 01:15:55.098395 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:15:55.098159    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41125f0c-f8ce-416b-92b0-4d6535255a5e-workload-certs" pod "41125f0c-f8ce-416b-92b0-4d6535255a5e" (UID: "41125f0c-f8ce-416b-92b0-4d6535255a5e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:15:55.098395 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:15:55.098174    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41125f0c-f8ce-416b-92b0-4d6535255a5e-workload-socket" pod "41125f0c-f8ce-416b-92b0-4d6535255a5e" (UID: "41125f0c-f8ce-416b-92b0-4d6535255a5e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:15:55.098510 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:15:55.098417    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41125f0c-f8ce-416b-92b0-4d6535255a5e-istio-data" pod "41125f0c-f8ce-416b-92b0-4d6535255a5e" (UID: "41125f0c-f8ce-416b-92b0-4d6535255a5e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:15:55.098564 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:15:55.098507    4964 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/41125f0c-f8ce-416b-92b0-4d6535255a5e-workload-certs\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:15:55.098564 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:15:55.098529    4964 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/41125f0c-f8ce-416b-92b0-4d6535255a5e-workload-socket\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:15:55.098785 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:15:55.098763    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41125f0c-f8ce-416b-92b0-4d6535255a5e-credential-socket" pod "41125f0c-f8ce-416b-92b0-4d6535255a5e" (UID: "41125f0c-f8ce-416b-92b0-4d6535255a5e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:15:55.098927 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:15:55.098894    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/41125f0c-f8ce-416b-92b0-4d6535255a5e-istiod-ca-cert" pod "41125f0c-f8ce-416b-92b0-4d6535255a5e" (UID: "41125f0c-f8ce-416b-92b0-4d6535255a5e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:15:55.100229 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:15:55.100205    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/41125f0c-f8ce-416b-92b0-4d6535255a5e-kube-api-access-xcx5h" pod "41125f0c-f8ce-416b-92b0-4d6535255a5e" (UID: "41125f0c-f8ce-416b-92b0-4d6535255a5e"). InnerVolumeSpecName "kube-api-access-xcx5h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:15:55.100670 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:15:55.100653    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41125f0c-f8ce-416b-92b0-4d6535255a5e-istio-envoy" pod "41125f0c-f8ce-416b-92b0-4d6535255a5e" (UID: "41125f0c-f8ce-416b-92b0-4d6535255a5e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:15:55.101135 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:15:55.101099    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/41125f0c-f8ce-416b-92b0-4d6535255a5e-istio-podinfo" pod "41125f0c-f8ce-416b-92b0-4d6535255a5e" (UID: "41125f0c-f8ce-416b-92b0-4d6535255a5e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 01:15:55.101135 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:15:55.101120    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/41125f0c-f8ce-416b-92b0-4d6535255a5e-istio-token" pod "41125f0c-f8ce-416b-92b0-4d6535255a5e" (UID: "41125f0c-f8ce-416b-92b0-4d6535255a5e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:15:55.165011 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:15:55.164974    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:15:55.199340 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:15:55.199284    4964 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/41125f0c-f8ce-416b-92b0-4d6535255a5e-istio-token\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:15:55.199340 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:15:55.199310    4964 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/41125f0c-f8ce-416b-92b0-4d6535255a5e-istio-envoy\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:15:55.199340 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:15:55.199339    4964 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/41125f0c-f8ce-416b-92b0-4d6535255a5e-istio-data\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:15:55.199553 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:15:55.199351    4964 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/41125f0c-f8ce-416b-92b0-4d6535255a5e-istiod-ca-cert\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:15:55.199553 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:15:55.199368    4964 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/41125f0c-f8ce-416b-92b0-4d6535255a5e-istio-podinfo\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:15:55.199553 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:15:55.199376    4964 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/41125f0c-f8ce-416b-92b0-4d6535255a5e-credential-socket\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:15:55.199553 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:15:55.199385    4964 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xcx5h\" (UniqueName: \"kubernetes.io/projected/41125f0c-f8ce-416b-92b0-4d6535255a5e-kube-api-access-xcx5h\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:15:55.352931 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:15:55.352821    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:15:55.664016 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:15:55.663930    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:15:55.759740 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:15:55.759705    4964 generic.go:356] "Generic (PLEG): container finished" podID="41125f0c-f8ce-416b-92b0-4d6535255a5e" containerID="9eb973ebe3085655bb8f60dc0681c07b56fd7889921fd5fb00bd8d731e256d20" exitCode=0
Oct 31 01:15:55.760141 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:15:55.759753    4964 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-a0c7a371-gw-openshift-default-7857fb8585-vnnng"
Oct 31 01:15:55.760141 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:15:55.759754    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-a0c7a371-gw-openshift-default-7857fb8585-vnnng" event={"ID":"41125f0c-f8ce-416b-92b0-4d6535255a5e","Type":"ContainerDied","Data":"9eb973ebe3085655bb8f60dc0681c07b56fd7889921fd5fb00bd8d731e256d20"}
Oct 31 01:15:55.760141 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:15:55.759877    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-a0c7a371-gw-openshift-default-7857fb8585-vnnng" event={"ID":"41125f0c-f8ce-416b-92b0-4d6535255a5e","Type":"ContainerDied","Data":"bc34b6c8524cf0689b48aa4496d44d0767fffcfb11969d052644855c5fc2d4a8"}
Oct 31 01:15:55.760141 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:15:55.759906    4964 scope.go:122] "RemoveContainer" containerID="9eb973ebe3085655bb8f60dc0681c07b56fd7889921fd5fb00bd8d731e256d20"
Oct 31 01:15:55.778030 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:15:55.778010    4964 scope.go:122] "RemoveContainer" containerID="d9f691765b032b8ac0927bcead1177abb538a4eb46b02af7d90f649c077e5c31"
Oct 31 01:15:55.780663 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:15:55.780642    4964 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="101c024d-b61f-47e5-a863-330ef7b7069d" path="/var/lib/kubelet/pods/101c024d-b61f-47e5-a863-330ef7b7069d/volumes"
Oct 31 01:15:55.786872 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:15:55.786843    4964 scope.go:122] "RemoveContainer" containerID="9eb973ebe3085655bb8f60dc0681c07b56fd7889921fd5fb00bd8d731e256d20"
Oct 31 01:15:55.787225 ip-10-0-133-197 kubenswrapper[4964]: E1031 01:15:55.787201    4964 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9eb973ebe3085655bb8f60dc0681c07b56fd7889921fd5fb00bd8d731e256d20\": container with ID starting with 9eb973ebe3085655bb8f60dc0681c07b56fd7889921fd5fb00bd8d731e256d20 not found: ID does not exist" containerID="9eb973ebe3085655bb8f60dc0681c07b56fd7889921fd5fb00bd8d731e256d20"
Oct 31 01:15:55.787287 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:15:55.787241    4964 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9eb973ebe3085655bb8f60dc0681c07b56fd7889921fd5fb00bd8d731e256d20"} err="failed to get container status \"9eb973ebe3085655bb8f60dc0681c07b56fd7889921fd5fb00bd8d731e256d20\": rpc error: code = NotFound desc = could not find container \"9eb973ebe3085655bb8f60dc0681c07b56fd7889921fd5fb00bd8d731e256d20\": container with ID starting with 9eb973ebe3085655bb8f60dc0681c07b56fd7889921fd5fb00bd8d731e256d20 not found: ID does not exist"
Oct 31 01:15:55.787287 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:15:55.787274    4964 scope.go:122] "RemoveContainer" containerID="d9f691765b032b8ac0927bcead1177abb538a4eb46b02af7d90f649c077e5c31"
Oct 31 01:15:55.787564 ip-10-0-133-197 kubenswrapper[4964]: E1031 01:15:55.787544    4964 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d9f691765b032b8ac0927bcead1177abb538a4eb46b02af7d90f649c077e5c31\": container with ID starting with d9f691765b032b8ac0927bcead1177abb538a4eb46b02af7d90f649c077e5c31 not found: ID does not exist" containerID="d9f691765b032b8ac0927bcead1177abb538a4eb46b02af7d90f649c077e5c31"
Oct 31 01:15:55.787601 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:15:55.787575    4964 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d9f691765b032b8ac0927bcead1177abb538a4eb46b02af7d90f649c077e5c31"} err="failed to get container status \"d9f691765b032b8ac0927bcead1177abb538a4eb46b02af7d90f649c077e5c31\": rpc error: code = NotFound desc = could not find container \"d9f691765b032b8ac0927bcead1177abb538a4eb46b02af7d90f649c077e5c31\": container with ID starting with d9f691765b032b8ac0927bcead1177abb538a4eb46b02af7d90f649c077e5c31 not found: ID does not exist"
Oct 31 01:15:55.923238 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:15:55.923154    4964 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-a0c7a371-gw-openshift-default-7857fb8585-vnnng"]
Oct 31 01:15:55.972950 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:15:55.972921    4964 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-migrate-a0c7a371-gw-openshift-default-7857fb8585-vnnng"]
Oct 31 01:15:57.776258 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:15:57.776224    4964 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="41125f0c-f8ce-416b-92b0-4d6535255a5e" path="/var/lib/kubelet/pods/41125f0c-f8ce-416b-92b0-4d6535255a5e/volumes"
Oct 31 01:15:59.476522 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:15:59.476480    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:16:00.997516 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:16:00.997418    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:16:01.466305 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:16:01.466206    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:16:01.654789 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:16:01.654756    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:16:01.863753 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:16:01.863715    4964 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-a8e157-5597f4b787-gvgv8"]
Oct 31 01:16:01.864203 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:16:01.864188    4964 cpu_manager.go:414] "containerMap: removing container" podUID="9dc20490-124e-40b4-9502-10949d62d531" containerName="maas-api"
Oct 31 01:16:01.864203 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:16:01.864202    4964 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9dc20490-124e-40b4-9502-10949d62d531" containerName="maas-api"
Oct 31 01:16:01.864363 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:16:01.864239    4964 cpu_manager.go:414] "containerMap: removing container" podUID="41125f0c-f8ce-416b-92b0-4d6535255a5e" containerName="istio-proxy"
Oct 31 01:16:01.864363 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:16:01.864244    4964 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="41125f0c-f8ce-416b-92b0-4d6535255a5e" containerName="istio-proxy"
Oct 31 01:16:01.864363 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:16:01.864253    4964 cpu_manager.go:414] "containerMap: removing container" podUID="41125f0c-f8ce-416b-92b0-4d6535255a5e" containerName="istio-proxy"
Oct 31 01:16:01.864363 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:16:01.864259    4964 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="41125f0c-f8ce-416b-92b0-4d6535255a5e" containerName="istio-proxy"
Oct 31 01:16:01.864363 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:16:01.864266    4964 cpu_manager.go:414] "containerMap: removing container" podUID="101c024d-b61f-47e5-a863-330ef7b7069d" containerName="maas-api"
Oct 31 01:16:01.864363 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:16:01.864271    4964 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="101c024d-b61f-47e5-a863-330ef7b7069d" containerName="maas-api"
Oct 31 01:16:01.864363 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:16:01.864278    4964 cpu_manager.go:414] "containerMap: removing container" podUID="b537e3cd-aa63-4840-8ec5-4447d616d56f" containerName="payload-processing"
Oct 31 01:16:01.864363 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:16:01.864283    4964 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b537e3cd-aa63-4840-8ec5-4447d616d56f" containerName="payload-processing"
Oct 31 01:16:01.864363 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:16:01.864359    4964 memory_manager.go:365] "RemoveStaleState removing state" podUID="9dc20490-124e-40b4-9502-10949d62d531" containerName="maas-api"
Oct 31 01:16:01.864792 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:16:01.864373    4964 memory_manager.go:365] "RemoveStaleState removing state" podUID="41125f0c-f8ce-416b-92b0-4d6535255a5e" containerName="istio-proxy"
Oct 31 01:16:01.864792 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:16:01.864379    4964 memory_manager.go:365] "RemoveStaleState removing state" podUID="41125f0c-f8ce-416b-92b0-4d6535255a5e" containerName="istio-proxy"
Oct 31 01:16:01.864792 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:16:01.864384    4964 memory_manager.go:365] "RemoveStaleState removing state" podUID="101c024d-b61f-47e5-a863-330ef7b7069d" containerName="maas-api"
Oct 31 01:16:01.864792 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:16:01.864388    4964 memory_manager.go:365] "RemoveStaleState removing state" podUID="b537e3cd-aa63-4840-8ec5-4447d616d56f" containerName="payload-processing"
Oct 31 01:16:01.869151 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:16:01.869129    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-a8e157-5597f4b787-gvgv8"
Oct 31 01:16:01.929928 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:16:01.928553    4964 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w3-a8e157-dockercfg-8jmcj\""
Oct 31 01:16:01.929928 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:16:01.928567    4964 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w3-a8e157\""
Oct 31 01:16:01.929928 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:16:01.929382    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-a8e157-5597f4b787-gvgv8"]
Oct 31 01:16:01.969908 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:16:01.969867    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3aae9913-7c43-45b0-87c8-bf1da6bf9183-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-a8e157-5597f4b787-gvgv8\" (UID: \"3aae9913-7c43-45b0-87c8-bf1da6bf9183\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-a8e157-5597f4b787-gvgv8"
Oct 31 01:16:01.970086 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:16:01.970011    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vk6hj\" (UniqueName: \"kubernetes.io/projected/3aae9913-7c43-45b0-87c8-bf1da6bf9183-kube-api-access-vk6hj\") pod \"payload-pre-processing-e2e-worker-w3-a8e157-5597f4b787-gvgv8\" (UID: \"3aae9913-7c43-45b0-87c8-bf1da6bf9183\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-a8e157-5597f4b787-gvgv8"
Oct 31 01:16:01.987563 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:16:01.987536    4964 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-a8e157-69ffccd74c-z9zls"]
Oct 31 01:16:01.991357 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:16:01.991308    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-a8e157-69ffccd74c-z9zls"
Oct 31 01:16:02.048491 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:16:02.048463    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-a8e157-69ffccd74c-z9zls"]
Oct 31 01:16:02.070710 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:16:02.070683    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mmnv7\" (UniqueName: \"kubernetes.io/projected/04c0b568-8251-4888-902b-e46c9483d931-kube-api-access-mmnv7\") pod \"payload-processing-e2e-worker-w3-a8e157-69ffccd74c-z9zls\" (UID: \"04c0b568-8251-4888-902b-e46c9483d931\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-a8e157-69ffccd74c-z9zls"
Oct 31 01:16:02.070882 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:16:02.070731    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vk6hj\" (UniqueName: \"kubernetes.io/projected/3aae9913-7c43-45b0-87c8-bf1da6bf9183-kube-api-access-vk6hj\") pod \"payload-pre-processing-e2e-worker-w3-a8e157-5597f4b787-gvgv8\" (UID: \"3aae9913-7c43-45b0-87c8-bf1da6bf9183\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-a8e157-5597f4b787-gvgv8"
Oct 31 01:16:02.070882 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:16:02.070838    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/04c0b568-8251-4888-902b-e46c9483d931-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-a8e157-69ffccd74c-z9zls\" (UID: \"04c0b568-8251-4888-902b-e46c9483d931\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-a8e157-69ffccd74c-z9zls"
Oct 31 01:16:02.071025 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:16:02.071005    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3aae9913-7c43-45b0-87c8-bf1da6bf9183-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-a8e157-5597f4b787-gvgv8\" (UID: \"3aae9913-7c43-45b0-87c8-bf1da6bf9183\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-a8e157-5597f4b787-gvgv8"
Oct 31 01:16:02.071950 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:16:02.071920    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3aae9913-7c43-45b0-87c8-bf1da6bf9183-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-a8e157-5597f4b787-gvgv8\" (UID: \"3aae9913-7c43-45b0-87c8-bf1da6bf9183\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-a8e157-5597f4b787-gvgv8"
Oct 31 01:16:02.128179 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:16:02.128099    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vk6hj\" (UniqueName: \"kubernetes.io/projected/3aae9913-7c43-45b0-87c8-bf1da6bf9183-kube-api-access-vk6hj\") pod \"payload-pre-processing-e2e-worker-w3-a8e157-5597f4b787-gvgv8\" (UID: \"3aae9913-7c43-45b0-87c8-bf1da6bf9183\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-a8e157-5597f4b787-gvgv8"
Oct 31 01:16:02.171785 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:16:02.171752    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mmnv7\" (UniqueName: \"kubernetes.io/projected/04c0b568-8251-4888-902b-e46c9483d931-kube-api-access-mmnv7\") pod \"payload-processing-e2e-worker-w3-a8e157-69ffccd74c-z9zls\" (UID: \"04c0b568-8251-4888-902b-e46c9483d931\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-a8e157-69ffccd74c-z9zls"
Oct 31 01:16:02.171965 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:16:02.171808    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/04c0b568-8251-4888-902b-e46c9483d931-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-a8e157-69ffccd74c-z9zls\" (UID: \"04c0b568-8251-4888-902b-e46c9483d931\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-a8e157-69ffccd74c-z9zls"
Oct 31 01:16:02.172510 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:16:02.172491    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/04c0b568-8251-4888-902b-e46c9483d931-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-a8e157-69ffccd74c-z9zls\" (UID: \"04c0b568-8251-4888-902b-e46c9483d931\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-a8e157-69ffccd74c-z9zls"
Oct 31 01:16:02.182483 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:16:02.182461    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-a8e157-5597f4b787-gvgv8"
Oct 31 01:16:02.229167 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:16:02.229135    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mmnv7\" (UniqueName: \"kubernetes.io/projected/04c0b568-8251-4888-902b-e46c9483d931-kube-api-access-mmnv7\") pod \"payload-processing-e2e-worker-w3-a8e157-69ffccd74c-z9zls\" (UID: \"04c0b568-8251-4888-902b-e46c9483d931\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-a8e157-69ffccd74c-z9zls"
Oct 31 01:16:02.307681 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:16:02.307642    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-a8e157-69ffccd74c-z9zls"
Oct 31 01:16:02.564076 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:16:02.564052    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-a8e157-5597f4b787-gvgv8"]
Oct 31 01:16:02.566049 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:16:02.566016    4964 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3aae9913_7c43_45b0_87c8_bf1da6bf9183.slice/crio-d24d70e0b406d51b504f701389ed3c8ccf8f5f1c0410cc0724c24d9e428f5b94 WatchSource:0}: Error finding container d24d70e0b406d51b504f701389ed3c8ccf8f5f1c0410cc0724c24d9e428f5b94: Status 404 returned error can't find the container with id d24d70e0b406d51b504f701389ed3c8ccf8f5f1c0410cc0724c24d9e428f5b94
Oct 31 01:16:02.621735 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:16:02.621711    4964 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-7164dd00-6dcd9d9bc9-24xtq"]
Oct 31 01:16:02.686133 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:16:02.686114    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-a8e157-69ffccd74c-z9zls"]
Oct 31 01:16:02.801681 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:16:02.801642    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-a8e157-69ffccd74c-z9zls" event={"ID":"04c0b568-8251-4888-902b-e46c9483d931","Type":"ContainerStarted","Data":"f42e6fe1b20bd1dc35351757e16228ae4d7b8cd79084274a94a12adfd3b29d95"}
Oct 31 01:16:02.801872 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:16:02.801714    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w3-a8e157-69ffccd74c-z9zls"
Oct 31 01:16:02.801872 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:16:02.801737    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-a8e157-69ffccd74c-z9zls" event={"ID":"04c0b568-8251-4888-902b-e46c9483d931","Type":"ContainerStarted","Data":"f8bba7183db9fd1ed6d3956e12f71f0d1ec43911e6db02d9914a98216cbba6ec"}
Oct 31 01:16:02.803264 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:16:02.803239    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-a8e157-5597f4b787-gvgv8" event={"ID":"3aae9913-7c43-45b0-87c8-bf1da6bf9183","Type":"ContainerStarted","Data":"47627fe7ac47a10530c752db69066a5f7e04a9952e5cd181616361006c1a9f88"}
Oct 31 01:16:02.803417 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:16:02.803276    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-a8e157-5597f4b787-gvgv8" event={"ID":"3aae9913-7c43-45b0-87c8-bf1da6bf9183","Type":"ContainerStarted","Data":"d24d70e0b406d51b504f701389ed3c8ccf8f5f1c0410cc0724c24d9e428f5b94"}
Oct 31 01:16:02.803417 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:16:02.803348    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-a8e157-5597f4b787-gvgv8"
Oct 31 01:16:02.920005 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:16:02.919909    4964 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w3-a8e157-69ffccd74c-z9zls" podStartSLOduration=1.919895941 podStartE2EDuration="1.919895941s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:16:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:16:02.917304261 +0000 UTC m=+731.393870641" watchObservedRunningTime="2026-10-31 01:16:02.919895941 +0000 UTC m=+731.396462321"
Oct 31 01:16:02.926734 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:16:02.926698    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:16:03.037085 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:16:03.037029    4964 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-a8e157-5597f4b787-gvgv8" podStartSLOduration=2.037015574 podStartE2EDuration="2.037015574s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:16:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:16:03.03457047 +0000 UTC m=+731.511136861" watchObservedRunningTime="2026-10-31 01:16:03.037015574 +0000 UTC m=+731.513581953"
Oct 31 01:16:03.110089 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:16:03.109853    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:16:05.072365 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:16:05.072285    4964 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-b9656f65-755b48cd75-lpd6d"]
Oct 31 01:16:05.372945 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:16:05.372858    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:16:05.565412 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:16:05.565377    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:16:07.396519 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:16:07.396483    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:16:07.622248 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:16:07.622195    4964 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-7164dd00-6dcd9d9bc9-24xtq" podUID="e044e488-b8db-46d9-afe1-b2a39b9a10c1" containerName="payload-pre-processing" containerID="cri-o://c844b0d7595c33cf569c97428c7829957e1df65a3701bea72a5f801822f4535d" gracePeriod=25
Oct 31 01:16:07.833659 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:16:07.833629    4964 generic.go:356] "Generic (PLEG): container finished" podID="e044e488-b8db-46d9-afe1-b2a39b9a10c1" containerID="c844b0d7595c33cf569c97428c7829957e1df65a3701bea72a5f801822f4535d" exitCode=0
Oct 31 01:16:07.833852 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:16:07.833697    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-7164dd00-6dcd9d9bc9-24xtq" event={"ID":"e044e488-b8db-46d9-afe1-b2a39b9a10c1","Type":"ContainerDied","Data":"c844b0d7595c33cf569c97428c7829957e1df65a3701bea72a5f801822f4535d"}
Oct 31 01:16:07.892102 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:16:07.892077    4964 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-7164dd00-6dcd9d9bc9-24xtq"
Oct 31 01:16:08.030971 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:16:08.030882    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e044e488-b8db-46d9-afe1-b2a39b9a10c1-kube-api-access-l4pcc\" (UniqueName: \"kubernetes.io/projected/e044e488-b8db-46d9-afe1-b2a39b9a10c1-kube-api-access-l4pcc\") pod \"e044e488-b8db-46d9-afe1-b2a39b9a10c1\" (UID: \"e044e488-b8db-46d9-afe1-b2a39b9a10c1\") "
Oct 31 01:16:08.031130 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:16:08.031006    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e044e488-b8db-46d9-afe1-b2a39b9a10c1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e044e488-b8db-46d9-afe1-b2a39b9a10c1-plugins-config-volume\") pod \"e044e488-b8db-46d9-afe1-b2a39b9a10c1\" (UID: \"e044e488-b8db-46d9-afe1-b2a39b9a10c1\") "
Oct 31 01:16:08.031449 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:16:08.031423    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e044e488-b8db-46d9-afe1-b2a39b9a10c1-plugins-config-volume" pod "e044e488-b8db-46d9-afe1-b2a39b9a10c1" (UID: "e044e488-b8db-46d9-afe1-b2a39b9a10c1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:16:08.033629 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:16:08.033596    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e044e488-b8db-46d9-afe1-b2a39b9a10c1-kube-api-access-l4pcc" pod "e044e488-b8db-46d9-afe1-b2a39b9a10c1" (UID: "e044e488-b8db-46d9-afe1-b2a39b9a10c1"). InnerVolumeSpecName "kube-api-access-l4pcc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:16:08.132676 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:16:08.132645    4964 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l4pcc\" (UniqueName: \"kubernetes.io/projected/e044e488-b8db-46d9-afe1-b2a39b9a10c1-kube-api-access-l4pcc\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:16:08.132676 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:16:08.132672    4964 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e044e488-b8db-46d9-afe1-b2a39b9a10c1-plugins-config-volume\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:16:08.840989 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:16:08.840954    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-7164dd00-6dcd9d9bc9-24xtq" event={"ID":"e044e488-b8db-46d9-afe1-b2a39b9a10c1","Type":"ContainerDied","Data":"393e6bb4cc1397d88fa5e449d6c331ebd839eca2c7d730c317bd4b31dbd5035d"}
Oct 31 01:16:08.840989 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:16:08.840971    4964 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-7164dd00-6dcd9d9bc9-24xtq"
Oct 31 01:16:08.841595 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:16:08.841023    4964 scope.go:122] "RemoveContainer" containerID="c844b0d7595c33cf569c97428c7829957e1df65a3701bea72a5f801822f4535d"
Oct 31 01:16:09.009847 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:16:09.007019    4964 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-7164dd00-6dcd9d9bc9-24xtq"]
Oct 31 01:16:09.053555 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:16:09.053520    4964 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-7164dd00-6dcd9d9bc9-24xtq"]
Oct 31 01:16:09.776013 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:16:09.775979    4964 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e044e488-b8db-46d9-afe1-b2a39b9a10c1" path="/var/lib/kubelet/pods/e044e488-b8db-46d9-afe1-b2a39b9a10c1/volumes"
Oct 31 01:16:10.072849 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:16:10.072755    4964 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-derive-b9656f65-755b48cd75-lpd6d" podUID="40d9e5d2-76da-4a1f-9ac9-78ab76bc653e" containerName="payload-pre-processing" containerID="cri-o://1983d52ece9992bcd1f2c5819ed17e18d03a3c20f37bcca587cf18a7713069e7" gracePeriod=25
Oct 31 01:16:10.361423 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:16:10.361297    4964 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-b9656f65-755b48cd75-lpd6d"
Oct 31 01:16:10.458155 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:16:10.458127    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/40d9e5d2-76da-4a1f-9ac9-78ab76bc653e-kube-api-access-99682\" (UniqueName: \"kubernetes.io/projected/40d9e5d2-76da-4a1f-9ac9-78ab76bc653e-kube-api-access-99682\") pod \"40d9e5d2-76da-4a1f-9ac9-78ab76bc653e\" (UID: \"40d9e5d2-76da-4a1f-9ac9-78ab76bc653e\") "
Oct 31 01:16:10.458371 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:16:10.458226    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/40d9e5d2-76da-4a1f-9ac9-78ab76bc653e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/40d9e5d2-76da-4a1f-9ac9-78ab76bc653e-plugins-config-volume\") pod \"40d9e5d2-76da-4a1f-9ac9-78ab76bc653e\" (UID: \"40d9e5d2-76da-4a1f-9ac9-78ab76bc653e\") "
Oct 31 01:16:10.458597 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:16:10.458575    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/40d9e5d2-76da-4a1f-9ac9-78ab76bc653e-plugins-config-volume" pod "40d9e5d2-76da-4a1f-9ac9-78ab76bc653e" (UID: "40d9e5d2-76da-4a1f-9ac9-78ab76bc653e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:16:10.460249 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:16:10.460229    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/40d9e5d2-76da-4a1f-9ac9-78ab76bc653e-kube-api-access-99682" pod "40d9e5d2-76da-4a1f-9ac9-78ab76bc653e" (UID: "40d9e5d2-76da-4a1f-9ac9-78ab76bc653e"). InnerVolumeSpecName "kube-api-access-99682". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:16:10.559816 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:16:10.559786    4964 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-99682\" (UniqueName: \"kubernetes.io/projected/40d9e5d2-76da-4a1f-9ac9-78ab76bc653e-kube-api-access-99682\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:16:10.559816 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:16:10.559811    4964 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/40d9e5d2-76da-4a1f-9ac9-78ab76bc653e-plugins-config-volume\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:16:10.854473 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:16:10.854436    4964 generic.go:356] "Generic (PLEG): container finished" podID="40d9e5d2-76da-4a1f-9ac9-78ab76bc653e" containerID="1983d52ece9992bcd1f2c5819ed17e18d03a3c20f37bcca587cf18a7713069e7" exitCode=0
Oct 31 01:16:10.854647 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:16:10.854499    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-b9656f65-755b48cd75-lpd6d" event={"ID":"40d9e5d2-76da-4a1f-9ac9-78ab76bc653e","Type":"ContainerDied","Data":"1983d52ece9992bcd1f2c5819ed17e18d03a3c20f37bcca587cf18a7713069e7"}
Oct 31 01:16:10.854647 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:16:10.854511    4964 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-b9656f65-755b48cd75-lpd6d"
Oct 31 01:16:10.854647 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:16:10.854546    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-b9656f65-755b48cd75-lpd6d" event={"ID":"40d9e5d2-76da-4a1f-9ac9-78ab76bc653e","Type":"ContainerDied","Data":"35795743f28a7a1ab43d93558c2d215ee362cb28997de39da8ed5aa85350baeb"}
Oct 31 01:16:10.854647 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:16:10.854569    4964 scope.go:122] "RemoveContainer" containerID="1983d52ece9992bcd1f2c5819ed17e18d03a3c20f37bcca587cf18a7713069e7"
Oct 31 01:16:10.864804 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:16:10.864785    4964 scope.go:122] "RemoveContainer" containerID="1983d52ece9992bcd1f2c5819ed17e18d03a3c20f37bcca587cf18a7713069e7"
Oct 31 01:16:10.865084 ip-10-0-133-197 kubenswrapper[4964]: E1031 01:16:10.865067    4964 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1983d52ece9992bcd1f2c5819ed17e18d03a3c20f37bcca587cf18a7713069e7\": container with ID starting with 1983d52ece9992bcd1f2c5819ed17e18d03a3c20f37bcca587cf18a7713069e7 not found: ID does not exist" containerID="1983d52ece9992bcd1f2c5819ed17e18d03a3c20f37bcca587cf18a7713069e7"
Oct 31 01:16:10.865132 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:16:10.865094    4964 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1983d52ece9992bcd1f2c5819ed17e18d03a3c20f37bcca587cf18a7713069e7"} err="failed to get container status \"1983d52ece9992bcd1f2c5819ed17e18d03a3c20f37bcca587cf18a7713069e7\": rpc error: code = NotFound desc = could not find container \"1983d52ece9992bcd1f2c5819ed17e18d03a3c20f37bcca587cf18a7713069e7\": container with ID starting with 1983d52ece9992bcd1f2c5819ed17e18d03a3c20f37bcca587cf18a7713069e7 not found: ID does not exist"
Oct 31 01:16:11.018622 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:16:11.018587    4964 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-b9656f65-755b48cd75-lpd6d"]
Oct 31 01:16:11.024579 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:16:11.024555    4964 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-b9656f65-755b48cd75-lpd6d"]
Oct 31 01:16:11.236603 ip-10-0-133-197 kubenswrapper[4964]: E1031 01:16:11.236570    4964 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q56x5" containerName="extractor"
Oct 31 01:16:11.777488 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:16:11.777457    4964 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="40d9e5d2-76da-4a1f-9ac9-78ab76bc653e" path="/var/lib/kubelet/pods/40d9e5d2-76da-4a1f-9ac9-78ab76bc653e/volumes"
Oct 31 01:16:12.151938 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:16:12.151860    4964 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-a8e157-584855c7c8-b66mk"]
Oct 31 01:16:12.152347 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:16:12.152333    4964 cpu_manager.go:414] "containerMap: removing container" podUID="40d9e5d2-76da-4a1f-9ac9-78ab76bc653e" containerName="payload-pre-processing"
Oct 31 01:16:12.152399 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:16:12.152349    4964 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="40d9e5d2-76da-4a1f-9ac9-78ab76bc653e" containerName="payload-pre-processing"
Oct 31 01:16:12.152399 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:16:12.152366    4964 cpu_manager.go:414] "containerMap: removing container" podUID="e044e488-b8db-46d9-afe1-b2a39b9a10c1" containerName="payload-pre-processing"
Oct 31 01:16:12.152399 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:16:12.152371    4964 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e044e488-b8db-46d9-afe1-b2a39b9a10c1" containerName="payload-pre-processing"
Oct 31 01:16:12.152494 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:16:12.152453    4964 memory_manager.go:365] "RemoveStaleState removing state" podUID="40d9e5d2-76da-4a1f-9ac9-78ab76bc653e" containerName="payload-pre-processing"
Oct 31 01:16:12.152494 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:16:12.152463    4964 memory_manager.go:365] "RemoveStaleState removing state" podUID="e044e488-b8db-46d9-afe1-b2a39b9a10c1" containerName="payload-pre-processing"
Oct 31 01:16:12.156804 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:16:12.156789    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-a8e157-584855c7c8-b66mk"
Oct 31 01:16:12.260563 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:16:12.260516    4964 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w3-a8e157-serving-cert\""
Oct 31 01:16:12.263158 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:16:12.263131    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-a8e157-584855c7c8-b66mk"]
Oct 31 01:16:12.276041 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:16:12.276018    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e8e369b7-335b-4ac4-8e8a-3879cf430b76-metrics-certs\") pod \"maas-api-e2e-worker-w3-a8e157-584855c7c8-b66mk\" (UID: \"e8e369b7-335b-4ac4-8e8a-3879cf430b76\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-a8e157-584855c7c8-b66mk"
Oct 31 01:16:12.276149 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:16:12.276066    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e8e369b7-335b-4ac4-8e8a-3879cf430b76-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-a8e157-584855c7c8-b66mk\" (UID: \"e8e369b7-335b-4ac4-8e8a-3879cf430b76\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-a8e157-584855c7c8-b66mk"
Oct 31 01:16:12.276199 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:16:12.276152    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e8e369b7-335b-4ac4-8e8a-3879cf430b76-maas-api-tls\") pod \"maas-api-e2e-worker-w3-a8e157-584855c7c8-b66mk\" (UID: \"e8e369b7-335b-4ac4-8e8a-3879cf430b76\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-a8e157-584855c7c8-b66mk"
Oct 31 01:16:12.276267 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:16:12.276253    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rmzxn\" (UniqueName: \"kubernetes.io/projected/e8e369b7-335b-4ac4-8e8a-3879cf430b76-kube-api-access-rmzxn\") pod \"maas-api-e2e-worker-w3-a8e157-584855c7c8-b66mk\" (UID: \"e8e369b7-335b-4ac4-8e8a-3879cf430b76\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-a8e157-584855c7c8-b66mk"
Oct 31 01:16:12.376908 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:16:12.376879    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e8e369b7-335b-4ac4-8e8a-3879cf430b76-metrics-certs\") pod \"maas-api-e2e-worker-w3-a8e157-584855c7c8-b66mk\" (UID: \"e8e369b7-335b-4ac4-8e8a-3879cf430b76\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-a8e157-584855c7c8-b66mk"
Oct 31 01:16:12.377081 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:16:12.376934    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e8e369b7-335b-4ac4-8e8a-3879cf430b76-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-a8e157-584855c7c8-b66mk\" (UID: \"e8e369b7-335b-4ac4-8e8a-3879cf430b76\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-a8e157-584855c7c8-b66mk"
Oct 31 01:16:12.377081 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:16:12.377059    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e8e369b7-335b-4ac4-8e8a-3879cf430b76-maas-api-tls\") pod \"maas-api-e2e-worker-w3-a8e157-584855c7c8-b66mk\" (UID: \"e8e369b7-335b-4ac4-8e8a-3879cf430b76\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-a8e157-584855c7c8-b66mk"
Oct 31 01:16:12.377172 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:16:12.377156    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rmzxn\" (UniqueName: \"kubernetes.io/projected/e8e369b7-335b-4ac4-8e8a-3879cf430b76-kube-api-access-rmzxn\") pod \"maas-api-e2e-worker-w3-a8e157-584855c7c8-b66mk\" (UID: \"e8e369b7-335b-4ac4-8e8a-3879cf430b76\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-a8e157-584855c7c8-b66mk"
Oct 31 01:16:12.377242 ip-10-0-133-197 kubenswrapper[4964]: E1031 01:16:12.377224    4964 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w3-a8e157-serving-cert: secret "maas-api-e2e-worker-w3-a8e157-serving-cert" not found
Oct 31 01:16:12.377310 ip-10-0-133-197 kubenswrapper[4964]: E1031 01:16:12.377297    4964 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e8e369b7-335b-4ac4-8e8a-3879cf430b76-maas-api-tls podName:e8e369b7-335b-4ac4-8e8a-3879cf430b76 nodeName:}" failed. No retries permitted until 2026-10-31 01:16:12.877277233 +0000 UTC m=+741.353843606 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/e8e369b7-335b-4ac4-8e8a-3879cf430b76-maas-api-tls") pod "maas-api-e2e-worker-w3-a8e157-584855c7c8-b66mk" (UID: "e8e369b7-335b-4ac4-8e8a-3879cf430b76") : secret "maas-api-e2e-worker-w3-a8e157-serving-cert" not found
Oct 31 01:16:12.377655 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:16:12.377639    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e8e369b7-335b-4ac4-8e8a-3879cf430b76-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-a8e157-584855c7c8-b66mk\" (UID: \"e8e369b7-335b-4ac4-8e8a-3879cf430b76\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-a8e157-584855c7c8-b66mk"
Oct 31 01:16:12.379535 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:16:12.379513    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e8e369b7-335b-4ac4-8e8a-3879cf430b76-metrics-certs\") pod \"maas-api-e2e-worker-w3-a8e157-584855c7c8-b66mk\" (UID: \"e8e369b7-335b-4ac4-8e8a-3879cf430b76\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-a8e157-584855c7c8-b66mk"
Oct 31 01:16:12.399066 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:16:12.399033    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:16:12.434211 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:16:12.434139    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rmzxn\" (UniqueName: \"kubernetes.io/projected/e8e369b7-335b-4ac4-8e8a-3879cf430b76-kube-api-access-rmzxn\") pod \"maas-api-e2e-worker-w3-a8e157-584855c7c8-b66mk\" (UID: \"e8e369b7-335b-4ac4-8e8a-3879cf430b76\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-a8e157-584855c7c8-b66mk"
Oct 31 01:16:12.593846 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:16:12.593558    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:16:12.774745 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:16:12.774710    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:16:12.881836 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:16:12.881807    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e8e369b7-335b-4ac4-8e8a-3879cf430b76-maas-api-tls\") pod \"maas-api-e2e-worker-w3-a8e157-584855c7c8-b66mk\" (UID: \"e8e369b7-335b-4ac4-8e8a-3879cf430b76\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-a8e157-584855c7c8-b66mk"
Oct 31 01:16:12.884456 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:16:12.884428    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e8e369b7-335b-4ac4-8e8a-3879cf430b76-maas-api-tls\") pod \"maas-api-e2e-worker-w3-a8e157-584855c7c8-b66mk\" (UID: \"e8e369b7-335b-4ac4-8e8a-3879cf430b76\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-a8e157-584855c7c8-b66mk"
Oct 31 01:16:13.038481 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:16:13.038386    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:16:13.069996 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:16:13.069952    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-a8e157-584855c7c8-b66mk"
Oct 31 01:16:13.402683 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:16:13.402597    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:16:13.451354 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:16:13.451305    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-a8e157-584855c7c8-b66mk"]
Oct 31 01:16:13.453666 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:16:13.453636    4964 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8e369b7_335b_4ac4_8e8a_3879cf430b76.slice/crio-385d1773da1e9db42be37c4660d0a86e0118c376edfb629c5294e43826c9c327 WatchSource:0}: Error finding container 385d1773da1e9db42be37c4660d0a86e0118c376edfb629c5294e43826c9c327: Status 404 returned error can't find the container with id 385d1773da1e9db42be37c4660d0a86e0118c376edfb629c5294e43826c9c327
Oct 31 01:16:13.690148 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:16:13.690068    4964 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-b9656f65-openshift-default-c6f497c7-r4pr4"]
Oct 31 01:16:13.690339 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:16:13.690287    4964 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-b9656f65-openshift-default-c6f497c7-r4pr4" podUID="9b23eb8c-e963-42d6-a1f2-61e474aaa829" containerName="istio-proxy" containerID="cri-o://49327c69dd44d704eb65dede3c38075873ae1c118a54578f04c659aba237ac9e" gracePeriod=30
Oct 31 01:16:13.775098 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:16:13.775047    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:16:13.811434 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:16:13.811404    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-a8e157-5597f4b787-gvgv8"
Oct 31 01:16:13.811804 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:16:13.811783    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w3-a8e157-69ffccd74c-z9zls"
Oct 31 01:16:13.874260 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:16:13.874231    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-a8e157-584855c7c8-b66mk" event={"ID":"e8e369b7-335b-4ac4-8e8a-3879cf430b76","Type":"ContainerStarted","Data":"385d1773da1e9db42be37c4660d0a86e0118c376edfb629c5294e43826c9c327"}
Oct 31 01:16:14.607379 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:16:14.607291    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:16:14.882242 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:16:14.882124    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-a8e157-584855c7c8-b66mk" event={"ID":"e8e369b7-335b-4ac4-8e8a-3879cf430b76","Type":"ContainerStarted","Data":"f9f9df7866ba5e58f49b0fdc6a4e6d3b7fba4aa69a89d73206b6f5419aa92268"}
Oct 31 01:16:14.882427 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:16:14.882343    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-a8e157-584855c7c8-b66mk"
Oct 31 01:16:14.997506 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:16:14.997439    4964 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-a8e157-584855c7c8-b66mk" podStartSLOduration=2.151729066 podStartE2EDuration="2.997418232s" totalImagesPullingTime="845.689166ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:16:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:16:14.99585978 +0000 UTC m=+743.472426162" watchObservedRunningTime="2026-10-31 01:16:14.997418232 +0000 UTC m=+743.473984614"
Oct 31 01:16:15.188665 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:16:15.188585    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:16:15.365502 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:16:15.365469    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:16:18.743941 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:16:18.743910    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:16:18.909911 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:16:18.909874    4964 generic.go:356] "Generic (PLEG): container finished" podID="9b23eb8c-e963-42d6-a1f2-61e474aaa829" containerID="49327c69dd44d704eb65dede3c38075873ae1c118a54578f04c659aba237ac9e" exitCode=0
Oct 31 01:16:18.910089 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:16:18.909961    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-b9656f65-openshift-default-c6f497c7-r4pr4" event={"ID":"9b23eb8c-e963-42d6-a1f2-61e474aaa829","Type":"ContainerDied","Data":"49327c69dd44d704eb65dede3c38075873ae1c118a54578f04c659aba237ac9e"}
Oct 31 01:16:18.910089 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:16:18.910024    4964 scope.go:122] "RemoveContainer" containerID="c33cd5188f07608e3088a7c08784ba8e02b959608c559196365e86d4e5589a7d"
Oct 31 01:16:18.964343 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:16:18.964265    4964 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-b9656f65-openshift-default-c6f497c7-r4pr4"
Oct 31 01:16:19.056841 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:16:19.056805    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9b23eb8c-e963-42d6-a1f2-61e474aaa829-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b23eb8c-e963-42d6-a1f2-61e474aaa829-workload-socket\") pod \"9b23eb8c-e963-42d6-a1f2-61e474aaa829\" (UID: \"9b23eb8c-e963-42d6-a1f2-61e474aaa829\") "
Oct 31 01:16:19.056841 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:16:19.056840    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9b23eb8c-e963-42d6-a1f2-61e474aaa829-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9b23eb8c-e963-42d6-a1f2-61e474aaa829-istio-data\") pod \"9b23eb8c-e963-42d6-a1f2-61e474aaa829\" (UID: \"9b23eb8c-e963-42d6-a1f2-61e474aaa829\") "
Oct 31 01:16:19.057098 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:16:19.056948    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9b23eb8c-e963-42d6-a1f2-61e474aaa829-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b23eb8c-e963-42d6-a1f2-61e474aaa829-credential-socket\") pod \"9b23eb8c-e963-42d6-a1f2-61e474aaa829\" (UID: \"9b23eb8c-e963-42d6-a1f2-61e474aaa829\") "
Oct 31 01:16:19.057098 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:16:19.056985    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9b23eb8c-e963-42d6-a1f2-61e474aaa829-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9b23eb8c-e963-42d6-a1f2-61e474aaa829-istiod-ca-cert\") pod \"9b23eb8c-e963-42d6-a1f2-61e474aaa829\" (UID: \"9b23eb8c-e963-42d6-a1f2-61e474aaa829\") "
Oct 31 01:16:19.057098 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:16:19.057050    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9b23eb8c-e963-42d6-a1f2-61e474aaa829-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9b23eb8c-e963-42d6-a1f2-61e474aaa829-istio-podinfo\") pod \"9b23eb8c-e963-42d6-a1f2-61e474aaa829\" (UID: \"9b23eb8c-e963-42d6-a1f2-61e474aaa829\") "
Oct 31 01:16:19.057098 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:16:19.057091    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9b23eb8c-e963-42d6-a1f2-61e474aaa829-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9b23eb8c-e963-42d6-a1f2-61e474aaa829-workload-certs\") pod \"9b23eb8c-e963-42d6-a1f2-61e474aaa829\" (UID: \"9b23eb8c-e963-42d6-a1f2-61e474aaa829\") "
Oct 31 01:16:19.057295 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:16:19.057113    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b23eb8c-e963-42d6-a1f2-61e474aaa829-workload-socket" pod "9b23eb8c-e963-42d6-a1f2-61e474aaa829" (UID: "9b23eb8c-e963-42d6-a1f2-61e474aaa829"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:16:19.057295 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:16:19.057148    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b23eb8c-e963-42d6-a1f2-61e474aaa829-istio-data" pod "9b23eb8c-e963-42d6-a1f2-61e474aaa829" (UID: "9b23eb8c-e963-42d6-a1f2-61e474aaa829"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:16:19.057295 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:16:19.057160    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9b23eb8c-e963-42d6-a1f2-61e474aaa829-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9b23eb8c-e963-42d6-a1f2-61e474aaa829-istio-envoy\") pod \"9b23eb8c-e963-42d6-a1f2-61e474aaa829\" (UID: \"9b23eb8c-e963-42d6-a1f2-61e474aaa829\") "
Oct 31 01:16:19.057295 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:16:19.057277    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9b23eb8c-e963-42d6-a1f2-61e474aaa829-istio-token\" (UniqueName: \"kubernetes.io/projected/9b23eb8c-e963-42d6-a1f2-61e474aaa829-istio-token\") pod \"9b23eb8c-e963-42d6-a1f2-61e474aaa829\" (UID: \"9b23eb8c-e963-42d6-a1f2-61e474aaa829\") "
Oct 31 01:16:19.057543 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:16:19.057273    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b23eb8c-e963-42d6-a1f2-61e474aaa829-credential-socket" pod "9b23eb8c-e963-42d6-a1f2-61e474aaa829" (UID: "9b23eb8c-e963-42d6-a1f2-61e474aaa829"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:16:19.057543 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:16:19.057339    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9b23eb8c-e963-42d6-a1f2-61e474aaa829-kube-api-access-rqb5p\" (UniqueName: \"kubernetes.io/projected/9b23eb8c-e963-42d6-a1f2-61e474aaa829-kube-api-access-rqb5p\") pod \"9b23eb8c-e963-42d6-a1f2-61e474aaa829\" (UID: \"9b23eb8c-e963-42d6-a1f2-61e474aaa829\") "
Oct 31 01:16:19.057543 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:16:19.057392    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b23eb8c-e963-42d6-a1f2-61e474aaa829-workload-certs" pod "9b23eb8c-e963-42d6-a1f2-61e474aaa829" (UID: "9b23eb8c-e963-42d6-a1f2-61e474aaa829"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:16:19.057543 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:16:19.057492    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9b23eb8c-e963-42d6-a1f2-61e474aaa829-istiod-ca-cert" pod "9b23eb8c-e963-42d6-a1f2-61e474aaa829" (UID: "9b23eb8c-e963-42d6-a1f2-61e474aaa829"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:16:19.057781 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:16:19.057744    4964 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9b23eb8c-e963-42d6-a1f2-61e474aaa829-workload-certs\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:16:19.057781 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:16:19.057768    4964 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b23eb8c-e963-42d6-a1f2-61e474aaa829-workload-socket\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:16:19.057882 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:16:19.057782    4964 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9b23eb8c-e963-42d6-a1f2-61e474aaa829-istio-data\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:16:19.057882 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:16:19.057795    4964 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b23eb8c-e963-42d6-a1f2-61e474aaa829-credential-socket\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:16:19.057882 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:16:19.057809    4964 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9b23eb8c-e963-42d6-a1f2-61e474aaa829-istiod-ca-cert\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:16:19.059512 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:16:19.059481    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b23eb8c-e963-42d6-a1f2-61e474aaa829-istio-envoy" pod "9b23eb8c-e963-42d6-a1f2-61e474aaa829" (UID: "9b23eb8c-e963-42d6-a1f2-61e474aaa829"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:16:19.059643 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:16:19.059557    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b23eb8c-e963-42d6-a1f2-61e474aaa829-kube-api-access-rqb5p" pod "9b23eb8c-e963-42d6-a1f2-61e474aaa829" (UID: "9b23eb8c-e963-42d6-a1f2-61e474aaa829"). InnerVolumeSpecName "kube-api-access-rqb5p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:16:19.059948 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:16:19.059924    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9b23eb8c-e963-42d6-a1f2-61e474aaa829-istio-podinfo" pod "9b23eb8c-e963-42d6-a1f2-61e474aaa829" (UID: "9b23eb8c-e963-42d6-a1f2-61e474aaa829"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 01:16:19.059948 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:16:19.059936    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b23eb8c-e963-42d6-a1f2-61e474aaa829-istio-token" pod "9b23eb8c-e963-42d6-a1f2-61e474aaa829" (UID: "9b23eb8c-e963-42d6-a1f2-61e474aaa829"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:16:19.137655 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:16:19.137542    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:16:19.158853 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:16:19.158818    4964 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9b23eb8c-e963-42d6-a1f2-61e474aaa829-istio-token\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:16:19.158853 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:16:19.158848    4964 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rqb5p\" (UniqueName: \"kubernetes.io/projected/9b23eb8c-e963-42d6-a1f2-61e474aaa829-kube-api-access-rqb5p\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:16:19.158853 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:16:19.158858    4964 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9b23eb8c-e963-42d6-a1f2-61e474aaa829-istio-podinfo\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:16:19.159168 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:16:19.158871    4964 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9b23eb8c-e963-42d6-a1f2-61e474aaa829-istio-envoy\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:16:19.321733 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:16:19.321694    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:16:19.720830 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:16:19.720799    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:16:19.917486 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:16:19.917451    4964 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-b9656f65-openshift-default-c6f497c7-r4pr4"
Oct 31 01:16:19.917944 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:16:19.917493    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-b9656f65-openshift-default-c6f497c7-r4pr4" event={"ID":"9b23eb8c-e963-42d6-a1f2-61e474aaa829","Type":"ContainerDied","Data":"43c627ee2d2b3d3a77b8821b66f03804fc3505ad854f3ddd82f56654ca7fa68f"}
Oct 31 01:16:19.917944 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:16:19.917552    4964 scope.go:122] "RemoveContainer" containerID="49327c69dd44d704eb65dede3c38075873ae1c118a54578f04c659aba237ac9e"
Oct 31 01:16:19.964329 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:16:19.964281    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:16:20.138628 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:16:20.138547    4964 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-b9656f65-openshift-default-c6f497c7-r4pr4"]
Oct 31 01:16:20.189204 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:16:20.189092    4964 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-b9656f65-openshift-default-c6f497c7-r4pr4"]
Oct 31 01:16:20.893025 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:16:20.892999    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-a8e157-584855c7c8-b66mk"
Oct 31 01:16:20.938645 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:16:20.938584    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:16:21.777112 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:16:21.777068    4964 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9b23eb8c-e963-42d6-a1f2-61e474aaa829" path="/var/lib/kubelet/pods/9b23eb8c-e963-42d6-a1f2-61e474aaa829/volumes"
Oct 31 01:16:21.792236 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:16:21.792209    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:16:22.318349 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:16:22.318279    4964 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-k5bkg"]
Oct 31 01:16:22.318888 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:16:22.318872    4964 cpu_manager.go:414] "containerMap: removing container" podUID="9b23eb8c-e963-42d6-a1f2-61e474aaa829" containerName="istio-proxy"
Oct 31 01:16:22.318965 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:16:22.318889    4964 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9b23eb8c-e963-42d6-a1f2-61e474aaa829" containerName="istio-proxy"
Oct 31 01:16:22.318965 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:16:22.318901    4964 cpu_manager.go:414] "containerMap: removing container" podUID="9b23eb8c-e963-42d6-a1f2-61e474aaa829" containerName="istio-proxy"
Oct 31 01:16:22.318965 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:16:22.318907    4964 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9b23eb8c-e963-42d6-a1f2-61e474aaa829" containerName="istio-proxy"
Oct 31 01:16:22.319076 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:16:22.318986    4964 memory_manager.go:365] "RemoveStaleState removing state" podUID="9b23eb8c-e963-42d6-a1f2-61e474aaa829" containerName="istio-proxy"
Oct 31 01:16:22.319076 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:16:22.318995    4964 memory_manager.go:365] "RemoveStaleState removing state" podUID="9b23eb8c-e963-42d6-a1f2-61e474aaa829" containerName="istio-proxy"
Oct 31 01:16:22.323947 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:16:22.323926    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-k5bkg"
Oct 31 01:16:22.376432 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:16:22.376404    4964 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-8f6b5b-openshift-default-dockercfg-6t9gt\""
Oct 31 01:16:22.429706 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:16:22.429659    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-k5bkg"]
Oct 31 01:16:22.449057 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:16:22.448982    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:16:22.492192 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:16:22.492154    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7855f\" (UniqueName: \"kubernetes.io/projected/197445f1-d262-48ee-87c5-7354132a69ba-kube-api-access-7855f\") pod \"e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-k5bkg\" (UID: \"197445f1-d262-48ee-87c5-7354132a69ba\") " pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-k5bkg"
Oct 31 01:16:22.492408 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:16:22.492201    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/197445f1-d262-48ee-87c5-7354132a69ba-istiod-ca-cert\") pod \"e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-k5bkg\" (UID: \"197445f1-d262-48ee-87c5-7354132a69ba\") " pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-k5bkg"
Oct 31 01:16:22.492408 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:16:22.492366    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/197445f1-d262-48ee-87c5-7354132a69ba-istio-token\") pod \"e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-k5bkg\" (UID: \"197445f1-d262-48ee-87c5-7354132a69ba\") " pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-k5bkg"
Oct 31 01:16:22.492529 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:16:22.492414    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/197445f1-d262-48ee-87c5-7354132a69ba-workload-socket\") pod \"e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-k5bkg\" (UID: \"197445f1-d262-48ee-87c5-7354132a69ba\") " pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-k5bkg"
Oct 31 01:16:22.492529 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:16:22.492488    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/197445f1-d262-48ee-87c5-7354132a69ba-istio-podinfo\") pod \"e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-k5bkg\" (UID: \"197445f1-d262-48ee-87c5-7354132a69ba\") " pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-k5bkg"
Oct 31 01:16:22.492628 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:16:22.492541    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/197445f1-d262-48ee-87c5-7354132a69ba-credential-socket\") pod \"e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-k5bkg\" (UID: \"197445f1-d262-48ee-87c5-7354132a69ba\") " pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-k5bkg"
Oct 31 01:16:22.492628 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:16:22.492561    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/197445f1-d262-48ee-87c5-7354132a69ba-istio-envoy\") pod \"e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-k5bkg\" (UID: \"197445f1-d262-48ee-87c5-7354132a69ba\") " pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-k5bkg"
Oct 31 01:16:22.492726 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:16:22.492626    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/197445f1-d262-48ee-87c5-7354132a69ba-workload-certs\") pod \"e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-k5bkg\" (UID: \"197445f1-d262-48ee-87c5-7354132a69ba\") " pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-k5bkg"
Oct 31 01:16:22.492726 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:16:22.492649    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/197445f1-d262-48ee-87c5-7354132a69ba-istio-data\") pod \"e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-k5bkg\" (UID: \"197445f1-d262-48ee-87c5-7354132a69ba\") " pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-k5bkg"
Oct 31 01:16:22.593374 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:16:22.593266    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/197445f1-d262-48ee-87c5-7354132a69ba-istio-token\") pod \"e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-k5bkg\" (UID: \"197445f1-d262-48ee-87c5-7354132a69ba\") " pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-k5bkg"
Oct 31 01:16:22.593374 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:16:22.593333    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/197445f1-d262-48ee-87c5-7354132a69ba-workload-socket\") pod \"e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-k5bkg\" (UID: \"197445f1-d262-48ee-87c5-7354132a69ba\") " pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-k5bkg"
Oct 31 01:16:22.593606 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:16:22.593385    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/197445f1-d262-48ee-87c5-7354132a69ba-istio-podinfo\") pod \"e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-k5bkg\" (UID: \"197445f1-d262-48ee-87c5-7354132a69ba\") " pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-k5bkg"
Oct 31 01:16:22.593606 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:16:22.593560    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/197445f1-d262-48ee-87c5-7354132a69ba-credential-socket\") pod \"e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-k5bkg\" (UID: \"197445f1-d262-48ee-87c5-7354132a69ba\") " pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-k5bkg"
Oct 31 01:16:22.593606 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:16:22.593585    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/197445f1-d262-48ee-87c5-7354132a69ba-istio-envoy\") pod \"e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-k5bkg\" (UID: \"197445f1-d262-48ee-87c5-7354132a69ba\") " pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-k5bkg"
Oct 31 01:16:22.593773 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:16:22.593655    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/197445f1-d262-48ee-87c5-7354132a69ba-workload-certs\") pod \"e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-k5bkg\" (UID: \"197445f1-d262-48ee-87c5-7354132a69ba\") " pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-k5bkg"
Oct 31 01:16:22.593773 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:16:22.593718    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/197445f1-d262-48ee-87c5-7354132a69ba-istio-data\") pod \"e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-k5bkg\" (UID: \"197445f1-d262-48ee-87c5-7354132a69ba\") " pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-k5bkg"
Oct 31 01:16:22.593773 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:16:22.593762    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7855f\" (UniqueName: \"kubernetes.io/projected/197445f1-d262-48ee-87c5-7354132a69ba-kube-api-access-7855f\") pod \"e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-k5bkg\" (UID: \"197445f1-d262-48ee-87c5-7354132a69ba\") " pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-k5bkg"
Oct 31 01:16:22.593939 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:16:22.593846    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/197445f1-d262-48ee-87c5-7354132a69ba-workload-socket\") pod \"e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-k5bkg\" (UID: \"197445f1-d262-48ee-87c5-7354132a69ba\") " pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-k5bkg"
Oct 31 01:16:22.594004 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:16:22.593968    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/197445f1-d262-48ee-87c5-7354132a69ba-istiod-ca-cert\") pod \"e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-k5bkg\" (UID: \"197445f1-d262-48ee-87c5-7354132a69ba\") " pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-k5bkg"
Oct 31 01:16:22.594004 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:16:22.593981    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/197445f1-d262-48ee-87c5-7354132a69ba-workload-certs\") pod \"e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-k5bkg\" (UID: \"197445f1-d262-48ee-87c5-7354132a69ba\") " pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-k5bkg"
Oct 31 01:16:22.594111 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:16:22.594059    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/197445f1-d262-48ee-87c5-7354132a69ba-istio-data\") pod \"e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-k5bkg\" (UID: \"197445f1-d262-48ee-87c5-7354132a69ba\") " pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-k5bkg"
Oct 31 01:16:22.594421 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:16:22.594389    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/197445f1-d262-48ee-87c5-7354132a69ba-credential-socket\") pod \"e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-k5bkg\" (UID: \"197445f1-d262-48ee-87c5-7354132a69ba\") " pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-k5bkg"
Oct 31 01:16:22.594675 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:16:22.594650    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/197445f1-d262-48ee-87c5-7354132a69ba-istiod-ca-cert\") pod \"e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-k5bkg\" (UID: \"197445f1-d262-48ee-87c5-7354132a69ba\") " pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-k5bkg"
Oct 31 01:16:22.596229 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:16:22.596208    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/197445f1-d262-48ee-87c5-7354132a69ba-istio-envoy\") pod \"e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-k5bkg\" (UID: \"197445f1-d262-48ee-87c5-7354132a69ba\") " pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-k5bkg"
Oct 31 01:16:22.597046 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:16:22.597024    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/197445f1-d262-48ee-87c5-7354132a69ba-istio-podinfo\") pod \"e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-k5bkg\" (UID: \"197445f1-d262-48ee-87c5-7354132a69ba\") " pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-k5bkg"
Oct 31 01:16:22.650156 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:16:22.650129    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/197445f1-d262-48ee-87c5-7354132a69ba-istio-token\") pod \"e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-k5bkg\" (UID: \"197445f1-d262-48ee-87c5-7354132a69ba\") " pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-k5bkg"
Oct 31 01:16:22.650353 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:16:22.650302    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7855f\" (UniqueName: \"kubernetes.io/projected/197445f1-d262-48ee-87c5-7354132a69ba-kube-api-access-7855f\") pod \"e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-k5bkg\" (UID: \"197445f1-d262-48ee-87c5-7354132a69ba\") " pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-k5bkg"
Oct 31 01:16:22.766479 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:16:22.766440    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:16:22.942898 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:16:22.942821    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-k5bkg"
Oct 31 01:16:22.981894 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:16:22.981853    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:16:23.160894 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:16:23.160859    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:16:23.357291 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:16:23.357263    4964 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-baacee-6fcddb5455-j2fpn"]
Oct 31 01:16:23.362369 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:16:23.362346    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-baacee-6fcddb5455-j2fpn"
Oct 31 01:16:23.372428 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:16:23.372378    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-k5bkg"]
Oct 31 01:16:23.375058 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:16:23.375038    4964 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod197445f1_d262_48ee_87c5_7354132a69ba.slice/crio-2fca55b360fee11bcd5f8395c67c48c9ee548c315168b50474f524f46a6446f7 WatchSource:0}: Error finding container 2fca55b360fee11bcd5f8395c67c48c9ee548c315168b50474f524f46a6446f7: Status 404 returned error can't find the container with id 2fca55b360fee11bcd5f8395c67c48c9ee548c315168b50474f524f46a6446f7
Oct 31 01:16:23.377390 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:16:23.377356    4964 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 01:16:23.377466 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:16:23.377423    4964 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 01:16:23.377466 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:16:23.377453    4964 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 01:16:23.416175 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:16:23.416150    4964 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w1-baacee-dockercfg-f62tl\""
Oct 31 01:16:23.418618 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:16:23.418595    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-baacee-6fcddb5455-j2fpn"]
Oct 31 01:16:23.503335 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:16:23.503272    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d2bef5ff-d60c-4d04-ac18-dd98c3f92457-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-baacee-6fcddb5455-j2fpn\" (UID: \"d2bef5ff-d60c-4d04-ac18-dd98c3f92457\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-baacee-6fcddb5455-j2fpn"
Oct 31 01:16:23.503520 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:16:23.503408    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n6bdg\" (UniqueName: \"kubernetes.io/projected/d2bef5ff-d60c-4d04-ac18-dd98c3f92457-kube-api-access-n6bdg\") pod \"payload-pre-processing-e2e-worker-w1-baacee-6fcddb5455-j2fpn\" (UID: \"d2bef5ff-d60c-4d04-ac18-dd98c3f92457\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-baacee-6fcddb5455-j2fpn"
Oct 31 01:16:23.510263 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:16:23.510239    4964 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w1-baacee\""
Oct 31 01:16:23.605213 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:16:23.604309    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n6bdg\" (UniqueName: \"kubernetes.io/projected/d2bef5ff-d60c-4d04-ac18-dd98c3f92457-kube-api-access-n6bdg\") pod \"payload-pre-processing-e2e-worker-w1-baacee-6fcddb5455-j2fpn\" (UID: \"d2bef5ff-d60c-4d04-ac18-dd98c3f92457\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-baacee-6fcddb5455-j2fpn"
Oct 31 01:16:23.605213 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:16:23.604526    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d2bef5ff-d60c-4d04-ac18-dd98c3f92457-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-baacee-6fcddb5455-j2fpn\" (UID: \"d2bef5ff-d60c-4d04-ac18-dd98c3f92457\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-baacee-6fcddb5455-j2fpn"
Oct 31 01:16:23.605213 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:16:23.605171    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d2bef5ff-d60c-4d04-ac18-dd98c3f92457-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-baacee-6fcddb5455-j2fpn\" (UID: \"d2bef5ff-d60c-4d04-ac18-dd98c3f92457\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-baacee-6fcddb5455-j2fpn"
Oct 31 01:16:23.621605 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:16:23.620500    4964 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-7164dd00-6c969c79f6-2g46h"]
Oct 31 01:16:23.621605 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:16:23.620682    4964 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-7164dd00-6c969c79f6-2g46h" podUID="8270a0a8-7261-45b3-849f-6c05a6b17f8e" containerName="maas-api" containerID="cri-o://c72aceac6d2ef75286f906229ff65c57423f8c6dd01cd6f2a45916f868269a00" gracePeriod=30
Oct 31 01:16:23.663426 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:16:23.663398    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n6bdg\" (UniqueName: \"kubernetes.io/projected/d2bef5ff-d60c-4d04-ac18-dd98c3f92457-kube-api-access-n6bdg\") pod \"payload-pre-processing-e2e-worker-w1-baacee-6fcddb5455-j2fpn\" (UID: \"d2bef5ff-d60c-4d04-ac18-dd98c3f92457\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-baacee-6fcddb5455-j2fpn"
Oct 31 01:16:23.676532 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:16:23.676506    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-baacee-6fcddb5455-j2fpn"
Oct 31 01:16:23.772036 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:16:23.771876    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:16:23.922943 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:16:23.922920    4964 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-7164dd00-6c969c79f6-2g46h"
Oct 31 01:16:23.943682 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:16:23.943647    4964 generic.go:356] "Generic (PLEG): container finished" podID="8270a0a8-7261-45b3-849f-6c05a6b17f8e" containerID="c72aceac6d2ef75286f906229ff65c57423f8c6dd01cd6f2a45916f868269a00" exitCode=0
Oct 31 01:16:23.943824 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:16:23.943702    4964 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-7164dd00-6c969c79f6-2g46h"
Oct 31 01:16:23.943824 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:16:23.943733    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-7164dd00-6c969c79f6-2g46h" event={"ID":"8270a0a8-7261-45b3-849f-6c05a6b17f8e","Type":"ContainerDied","Data":"c72aceac6d2ef75286f906229ff65c57423f8c6dd01cd6f2a45916f868269a00"}
Oct 31 01:16:23.943824 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:16:23.943787    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-7164dd00-6c969c79f6-2g46h" event={"ID":"8270a0a8-7261-45b3-849f-6c05a6b17f8e","Type":"ContainerDied","Data":"d715d222023a68bd40b2de940a837d31679030a65b0ed097adfdf843d898651b"}
Oct 31 01:16:23.943824 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:16:23.943818    4964 scope.go:122] "RemoveContainer" containerID="c72aceac6d2ef75286f906229ff65c57423f8c6dd01cd6f2a45916f868269a00"
Oct 31 01:16:23.945479 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:16:23.945453    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-k5bkg" event={"ID":"197445f1-d262-48ee-87c5-7354132a69ba","Type":"ContainerStarted","Data":"2da91d4df5685d2274c1062cb98ca34fa297d1f9e78c8e9e4ed6e4662122a1c4"}
Oct 31 01:16:23.945618 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:16:23.945502    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-k5bkg" event={"ID":"197445f1-d262-48ee-87c5-7354132a69ba","Type":"ContainerStarted","Data":"2fca55b360fee11bcd5f8395c67c48c9ee548c315168b50474f524f46a6446f7"}
Oct 31 01:16:23.953296 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:16:23.953122    4964 scope.go:122] "RemoveContainer" containerID="c72aceac6d2ef75286f906229ff65c57423f8c6dd01cd6f2a45916f868269a00"
Oct 31 01:16:23.953445 ip-10-0-133-197 kubenswrapper[4964]: E1031 01:16:23.953425    4964 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c72aceac6d2ef75286f906229ff65c57423f8c6dd01cd6f2a45916f868269a00\": container with ID starting with c72aceac6d2ef75286f906229ff65c57423f8c6dd01cd6f2a45916f868269a00 not found: ID does not exist" containerID="c72aceac6d2ef75286f906229ff65c57423f8c6dd01cd6f2a45916f868269a00"
Oct 31 01:16:23.953525 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:16:23.953456    4964 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c72aceac6d2ef75286f906229ff65c57423f8c6dd01cd6f2a45916f868269a00"} err="failed to get container status \"c72aceac6d2ef75286f906229ff65c57423f8c6dd01cd6f2a45916f868269a00\": rpc error: code = NotFound desc = could not find container \"c72aceac6d2ef75286f906229ff65c57423f8c6dd01cd6f2a45916f868269a00\": container with ID starting with c72aceac6d2ef75286f906229ff65c57423f8c6dd01cd6f2a45916f868269a00 not found: ID does not exist"
Oct 31 01:16:24.076712 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:16:24.076689    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-baacee-6fcddb5455-j2fpn"]
Oct 31 01:16:24.080044 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:16:24.080013    4964 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd2bef5ff_d60c_4d04_ac18_dd98c3f92457.slice/crio-8d01963238b03719f8ad233d2b6abaa7876b92d3e155d4c70e97596d1bebbfbe WatchSource:0}: Error finding container 8d01963238b03719f8ad233d2b6abaa7876b92d3e155d4c70e97596d1bebbfbe: Status 404 returned error can't find the container with id 8d01963238b03719f8ad233d2b6abaa7876b92d3e155d4c70e97596d1bebbfbe
Oct 31 01:16:24.112329 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:16:24.112284    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8270a0a8-7261-45b3-849f-6c05a6b17f8e-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8270a0a8-7261-45b3-849f-6c05a6b17f8e-maas-api-tls\") pod \"8270a0a8-7261-45b3-849f-6c05a6b17f8e\" (UID: \"8270a0a8-7261-45b3-849f-6c05a6b17f8e\") "
Oct 31 01:16:24.112425 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:16:24.112381    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8270a0a8-7261-45b3-849f-6c05a6b17f8e-metrics-certs\" (UniqueName: \"kubernetes.io/secret/8270a0a8-7261-45b3-849f-6c05a6b17f8e-metrics-certs\") pod \"8270a0a8-7261-45b3-849f-6c05a6b17f8e\" (UID: \"8270a0a8-7261-45b3-849f-6c05a6b17f8e\") "
Oct 31 01:16:24.112480 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:16:24.112427    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8270a0a8-7261-45b3-849f-6c05a6b17f8e-kube-api-access-7p987\" (UniqueName: \"kubernetes.io/projected/8270a0a8-7261-45b3-849f-6c05a6b17f8e-kube-api-access-7p987\") pod \"8270a0a8-7261-45b3-849f-6c05a6b17f8e\" (UID: \"8270a0a8-7261-45b3-849f-6c05a6b17f8e\") "
Oct 31 01:16:24.112603 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:16:24.112586    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8270a0a8-7261-45b3-849f-6c05a6b17f8e-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8270a0a8-7261-45b3-849f-6c05a6b17f8e-openshift-service-ca\") pod \"8270a0a8-7261-45b3-849f-6c05a6b17f8e\" (UID: \"8270a0a8-7261-45b3-849f-6c05a6b17f8e\") "
Oct 31 01:16:24.113532 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:16:24.113501    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8270a0a8-7261-45b3-849f-6c05a6b17f8e-openshift-service-ca" pod "8270a0a8-7261-45b3-849f-6c05a6b17f8e" (UID: "8270a0a8-7261-45b3-849f-6c05a6b17f8e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:16:24.115554 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:16:24.115531    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8270a0a8-7261-45b3-849f-6c05a6b17f8e-metrics-certs" pod "8270a0a8-7261-45b3-849f-6c05a6b17f8e" (UID: "8270a0a8-7261-45b3-849f-6c05a6b17f8e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:16:24.115628 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:16:24.115603    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8270a0a8-7261-45b3-849f-6c05a6b17f8e-maas-api-tls" pod "8270a0a8-7261-45b3-849f-6c05a6b17f8e" (UID: "8270a0a8-7261-45b3-849f-6c05a6b17f8e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:16:24.115688 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:16:24.115632    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8270a0a8-7261-45b3-849f-6c05a6b17f8e-kube-api-access-7p987" pod "8270a0a8-7261-45b3-849f-6c05a6b17f8e" (UID: "8270a0a8-7261-45b3-849f-6c05a6b17f8e"). InnerVolumeSpecName "kube-api-access-7p987". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:16:24.130389 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:16:24.130367    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:16:24.157630 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:16:24.157584    4964 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-k5bkg" podStartSLOduration=2.157568258 podStartE2EDuration="2.157568258s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:16:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:16:24.155097195 +0000 UTC m=+752.631663592" watchObservedRunningTime="2026-10-31 01:16:24.157568258 +0000 UTC m=+752.634134639"
Oct 31 01:16:24.214054 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:16:24.214010    4964 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8270a0a8-7261-45b3-849f-6c05a6b17f8e-openshift-service-ca\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:16:24.214054 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:16:24.214044    4964 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8270a0a8-7261-45b3-849f-6c05a6b17f8e-maas-api-tls\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:16:24.214054 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:16:24.214055    4964 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8270a0a8-7261-45b3-849f-6c05a6b17f8e-metrics-certs\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:16:24.214054 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:16:24.214064    4964 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7p987\" (UniqueName: \"kubernetes.io/projected/8270a0a8-7261-45b3-849f-6c05a6b17f8e-kube-api-access-7p987\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:16:24.367439 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:16:24.367405    4964 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-7164dd00-6c969c79f6-2g46h"]
Oct 31 01:16:24.369565 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:16:24.369537    4964 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-7164dd00-6c969c79f6-2g46h"]
Oct 31 01:16:24.782201 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:16:24.782166    4964 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-870eefeb-cc99bd9cb-g7445"]
Oct 31 01:16:24.782395 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:16:24.782378    4964 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-870eefeb-cc99bd9cb-g7445" podUID="61037cfe-c14f-4293-b6f6-e49775dc2c94" containerName="maas-api" containerID="cri-o://2e931c55a2bd29092f6d2d3f578aece147c2468de2bcbac0aa5411cc767e4c90" gracePeriod=30
Oct 31 01:16:24.936052 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:16:24.936009    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:16:24.943756 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:16:24.943730    4964 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-k5bkg"
Oct 31 01:16:24.950158 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:16:24.950123    4964 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-k5bkg"
Oct 31 01:16:24.953869 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:16:24.953844    4964 generic.go:356] "Generic (PLEG): container finished" podID="61037cfe-c14f-4293-b6f6-e49775dc2c94" containerID="2e931c55a2bd29092f6d2d3f578aece147c2468de2bcbac0aa5411cc767e4c90" exitCode=0
Oct 31 01:16:24.954011 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:16:24.953917    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-870eefeb-cc99bd9cb-g7445" event={"ID":"61037cfe-c14f-4293-b6f6-e49775dc2c94","Type":"ContainerDied","Data":"2e931c55a2bd29092f6d2d3f578aece147c2468de2bcbac0aa5411cc767e4c90"}
Oct 31 01:16:24.973235 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:16:24.973196    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-baacee-6fcddb5455-j2fpn" event={"ID":"d2bef5ff-d60c-4d04-ac18-dd98c3f92457","Type":"ContainerStarted","Data":"811d489d988679399ecc75e71dec6e369930bd163c8fd7618a5a39aaef54919c"}
Oct 31 01:16:24.973508 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:16:24.973486    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-baacee-6fcddb5455-j2fpn" event={"ID":"d2bef5ff-d60c-4d04-ac18-dd98c3f92457","Type":"ContainerStarted","Data":"8d01963238b03719f8ad233d2b6abaa7876b92d3e155d4c70e97596d1bebbfbe"}
Oct 31 01:16:24.973634 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:16:24.973620    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-k5bkg"
Oct 31 01:16:24.973741 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:16:24.973723    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-baacee-6fcddb5455-j2fpn"
Oct 31 01:16:24.977650 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:16:24.977629    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-k5bkg"
Oct 31 01:16:25.046665 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:16:25.046643    4964 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-870eefeb-cc99bd9cb-g7445"
Oct 31 01:16:25.123712 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:16:25.123684    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/61037cfe-c14f-4293-b6f6-e49775dc2c94-kube-api-access-xk9fd\" (UniqueName: \"kubernetes.io/projected/61037cfe-c14f-4293-b6f6-e49775dc2c94-kube-api-access-xk9fd\") pod \"61037cfe-c14f-4293-b6f6-e49775dc2c94\" (UID: \"61037cfe-c14f-4293-b6f6-e49775dc2c94\") "
Oct 31 01:16:25.123712 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:16:25.123716    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/61037cfe-c14f-4293-b6f6-e49775dc2c94-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/61037cfe-c14f-4293-b6f6-e49775dc2c94-openshift-service-ca\") pod \"61037cfe-c14f-4293-b6f6-e49775dc2c94\" (UID: \"61037cfe-c14f-4293-b6f6-e49775dc2c94\") "
Oct 31 01:16:25.123907 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:16:25.123744    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/61037cfe-c14f-4293-b6f6-e49775dc2c94-metrics-certs\" (UniqueName: \"kubernetes.io/secret/61037cfe-c14f-4293-b6f6-e49775dc2c94-metrics-certs\") pod \"61037cfe-c14f-4293-b6f6-e49775dc2c94\" (UID: \"61037cfe-c14f-4293-b6f6-e49775dc2c94\") "
Oct 31 01:16:25.123907 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:16:25.123786    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/61037cfe-c14f-4293-b6f6-e49775dc2c94-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/61037cfe-c14f-4293-b6f6-e49775dc2c94-maas-api-tls\") pod \"61037cfe-c14f-4293-b6f6-e49775dc2c94\" (UID: \"61037cfe-c14f-4293-b6f6-e49775dc2c94\") "
Oct 31 01:16:25.124111 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:16:25.124076    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/61037cfe-c14f-4293-b6f6-e49775dc2c94-openshift-service-ca" pod "61037cfe-c14f-4293-b6f6-e49775dc2c94" (UID: "61037cfe-c14f-4293-b6f6-e49775dc2c94"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:16:25.126049 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:16:25.126019    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/61037cfe-c14f-4293-b6f6-e49775dc2c94-maas-api-tls" pod "61037cfe-c14f-4293-b6f6-e49775dc2c94" (UID: "61037cfe-c14f-4293-b6f6-e49775dc2c94"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:16:25.126164 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:16:25.126140    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/61037cfe-c14f-4293-b6f6-e49775dc2c94-metrics-certs" pod "61037cfe-c14f-4293-b6f6-e49775dc2c94" (UID: "61037cfe-c14f-4293-b6f6-e49775dc2c94"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:16:25.126215 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:16:25.126143    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/61037cfe-c14f-4293-b6f6-e49775dc2c94-kube-api-access-xk9fd" pod "61037cfe-c14f-4293-b6f6-e49775dc2c94" (UID: "61037cfe-c14f-4293-b6f6-e49775dc2c94"). InnerVolumeSpecName "kube-api-access-xk9fd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:16:25.182782 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:16:25.182699    4964 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-baacee-6fcddb5455-j2fpn" podStartSLOduration=2.182686038 podStartE2EDuration="2.182686038s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:16:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:16:25.181556037 +0000 UTC m=+753.658122417" watchObservedRunningTime="2026-10-31 01:16:25.182686038 +0000 UTC m=+753.659252418"
Oct 31 01:16:25.225525 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:16:25.225496    4964 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xk9fd\" (UniqueName: \"kubernetes.io/projected/61037cfe-c14f-4293-b6f6-e49775dc2c94-kube-api-access-xk9fd\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:16:25.225525 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:16:25.225523    4964 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/61037cfe-c14f-4293-b6f6-e49775dc2c94-openshift-service-ca\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:16:25.225689 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:16:25.225537    4964 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/61037cfe-c14f-4293-b6f6-e49775dc2c94-metrics-certs\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:16:25.225689 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:16:25.225551    4964 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/61037cfe-c14f-4293-b6f6-e49775dc2c94-maas-api-tls\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:16:25.359772 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:16:25.359737    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:16:25.613205 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:16:25.613176    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:16:25.777276 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:16:25.777244    4964 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8270a0a8-7261-45b3-849f-6c05a6b17f8e" path="/var/lib/kubelet/pods/8270a0a8-7261-45b3-849f-6c05a6b17f8e/volumes"
Oct 31 01:16:25.979375 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:16:25.979342    4964 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-870eefeb-cc99bd9cb-g7445"
Oct 31 01:16:25.979550 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:16:25.979363    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-870eefeb-cc99bd9cb-g7445" event={"ID":"61037cfe-c14f-4293-b6f6-e49775dc2c94","Type":"ContainerDied","Data":"a87ca34da28947b69cf60e618575dc55ca2da07e0af4b1224ab554ac2075725e"}
Oct 31 01:16:25.979550 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:16:25.979440    4964 scope.go:122] "RemoveContainer" containerID="2e931c55a2bd29092f6d2d3f578aece147c2468de2bcbac0aa5411cc767e4c90"
Oct 31 01:16:26.143697 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:16:26.143664    4964 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-870eefeb-cc99bd9cb-g7445"]
Oct 31 01:16:26.191563 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:16:26.191531    4964 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-870eefeb-cc99bd9cb-g7445"]
Oct 31 01:16:26.585540 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:16:26.585508    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:16:26.858467 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:16:26.858380    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:16:27.034432 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:16:27.034391    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:16:27.219177 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:16:27.219146    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:16:27.775628 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:16:27.775595    4964 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="61037cfe-c14f-4293-b6f6-e49775dc2c94" path="/var/lib/kubelet/pods/61037cfe-c14f-4293-b6f6-e49775dc2c94/volumes"
Oct 31 01:16:28.058430 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:16:28.058348    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:16:28.515960 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:16:28.515927    4964 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-z5xgw"]
Oct 31 01:16:28.516451 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:16:28.516437    4964 cpu_manager.go:414] "containerMap: removing container" podUID="61037cfe-c14f-4293-b6f6-e49775dc2c94" containerName="maas-api"
Oct 31 01:16:28.516496 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:16:28.516452    4964 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="61037cfe-c14f-4293-b6f6-e49775dc2c94" containerName="maas-api"
Oct 31 01:16:28.516496 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:16:28.516487    4964 cpu_manager.go:414] "containerMap: removing container" podUID="8270a0a8-7261-45b3-849f-6c05a6b17f8e" containerName="maas-api"
Oct 31 01:16:28.516496 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:16:28.516493    4964 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8270a0a8-7261-45b3-849f-6c05a6b17f8e" containerName="maas-api"
Oct 31 01:16:28.516590 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:16:28.516580    4964 memory_manager.go:365] "RemoveStaleState removing state" podUID="8270a0a8-7261-45b3-849f-6c05a6b17f8e" containerName="maas-api"
Oct 31 01:16:28.516628 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:16:28.516596    4964 memory_manager.go:365] "RemoveStaleState removing state" podUID="61037cfe-c14f-4293-b6f6-e49775dc2c94" containerName="maas-api"
Oct 31 01:16:28.521385 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:16:28.521361    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-z5xgw"
Oct 31 01:16:28.561345 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:16:28.558833    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f4qzp\" (UniqueName: \"kubernetes.io/projected/b8249620-2e20-4e80-a8b8-7d8e566f8973-kube-api-access-f4qzp\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-z5xgw\" (UID: \"b8249620-2e20-4e80-a8b8-7d8e566f8973\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-z5xgw"
Oct 31 01:16:28.561345 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:16:28.558886    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b8249620-2e20-4e80-a8b8-7d8e566f8973-istio-token\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-z5xgw\" (UID: \"b8249620-2e20-4e80-a8b8-7d8e566f8973\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-z5xgw"
Oct 31 01:16:28.561345 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:16:28.558949    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b8249620-2e20-4e80-a8b8-7d8e566f8973-istiod-ca-cert\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-z5xgw\" (UID: \"b8249620-2e20-4e80-a8b8-7d8e566f8973\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-z5xgw"
Oct 31 01:16:28.561345 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:16:28.558977    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b8249620-2e20-4e80-a8b8-7d8e566f8973-istio-envoy\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-z5xgw\" (UID: \"b8249620-2e20-4e80-a8b8-7d8e566f8973\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-z5xgw"
Oct 31 01:16:28.561345 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:16:28.559035    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8249620-2e20-4e80-a8b8-7d8e566f8973-credential-socket\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-z5xgw\" (UID: \"b8249620-2e20-4e80-a8b8-7d8e566f8973\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-z5xgw"
Oct 31 01:16:28.561345 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:16:28.559062    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8249620-2e20-4e80-a8b8-7d8e566f8973-workload-socket\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-z5xgw\" (UID: \"b8249620-2e20-4e80-a8b8-7d8e566f8973\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-z5xgw"
Oct 31 01:16:28.561345 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:16:28.559090    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b8249620-2e20-4e80-a8b8-7d8e566f8973-workload-certs\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-z5xgw\" (UID: \"b8249620-2e20-4e80-a8b8-7d8e566f8973\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-z5xgw"
Oct 31 01:16:28.561345 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:16:28.559135    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b8249620-2e20-4e80-a8b8-7d8e566f8973-istio-podinfo\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-z5xgw\" (UID: \"b8249620-2e20-4e80-a8b8-7d8e566f8973\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-z5xgw"
Oct 31 01:16:28.561345 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:16:28.559173    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b8249620-2e20-4e80-a8b8-7d8e566f8973-istio-data\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-z5xgw\" (UID: \"b8249620-2e20-4e80-a8b8-7d8e566f8973\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-z5xgw"
Oct 31 01:16:28.581548 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:16:28.575678    4964 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-00e840-openshift-default-dockercfg-2nfqp\""
Oct 31 01:16:28.629466 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:16:28.629434    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-z5xgw"]
Oct 31 01:16:28.648467 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:16:28.648437    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:16:28.660540 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:16:28.660511    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b8249620-2e20-4e80-a8b8-7d8e566f8973-istio-data\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-z5xgw\" (UID: \"b8249620-2e20-4e80-a8b8-7d8e566f8973\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-z5xgw"
Oct 31 01:16:28.660715 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:16:28.660604    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f4qzp\" (UniqueName: \"kubernetes.io/projected/b8249620-2e20-4e80-a8b8-7d8e566f8973-kube-api-access-f4qzp\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-z5xgw\" (UID: \"b8249620-2e20-4e80-a8b8-7d8e566f8973\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-z5xgw"
Oct 31 01:16:28.660715 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:16:28.660637    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b8249620-2e20-4e80-a8b8-7d8e566f8973-istio-token\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-z5xgw\" (UID: \"b8249620-2e20-4e80-a8b8-7d8e566f8973\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-z5xgw"
Oct 31 01:16:28.660877 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:16:28.660856    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b8249620-2e20-4e80-a8b8-7d8e566f8973-istiod-ca-cert\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-z5xgw\" (UID: \"b8249620-2e20-4e80-a8b8-7d8e566f8973\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-z5xgw"
Oct 31 01:16:28.660979 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:16:28.660951    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b8249620-2e20-4e80-a8b8-7d8e566f8973-istio-data\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-z5xgw\" (UID: \"b8249620-2e20-4e80-a8b8-7d8e566f8973\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-z5xgw"
Oct 31 01:16:28.661537 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:16:28.661509    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b8249620-2e20-4e80-a8b8-7d8e566f8973-istiod-ca-cert\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-z5xgw\" (UID: \"b8249620-2e20-4e80-a8b8-7d8e566f8973\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-z5xgw"
Oct 31 01:16:28.661650 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:16:28.661576    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b8249620-2e20-4e80-a8b8-7d8e566f8973-istio-envoy\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-z5xgw\" (UID: \"b8249620-2e20-4e80-a8b8-7d8e566f8973\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-z5xgw"
Oct 31 01:16:28.661650 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:16:28.661637    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8249620-2e20-4e80-a8b8-7d8e566f8973-credential-socket\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-z5xgw\" (UID: \"b8249620-2e20-4e80-a8b8-7d8e566f8973\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-z5xgw"
Oct 31 01:16:28.661755 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:16:28.661667    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8249620-2e20-4e80-a8b8-7d8e566f8973-workload-socket\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-z5xgw\" (UID: \"b8249620-2e20-4e80-a8b8-7d8e566f8973\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-z5xgw"
Oct 31 01:16:28.661810 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:16:28.661791    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b8249620-2e20-4e80-a8b8-7d8e566f8973-workload-certs\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-z5xgw\" (UID: \"b8249620-2e20-4e80-a8b8-7d8e566f8973\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-z5xgw"
Oct 31 01:16:28.661899 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:16:28.661883    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b8249620-2e20-4e80-a8b8-7d8e566f8973-istio-podinfo\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-z5xgw\" (UID: \"b8249620-2e20-4e80-a8b8-7d8e566f8973\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-z5xgw"
Oct 31 01:16:28.661982 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:16:28.661959    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8249620-2e20-4e80-a8b8-7d8e566f8973-workload-socket\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-z5xgw\" (UID: \"b8249620-2e20-4e80-a8b8-7d8e566f8973\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-z5xgw"
Oct 31 01:16:28.662067 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:16:28.662040    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8249620-2e20-4e80-a8b8-7d8e566f8973-credential-socket\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-z5xgw\" (UID: \"b8249620-2e20-4e80-a8b8-7d8e566f8973\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-z5xgw"
Oct 31 01:16:28.662300 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:16:28.662276    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b8249620-2e20-4e80-a8b8-7d8e566f8973-workload-certs\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-z5xgw\" (UID: \"b8249620-2e20-4e80-a8b8-7d8e566f8973\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-z5xgw"
Oct 31 01:16:28.664478 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:16:28.664453    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b8249620-2e20-4e80-a8b8-7d8e566f8973-istio-envoy\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-z5xgw\" (UID: \"b8249620-2e20-4e80-a8b8-7d8e566f8973\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-z5xgw"
Oct 31 01:16:28.664848 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:16:28.664822    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b8249620-2e20-4e80-a8b8-7d8e566f8973-istio-podinfo\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-z5xgw\" (UID: \"b8249620-2e20-4e80-a8b8-7d8e566f8973\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-z5xgw"
Oct 31 01:16:28.720912 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:16:28.720888    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b8249620-2e20-4e80-a8b8-7d8e566f8973-istio-token\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-z5xgw\" (UID: \"b8249620-2e20-4e80-a8b8-7d8e566f8973\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-z5xgw"
Oct 31 01:16:28.721178 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:16:28.721160    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f4qzp\" (UniqueName: \"kubernetes.io/projected/b8249620-2e20-4e80-a8b8-7d8e566f8973-kube-api-access-f4qzp\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-z5xgw\" (UID: \"b8249620-2e20-4e80-a8b8-7d8e566f8973\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-z5xgw"
Oct 31 01:16:28.852308 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:16:28.852226    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-z5xgw"
Oct 31 01:16:29.018544 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:16:29.018502    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:16:29.279783 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:16:29.279758    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-z5xgw"]
Oct 31 01:16:29.281574 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:16:29.281548    4964 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb8249620_2e20_4e80_a8b8_7d8e566f8973.slice/crio-eb82087df74af2e96066f2973438d8ef40647db391c64ebc1aa64de4a875299b WatchSource:0}: Error finding container eb82087df74af2e96066f2973438d8ef40647db391c64ebc1aa64de4a875299b: Status 404 returned error can't find the container with id eb82087df74af2e96066f2973438d8ef40647db391c64ebc1aa64de4a875299b
Oct 31 01:16:29.283853 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:16:29.283823    4964 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 01:16:29.283916 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:16:29.283889    4964 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 01:16:29.283956 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:16:29.283928    4964 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 01:16:30.006659 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:16:30.006622    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-z5xgw" event={"ID":"b8249620-2e20-4e80-a8b8-7d8e566f8973","Type":"ContainerStarted","Data":"fb10c38a790062d0c788cfe72ba32e556f4b7975316f7e9459bd3b5ada405c17"}
Oct 31 01:16:30.006863 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:16:30.006676    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-z5xgw" event={"ID":"b8249620-2e20-4e80-a8b8-7d8e566f8973","Type":"ContainerStarted","Data":"eb82087df74af2e96066f2973438d8ef40647db391c64ebc1aa64de4a875299b"}
Oct 31 01:16:30.175371 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:16:30.175274    4964 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-z5xgw" podStartSLOduration=2.175260122 podStartE2EDuration="2.175260122s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:16:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:16:30.172650285 +0000 UTC m=+758.649216665" watchObservedRunningTime="2026-10-31 01:16:30.175260122 +0000 UTC m=+758.651826502"
Oct 31 01:16:30.852558 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:16:30.852524    4964 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-z5xgw"
Oct 31 01:16:30.857709 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:16:30.857685    4964 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-z5xgw"
Oct 31 01:16:31.012408 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:16:31.012379    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-z5xgw"
Oct 31 01:16:31.013822 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:16:31.013804    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-z5xgw"
Oct 31 01:16:32.002802 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:16:32.002770    4964 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-baacee-77bdffbd-4vbp7"]
Oct 31 01:16:32.006690 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:16:32.006670    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-baacee-77bdffbd-4vbp7"
Oct 31 01:16:32.058792 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:16:32.058769    4964 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w1-baacee-serving-cert\""
Oct 31 01:16:32.099603 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:16:32.099579    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/803fb9d4-2f24-48f3-bea0-f62833d91796-metrics-certs\") pod \"maas-api-e2e-worker-w1-baacee-77bdffbd-4vbp7\" (UID: \"803fb9d4-2f24-48f3-bea0-f62833d91796\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-baacee-77bdffbd-4vbp7"
Oct 31 01:16:32.099720 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:16:32.099625    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/803fb9d4-2f24-48f3-bea0-f62833d91796-maas-api-tls\") pod \"maas-api-e2e-worker-w1-baacee-77bdffbd-4vbp7\" (UID: \"803fb9d4-2f24-48f3-bea0-f62833d91796\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-baacee-77bdffbd-4vbp7"
Oct 31 01:16:32.099720 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:16:32.099680    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/803fb9d4-2f24-48f3-bea0-f62833d91796-openshift-service-ca\") pod \"maas-api-e2e-worker-w1-baacee-77bdffbd-4vbp7\" (UID: \"803fb9d4-2f24-48f3-bea0-f62833d91796\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-baacee-77bdffbd-4vbp7"
Oct 31 01:16:32.099798 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:16:32.099754    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t6vvh\" (UniqueName: \"kubernetes.io/projected/803fb9d4-2f24-48f3-bea0-f62833d91796-kube-api-access-t6vvh\") pod \"maas-api-e2e-worker-w1-baacee-77bdffbd-4vbp7\" (UID: \"803fb9d4-2f24-48f3-bea0-f62833d91796\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-baacee-77bdffbd-4vbp7"
Oct 31 01:16:32.111480 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:16:32.111456    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-baacee-77bdffbd-4vbp7"]
Oct 31 01:16:32.201356 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:16:32.201327    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t6vvh\" (UniqueName: \"kubernetes.io/projected/803fb9d4-2f24-48f3-bea0-f62833d91796-kube-api-access-t6vvh\") pod \"maas-api-e2e-worker-w1-baacee-77bdffbd-4vbp7\" (UID: \"803fb9d4-2f24-48f3-bea0-f62833d91796\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-baacee-77bdffbd-4vbp7"
Oct 31 01:16:32.201550 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:16:32.201529    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/803fb9d4-2f24-48f3-bea0-f62833d91796-metrics-certs\") pod \"maas-api-e2e-worker-w1-baacee-77bdffbd-4vbp7\" (UID: \"803fb9d4-2f24-48f3-bea0-f62833d91796\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-baacee-77bdffbd-4vbp7"
Oct 31 01:16:32.201627 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:16:32.201589    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/803fb9d4-2f24-48f3-bea0-f62833d91796-maas-api-tls\") pod \"maas-api-e2e-worker-w1-baacee-77bdffbd-4vbp7\" (UID: \"803fb9d4-2f24-48f3-bea0-f62833d91796\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-baacee-77bdffbd-4vbp7"
Oct 31 01:16:32.201627 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:16:32.201616    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/803fb9d4-2f24-48f3-bea0-f62833d91796-openshift-service-ca\") pod \"maas-api-e2e-worker-w1-baacee-77bdffbd-4vbp7\" (UID: \"803fb9d4-2f24-48f3-bea0-f62833d91796\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-baacee-77bdffbd-4vbp7"
Oct 31 01:16:32.201759 ip-10-0-133-197 kubenswrapper[4964]: E1031 01:16:32.201738    4964 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w1-baacee-serving-cert: secret "maas-api-e2e-worker-w1-baacee-serving-cert" not found
Oct 31 01:16:32.201818 ip-10-0-133-197 kubenswrapper[4964]: E1031 01:16:32.201807    4964 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/803fb9d4-2f24-48f3-bea0-f62833d91796-maas-api-tls podName:803fb9d4-2f24-48f3-bea0-f62833d91796 nodeName:}" failed. No retries permitted until 2026-10-31 01:16:32.701788154 +0000 UTC m=+761.178354522 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/803fb9d4-2f24-48f3-bea0-f62833d91796-maas-api-tls") pod "maas-api-e2e-worker-w1-baacee-77bdffbd-4vbp7" (UID: "803fb9d4-2f24-48f3-bea0-f62833d91796") : secret "maas-api-e2e-worker-w1-baacee-serving-cert" not found
Oct 31 01:16:32.202219 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:16:32.202197    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/803fb9d4-2f24-48f3-bea0-f62833d91796-openshift-service-ca\") pod \"maas-api-e2e-worker-w1-baacee-77bdffbd-4vbp7\" (UID: \"803fb9d4-2f24-48f3-bea0-f62833d91796\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-baacee-77bdffbd-4vbp7"
Oct 31 01:16:32.203854 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:16:32.203836    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/803fb9d4-2f24-48f3-bea0-f62833d91796-metrics-certs\") pod \"maas-api-e2e-worker-w1-baacee-77bdffbd-4vbp7\" (UID: \"803fb9d4-2f24-48f3-bea0-f62833d91796\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-baacee-77bdffbd-4vbp7"
Oct 31 01:16:32.241267 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:16:32.241239    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:16:32.258992 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:16:32.258929    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t6vvh\" (UniqueName: \"kubernetes.io/projected/803fb9d4-2f24-48f3-bea0-f62833d91796-kube-api-access-t6vvh\") pod \"maas-api-e2e-worker-w1-baacee-77bdffbd-4vbp7\" (UID: \"803fb9d4-2f24-48f3-bea0-f62833d91796\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-baacee-77bdffbd-4vbp7"
Oct 31 01:16:32.468109 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:16:32.468067    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:16:32.706468 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:16:32.706389    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/803fb9d4-2f24-48f3-bea0-f62833d91796-maas-api-tls\") pod \"maas-api-e2e-worker-w1-baacee-77bdffbd-4vbp7\" (UID: \"803fb9d4-2f24-48f3-bea0-f62833d91796\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-baacee-77bdffbd-4vbp7"
Oct 31 01:16:32.709137 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:16:32.709116    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/803fb9d4-2f24-48f3-bea0-f62833d91796-maas-api-tls\") pod \"maas-api-e2e-worker-w1-baacee-77bdffbd-4vbp7\" (UID: \"803fb9d4-2f24-48f3-bea0-f62833d91796\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-baacee-77bdffbd-4vbp7"
Oct 31 01:16:32.920491 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:16:32.920464    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-baacee-77bdffbd-4vbp7"
Oct 31 01:16:33.297413 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:16:33.297383    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-baacee-77bdffbd-4vbp7"]
Oct 31 01:16:33.299562 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:16:33.299539    4964 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod803fb9d4_2f24_48f3_bea0_f62833d91796.slice/crio-903640ff765eb7eb63d0197ca2baef33ed0abf54633c4b1381c508fdb78ebea2 WatchSource:0}: Error finding container 903640ff765eb7eb63d0197ca2baef33ed0abf54633c4b1381c508fdb78ebea2: Status 404 returned error can't find the container with id 903640ff765eb7eb63d0197ca2baef33ed0abf54633c4b1381c508fdb78ebea2
Oct 31 01:16:33.717863 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:16:33.717779    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:16:33.973213 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:16:33.973174    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:16:34.031616 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:16:34.031587    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-baacee-77bdffbd-4vbp7" event={"ID":"803fb9d4-2f24-48f3-bea0-f62833d91796","Type":"ContainerStarted","Data":"903640ff765eb7eb63d0197ca2baef33ed0abf54633c4b1381c508fdb78ebea2"}
Oct 31 01:16:35.039060 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:16:35.039027    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-baacee-77bdffbd-4vbp7" event={"ID":"803fb9d4-2f24-48f3-bea0-f62833d91796","Type":"ContainerStarted","Data":"19c7ec33e5e6f13b92f9de6dee708277d3f915fe3cc9784e7782e2777f5b4be1"}
Oct 31 01:16:35.039573 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:16:35.039140    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-baacee-77bdffbd-4vbp7"
Oct 31 01:16:35.159774 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:16:35.159719    4964 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-baacee-77bdffbd-4vbp7" podStartSLOduration=3.214335161 podStartE2EDuration="4.159705499s" totalImagesPullingTime="945.370338ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:16:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:16:35.156833672 +0000 UTC m=+763.633400052" watchObservedRunningTime="2026-10-31 01:16:35.159705499 +0000 UTC m=+763.636271878"
Oct 31 01:16:35.230694 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:16:35.230666    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:16:35.921433 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:16:35.921398    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:16:36.158814 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:16:36.158785    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:16:36.449374 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:16:36.449337    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:16:36.989934 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:16:36.989896    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-baacee-6fcddb5455-j2fpn"
Oct 31 01:16:37.801298 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:16:37.801208    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:16:38.301043 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:16:38.301012    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:16:38.534137 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:16:38.534106    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:16:39.546154 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:16:39.546121    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:16:40.190128 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:16:40.190088    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:16:40.385966 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:16:40.385932    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:16:41.050839 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:16:41.050811    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-baacee-77bdffbd-4vbp7"
Oct 31 01:16:41.238295 ip-10-0-133-197 kubenswrapper[4964]: E1031 01:16:41.238260    4964 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q56x5" containerName="extractor"
Oct 31 01:16:42.906460 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:16:42.906419    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:16:44.899526 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:16:44.899465    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:16:45.103909 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:16:45.103868    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:16:47.288912 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:16:47.288878    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:16:47.331833 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:16:47.331805    4964 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-00e840-58b9774b57-b6vbm"]
Oct 31 01:16:47.336668 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:16:47.336648    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-00e840-58b9774b57-b6vbm"
Oct 31 01:16:47.391893 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:16:47.391868    4964 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w2-00e840-serving-cert\""
Oct 31 01:16:47.443989 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:16:47.443962    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-00e840-58b9774b57-b6vbm"]
Oct 31 01:16:47.457814 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:16:47.457782    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dtwf6\" (UniqueName: \"kubernetes.io/projected/750dc732-c296-44cb-9f7f-a7f7be7b4144-kube-api-access-dtwf6\") pod \"maas-api-e2e-worker-w2-00e840-58b9774b57-b6vbm\" (UID: \"750dc732-c296-44cb-9f7f-a7f7be7b4144\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-00e840-58b9774b57-b6vbm"
Oct 31 01:16:47.457956 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:16:47.457848    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/750dc732-c296-44cb-9f7f-a7f7be7b4144-openshift-service-ca\") pod \"maas-api-e2e-worker-w2-00e840-58b9774b57-b6vbm\" (UID: \"750dc732-c296-44cb-9f7f-a7f7be7b4144\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-00e840-58b9774b57-b6vbm"
Oct 31 01:16:47.458051 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:16:47.458027    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/750dc732-c296-44cb-9f7f-a7f7be7b4144-maas-api-tls\") pod \"maas-api-e2e-worker-w2-00e840-58b9774b57-b6vbm\" (UID: \"750dc732-c296-44cb-9f7f-a7f7be7b4144\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-00e840-58b9774b57-b6vbm"
Oct 31 01:16:47.458092 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:16:47.458065    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/750dc732-c296-44cb-9f7f-a7f7be7b4144-metrics-certs\") pod \"maas-api-e2e-worker-w2-00e840-58b9774b57-b6vbm\" (UID: \"750dc732-c296-44cb-9f7f-a7f7be7b4144\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-00e840-58b9774b57-b6vbm"
Oct 31 01:16:47.559137 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:16:47.559041    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dtwf6\" (UniqueName: \"kubernetes.io/projected/750dc732-c296-44cb-9f7f-a7f7be7b4144-kube-api-access-dtwf6\") pod \"maas-api-e2e-worker-w2-00e840-58b9774b57-b6vbm\" (UID: \"750dc732-c296-44cb-9f7f-a7f7be7b4144\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-00e840-58b9774b57-b6vbm"
Oct 31 01:16:47.559137 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:16:47.559114    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/750dc732-c296-44cb-9f7f-a7f7be7b4144-openshift-service-ca\") pod \"maas-api-e2e-worker-w2-00e840-58b9774b57-b6vbm\" (UID: \"750dc732-c296-44cb-9f7f-a7f7be7b4144\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-00e840-58b9774b57-b6vbm"
Oct 31 01:16:47.559387 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:16:47.559227    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/750dc732-c296-44cb-9f7f-a7f7be7b4144-maas-api-tls\") pod \"maas-api-e2e-worker-w2-00e840-58b9774b57-b6vbm\" (UID: \"750dc732-c296-44cb-9f7f-a7f7be7b4144\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-00e840-58b9774b57-b6vbm"
Oct 31 01:16:47.559387 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:16:47.559252    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/750dc732-c296-44cb-9f7f-a7f7be7b4144-metrics-certs\") pod \"maas-api-e2e-worker-w2-00e840-58b9774b57-b6vbm\" (UID: \"750dc732-c296-44cb-9f7f-a7f7be7b4144\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-00e840-58b9774b57-b6vbm"
Oct 31 01:16:47.559500 ip-10-0-133-197 kubenswrapper[4964]: E1031 01:16:47.559455    4964 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w2-00e840-serving-cert: secret "maas-api-e2e-worker-w2-00e840-serving-cert" not found
Oct 31 01:16:47.559552 ip-10-0-133-197 kubenswrapper[4964]: E1031 01:16:47.559533    4964 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/750dc732-c296-44cb-9f7f-a7f7be7b4144-maas-api-tls podName:750dc732-c296-44cb-9f7f-a7f7be7b4144 nodeName:}" failed. No retries permitted until 2026-10-31 01:16:48.05951263 +0000 UTC m=+776.536079007 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/750dc732-c296-44cb-9f7f-a7f7be7b4144-maas-api-tls") pod "maas-api-e2e-worker-w2-00e840-58b9774b57-b6vbm" (UID: "750dc732-c296-44cb-9f7f-a7f7be7b4144") : secret "maas-api-e2e-worker-w2-00e840-serving-cert" not found
Oct 31 01:16:47.559930 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:16:47.559906    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/750dc732-c296-44cb-9f7f-a7f7be7b4144-openshift-service-ca\") pod \"maas-api-e2e-worker-w2-00e840-58b9774b57-b6vbm\" (UID: \"750dc732-c296-44cb-9f7f-a7f7be7b4144\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-00e840-58b9774b57-b6vbm"
Oct 31 01:16:47.561934 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:16:47.561907    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/750dc732-c296-44cb-9f7f-a7f7be7b4144-metrics-certs\") pod \"maas-api-e2e-worker-w2-00e840-58b9774b57-b6vbm\" (UID: \"750dc732-c296-44cb-9f7f-a7f7be7b4144\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-00e840-58b9774b57-b6vbm"
Oct 31 01:16:47.583614 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:16:47.583569    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:16:47.617076 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:16:47.617045    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dtwf6\" (UniqueName: \"kubernetes.io/projected/750dc732-c296-44cb-9f7f-a7f7be7b4144-kube-api-access-dtwf6\") pod \"maas-api-e2e-worker-w2-00e840-58b9774b57-b6vbm\" (UID: \"750dc732-c296-44cb-9f7f-a7f7be7b4144\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-00e840-58b9774b57-b6vbm"
Oct 31 01:16:47.775218 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:16:47.775178    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:16:47.970089 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:16:47.970047    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:16:48.064980 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:16:48.064946    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/750dc732-c296-44cb-9f7f-a7f7be7b4144-maas-api-tls\") pod \"maas-api-e2e-worker-w2-00e840-58b9774b57-b6vbm\" (UID: \"750dc732-c296-44cb-9f7f-a7f7be7b4144\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-00e840-58b9774b57-b6vbm"
Oct 31 01:16:48.067619 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:16:48.067595    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/750dc732-c296-44cb-9f7f-a7f7be7b4144-maas-api-tls\") pod \"maas-api-e2e-worker-w2-00e840-58b9774b57-b6vbm\" (UID: \"750dc732-c296-44cb-9f7f-a7f7be7b4144\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-00e840-58b9774b57-b6vbm"
Oct 31 01:16:48.187483 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:16:48.187442    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:16:48.249859 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:16:48.249775    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-00e840-58b9774b57-b6vbm"
Oct 31 01:16:48.628573 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:16:48.628551    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-00e840-58b9774b57-b6vbm"]
Oct 31 01:16:48.631218 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:16:48.631170    4964 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod750dc732_c296_44cb_9f7f_a7f7be7b4144.slice/crio-b378eb77f8fb2bb199ffb059a6d0ee57c8a8cc4dbbbb63de3f05b15148a7303f WatchSource:0}: Error finding container b378eb77f8fb2bb199ffb059a6d0ee57c8a8cc4dbbbb63de3f05b15148a7303f: Status 404 returned error can't find the container with id b378eb77f8fb2bb199ffb059a6d0ee57c8a8cc4dbbbb63de3f05b15148a7303f
Oct 31 01:16:49.122052 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:16:49.122010    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-00e840-58b9774b57-b6vbm" event={"ID":"750dc732-c296-44cb-9f7f-a7f7be7b4144","Type":"ContainerStarted","Data":"b378eb77f8fb2bb199ffb059a6d0ee57c8a8cc4dbbbb63de3f05b15148a7303f"}
Oct 31 01:16:50.130190 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:16:50.130154    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-00e840-58b9774b57-b6vbm" event={"ID":"750dc732-c296-44cb-9f7f-a7f7be7b4144","Type":"ContainerStarted","Data":"c71c70024f09d0094139d714b4881cb0c42675bd9569bd1d03955b9c01ffcdf5"}
Oct 31 01:16:50.130654 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:16:50.130240    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-00e840-58b9774b57-b6vbm"
Oct 31 01:16:50.245419 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:16:50.245352    4964 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-00e840-58b9774b57-b6vbm" podStartSLOduration=2.4085341 podStartE2EDuration="3.245306005s" totalImagesPullingTime="836.771905ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:16:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:16:50.244368025 +0000 UTC m=+778.720934402" watchObservedRunningTime="2026-10-31 01:16:50.245306005 +0000 UTC m=+778.721872406"
Oct 31 01:16:52.800036 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:16:52.799991    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:16:53.053900 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:16:53.053808    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:16:53.244934 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:16:53.244896    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:16:53.434344 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:16:53.434238    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:16:56.141610 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:16:56.141579    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-00e840-58b9774b57-b6vbm"
Oct 31 01:16:56.616038 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:16:56.615988    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:16:56.815355 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:16:56.814829    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:16:58.022292 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:16:58.022260    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:16:58.208267 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:16:58.208231    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:16:59.273648 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:16:59.273611    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:16:59.856952 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:16:59.856916    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:17:00.106126 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:17:00.106084    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:17:02.054511 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:17:02.054462    4964 patch_prober.go:29] interesting pod/e2e-worker-w2-00e840-openshift-default-59b96fb879-z5xgw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 01:17:02.054966 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:17:02.054549    4964 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-z5xgw" podUID="b8249620-2e20-4e80-a8b8-7d8e566f8973" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:17:03.205662 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:17:03.205629    4964 generic.go:356] "Generic (PLEG): container finished" podID="b8249620-2e20-4e80-a8b8-7d8e566f8973" containerID="fb10c38a790062d0c788cfe72ba32e556f4b7975316f7e9459bd3b5ada405c17" exitCode=0
Oct 31 01:17:03.206107 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:17:03.205707    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-z5xgw" event={"ID":"b8249620-2e20-4e80-a8b8-7d8e566f8973","Type":"ContainerDied","Data":"fb10c38a790062d0c788cfe72ba32e556f4b7975316f7e9459bd3b5ada405c17"}
Oct 31 01:17:03.206160 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:17:03.206135    4964 scope.go:122] "RemoveContainer" containerID="fb10c38a790062d0c788cfe72ba32e556f4b7975316f7e9459bd3b5ada405c17"
Oct 31 01:17:03.207470 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:17:03.207444    4964 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 01:17:03.207533 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:17:03.207510    4964 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 01:17:03.207569 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:17:03.207541    4964 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 01:17:03.463800 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:17:03.463706    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:17:04.222250 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:17:04.222218    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-z5xgw" event={"ID":"b8249620-2e20-4e80-a8b8-7d8e566f8973","Type":"ContainerStarted","Data":"7827d6024a864c5b388e226d2fcf57413ad5424cfeddd516ddaa4b8d1e73be9d"}
Oct 31 01:17:04.852498 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:17:04.852466    4964 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-z5xgw"
Oct 31 01:17:04.852676 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:17:04.852653    4964 patch_prober.go:29] interesting pod/e2e-worker-w2-00e840-openshift-default-59b96fb879-z5xgw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" start-of-body=
Oct 31 01:17:04.852734 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:17:04.852696    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-z5xgw" podUID="b8249620-2e20-4e80-a8b8-7d8e566f8973" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused"
Oct 31 01:17:05.120746 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:17:05.120660    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:17:05.853551 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:17:05.853509    4964 patch_prober.go:29] interesting pod/e2e-worker-w2-00e840-openshift-default-59b96fb879-z5xgw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" start-of-body=
Oct 31 01:17:05.854043 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:17:05.853565    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-z5xgw" podUID="b8249620-2e20-4e80-a8b8-7d8e566f8973" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused"
Oct 31 01:17:06.853083 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:17:06.853051    4964 patch_prober.go:29] interesting pod/e2e-worker-w2-00e840-openshift-default-59b96fb879-z5xgw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" start-of-body=
Oct 31 01:17:06.853261 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:17:06.853112    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-z5xgw" podUID="b8249620-2e20-4e80-a8b8-7d8e566f8973" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused"
Oct 31 01:17:07.220667 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:17:07.220629    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:17:07.853138 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:17:07.853102    4964 patch_prober.go:29] interesting pod/e2e-worker-w2-00e840-openshift-default-59b96fb879-z5xgw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" start-of-body=
Oct 31 01:17:07.853351 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:17:07.853159    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-z5xgw" podUID="b8249620-2e20-4e80-a8b8-7d8e566f8973" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused"
Oct 31 01:17:08.434410 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:17:08.434374    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:17:08.852816 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:17:08.852779    4964 patch_prober.go:29] interesting pod/e2e-worker-w2-00e840-openshift-default-59b96fb879-z5xgw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" start-of-body=
Oct 31 01:17:08.853029 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:17:08.852840    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-z5xgw" podUID="b8249620-2e20-4e80-a8b8-7d8e566f8973" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused"
Oct 31 01:17:09.856575 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:17:09.856545    4964 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-z5xgw"
Oct 31 01:17:09.856984 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:17:09.856964    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-z5xgw"
Oct 31 01:17:09.857800 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:17:09.857781    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-z5xgw"
Oct 31 01:17:11.237408 ip-10-0-133-197 kubenswrapper[4964]: E1031 01:17:11.237364    4964 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q56x5" containerName="extractor"
Oct 31 01:17:13.200138 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:17:13.200106    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:17:17.715490 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:17:17.715457    4964 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-057e2d-7bd65c5c5f-zq9zv"]
Oct 31 01:17:17.719745 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:17:17.719720    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-057e2d-7bd65c5c5f-zq9zv"
Oct 31 01:17:17.818298 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:17:17.818256    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1898febe-c54e-4e24-a8d9-7f7ed5fe5ada-maas-api-tls\") pod \"maas-api-e2e-worker-w0-057e2d-7bd65c5c5f-zq9zv\" (UID: \"1898febe-c54e-4e24-a8d9-7f7ed5fe5ada\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-057e2d-7bd65c5c5f-zq9zv"
Oct 31 01:17:17.818523 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:17:17.818344    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1898febe-c54e-4e24-a8d9-7f7ed5fe5ada-metrics-certs\") pod \"maas-api-e2e-worker-w0-057e2d-7bd65c5c5f-zq9zv\" (UID: \"1898febe-c54e-4e24-a8d9-7f7ed5fe5ada\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-057e2d-7bd65c5c5f-zq9zv"
Oct 31 01:17:17.818602 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:17:17.818533    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lv88h\" (UniqueName: \"kubernetes.io/projected/1898febe-c54e-4e24-a8d9-7f7ed5fe5ada-kube-api-access-lv88h\") pod \"maas-api-e2e-worker-w0-057e2d-7bd65c5c5f-zq9zv\" (UID: \"1898febe-c54e-4e24-a8d9-7f7ed5fe5ada\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-057e2d-7bd65c5c5f-zq9zv"
Oct 31 01:17:17.818664 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:17:17.818634    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1898febe-c54e-4e24-a8d9-7f7ed5fe5ada-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-057e2d-7bd65c5c5f-zq9zv\" (UID: \"1898febe-c54e-4e24-a8d9-7f7ed5fe5ada\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-057e2d-7bd65c5c5f-zq9zv"
Oct 31 01:17:17.823022 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:17:17.823003    4964 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w0-057e2d-serving-cert\""
Oct 31 01:17:17.825529 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:17:17.825510    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-057e2d-7bd65c5c5f-zq9zv"]
Oct 31 01:17:17.919725 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:17:17.919689    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1898febe-c54e-4e24-a8d9-7f7ed5fe5ada-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-057e2d-7bd65c5c5f-zq9zv\" (UID: \"1898febe-c54e-4e24-a8d9-7f7ed5fe5ada\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-057e2d-7bd65c5c5f-zq9zv"
Oct 31 01:17:17.919939 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:17:17.919788    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1898febe-c54e-4e24-a8d9-7f7ed5fe5ada-maas-api-tls\") pod \"maas-api-e2e-worker-w0-057e2d-7bd65c5c5f-zq9zv\" (UID: \"1898febe-c54e-4e24-a8d9-7f7ed5fe5ada\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-057e2d-7bd65c5c5f-zq9zv"
Oct 31 01:17:17.919939 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:17:17.919826    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1898febe-c54e-4e24-a8d9-7f7ed5fe5ada-metrics-certs\") pod \"maas-api-e2e-worker-w0-057e2d-7bd65c5c5f-zq9zv\" (UID: \"1898febe-c54e-4e24-a8d9-7f7ed5fe5ada\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-057e2d-7bd65c5c5f-zq9zv"
Oct 31 01:17:17.919939 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:17:17.919867    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lv88h\" (UniqueName: \"kubernetes.io/projected/1898febe-c54e-4e24-a8d9-7f7ed5fe5ada-kube-api-access-lv88h\") pod \"maas-api-e2e-worker-w0-057e2d-7bd65c5c5f-zq9zv\" (UID: \"1898febe-c54e-4e24-a8d9-7f7ed5fe5ada\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-057e2d-7bd65c5c5f-zq9zv"
Oct 31 01:17:17.920116 ip-10-0-133-197 kubenswrapper[4964]: E1031 01:17:17.919946    4964 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w0-057e2d-serving-cert: secret "maas-api-e2e-worker-w0-057e2d-serving-cert" not found
Oct 31 01:17:17.920116 ip-10-0-133-197 kubenswrapper[4964]: E1031 01:17:17.920024    4964 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1898febe-c54e-4e24-a8d9-7f7ed5fe5ada-maas-api-tls podName:1898febe-c54e-4e24-a8d9-7f7ed5fe5ada nodeName:}" failed. No retries permitted until 2026-10-31 01:17:18.42000469 +0000 UTC m=+806.896571062 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/1898febe-c54e-4e24-a8d9-7f7ed5fe5ada-maas-api-tls") pod "maas-api-e2e-worker-w0-057e2d-7bd65c5c5f-zq9zv" (UID: "1898febe-c54e-4e24-a8d9-7f7ed5fe5ada") : secret "maas-api-e2e-worker-w0-057e2d-serving-cert" not found
Oct 31 01:17:17.920519 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:17:17.920496    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1898febe-c54e-4e24-a8d9-7f7ed5fe5ada-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-057e2d-7bd65c5c5f-zq9zv\" (UID: \"1898febe-c54e-4e24-a8d9-7f7ed5fe5ada\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-057e2d-7bd65c5c5f-zq9zv"
Oct 31 01:17:17.922414 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:17:17.922396    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1898febe-c54e-4e24-a8d9-7f7ed5fe5ada-metrics-certs\") pod \"maas-api-e2e-worker-w0-057e2d-7bd65c5c5f-zq9zv\" (UID: \"1898febe-c54e-4e24-a8d9-7f7ed5fe5ada\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-057e2d-7bd65c5c5f-zq9zv"
Oct 31 01:17:17.968310 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:17:17.968223    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:17:17.979176 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:17:17.979154    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lv88h\" (UniqueName: \"kubernetes.io/projected/1898febe-c54e-4e24-a8d9-7f7ed5fe5ada-kube-api-access-lv88h\") pod \"maas-api-e2e-worker-w0-057e2d-7bd65c5c5f-zq9zv\" (UID: \"1898febe-c54e-4e24-a8d9-7f7ed5fe5ada\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-057e2d-7bd65c5c5f-zq9zv"
Oct 31 01:17:18.187718 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:17:18.187678    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:17:18.428990 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:17:18.428890    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1898febe-c54e-4e24-a8d9-7f7ed5fe5ada-maas-api-tls\") pod \"maas-api-e2e-worker-w0-057e2d-7bd65c5c5f-zq9zv\" (UID: \"1898febe-c54e-4e24-a8d9-7f7ed5fe5ada\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-057e2d-7bd65c5c5f-zq9zv"
Oct 31 01:17:18.431532 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:17:18.431508    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1898febe-c54e-4e24-a8d9-7f7ed5fe5ada-maas-api-tls\") pod \"maas-api-e2e-worker-w0-057e2d-7bd65c5c5f-zq9zv\" (UID: \"1898febe-c54e-4e24-a8d9-7f7ed5fe5ada\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-057e2d-7bd65c5c5f-zq9zv"
Oct 31 01:17:18.632429 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:17:18.632398    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-057e2d-7bd65c5c5f-zq9zv"
Oct 31 01:17:19.013049 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:17:19.013023    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-057e2d-7bd65c5c5f-zq9zv"]
Oct 31 01:17:19.015065 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:17:19.015035    4964 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1898febe_c54e_4e24_a8d9_7f7ed5fe5ada.slice/crio-2d5f2828966d9520e4895d38a8284d4a95879cadf75079dcbdb3605f33b35697 WatchSource:0}: Error finding container 2d5f2828966d9520e4895d38a8284d4a95879cadf75079dcbdb3605f33b35697: Status 404 returned error can't find the container with id 2d5f2828966d9520e4895d38a8284d4a95879cadf75079dcbdb3605f33b35697
Oct 31 01:17:19.308865 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:17:19.308785    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-057e2d-7bd65c5c5f-zq9zv" event={"ID":"1898febe-c54e-4e24-a8d9-7f7ed5fe5ada","Type":"ContainerStarted","Data":"2d5f2828966d9520e4895d38a8284d4a95879cadf75079dcbdb3605f33b35697"}
Oct 31 01:17:20.315986 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:17:20.315945    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-057e2d-7bd65c5c5f-zq9zv" event={"ID":"1898febe-c54e-4e24-a8d9-7f7ed5fe5ada","Type":"ContainerStarted","Data":"1945f3f29fb4009bdf005c8fa195700889536695c8a6c6ab092851a8a594f441"}
Oct 31 01:17:20.316483 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:17:20.316022    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-057e2d-7bd65c5c5f-zq9zv"
Oct 31 01:17:20.433239 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:17:20.433186    4964 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-057e2d-7bd65c5c5f-zq9zv" podStartSLOduration=2.508619375 podStartE2EDuration="3.433171715s" totalImagesPullingTime="924.55234ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:17:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:17:20.431632136 +0000 UTC m=+808.908198528" watchObservedRunningTime="2026-10-31 01:17:20.433171715 +0000 UTC m=+808.909738095"
Oct 31 01:17:23.296605 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:17:23.296569    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:17:23.483785 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:17:23.483745    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:17:24.057952 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:17:24.057914    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:17:24.245425 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:17:24.245363    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:17:26.327376 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:17:26.327346    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-057e2d-7bd65c5c5f-zq9zv"
Oct 31 01:17:28.130424 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:17:28.130389    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:17:31.777853 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:17:31.777817    4964 prober.go:122] "Probe failed" probeType="Liveness" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-197.us-west-1.compute.internal" podUID="8dea7006d41de627c8e06bf18f518df6" containerName="haproxy" probeResult="failure" output="Get \"https://172.20.0.1:6443/version\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:17:32.522166 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:17:32.522120    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:17:32.921442 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:17:32.921238    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:17:37.729950 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:17:37.729915    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:17:38.044298 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:17:38.044211    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:17:38.283445 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:17:38.283411    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:17:38.715469 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:17:38.715434    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:17:38.911926 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:17:38.911871    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:17:41.014494 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:17:41.014454    4964 patch_prober.go:29] interesting pod/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-k5bkg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.66:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 01:17:41.014904 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:17:41.014521    4964 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-k5bkg" podUID="197445f1-d262-48ee-87c5-7354132a69ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.66:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:17:41.241879 ip-10-0-133-197 kubenswrapper[4964]: E1031 01:17:41.241843    4964 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q56x5" containerName="extractor"
Oct 31 01:17:41.476134 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:17:41.476106    4964 generic.go:356] "Generic (PLEG): container finished" podID="197445f1-d262-48ee-87c5-7354132a69ba" containerID="2da91d4df5685d2274c1062cb98ca34fa297d1f9e78c8e9e4ed6e4662122a1c4" exitCode=0
Oct 31 01:17:41.476294 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:17:41.476177    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-k5bkg" event={"ID":"197445f1-d262-48ee-87c5-7354132a69ba","Type":"ContainerDied","Data":"2da91d4df5685d2274c1062cb98ca34fa297d1f9e78c8e9e4ed6e4662122a1c4"}
Oct 31 01:17:41.476584 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:17:41.476565    4964 scope.go:122] "RemoveContainer" containerID="2da91d4df5685d2274c1062cb98ca34fa297d1f9e78c8e9e4ed6e4662122a1c4"
Oct 31 01:17:41.485704 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:17:41.485565    4964 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 01:17:41.485830 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:17:41.485756    4964 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 01:17:41.485896 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:17:41.485827    4964 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 01:17:42.483467 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:17:42.483433    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-k5bkg" event={"ID":"197445f1-d262-48ee-87c5-7354132a69ba","Type":"ContainerStarted","Data":"502ae917f77852b871ed373eec53cd1752926cb6ecf3861f531da60d67d10253"}
Oct 31 01:17:42.944003 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:17:42.943915    4964 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-k5bkg"
Oct 31 01:17:42.944186 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:17:42.944159    4964 patch_prober.go:29] interesting pod/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-k5bkg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" start-of-body=
Oct 31 01:17:42.944274 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:17:42.944249    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-k5bkg" podUID="197445f1-d262-48ee-87c5-7354132a69ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused"
Oct 31 01:17:43.021036 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:17:43.020999    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:17:43.943746 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:17:43.943661    4964 patch_prober.go:29] interesting pod/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-k5bkg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" start-of-body=
Oct 31 01:17:43.943746 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:17:43.943729    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-k5bkg" podUID="197445f1-d262-48ee-87c5-7354132a69ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused"
Oct 31 01:17:44.943529 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:17:44.943488    4964 patch_prober.go:29] interesting pod/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-k5bkg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" start-of-body=
Oct 31 01:17:44.943711 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:17:44.943559    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-k5bkg" podUID="197445f1-d262-48ee-87c5-7354132a69ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused"
Oct 31 01:17:45.944168 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:17:45.944136    4964 patch_prober.go:29] interesting pod/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-k5bkg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" start-of-body=
Oct 31 01:17:45.944574 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:17:45.944196    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-k5bkg" podUID="197445f1-d262-48ee-87c5-7354132a69ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused"
Oct 31 01:17:46.943580 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:17:46.943540    4964 patch_prober.go:29] interesting pod/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-k5bkg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" start-of-body=
Oct 31 01:17:46.943798 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:17:46.943594    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-k5bkg" podUID="197445f1-d262-48ee-87c5-7354132a69ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused"
Oct 31 01:17:47.948173 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:17:47.948144    4964 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-k5bkg"
Oct 31 01:17:47.948619 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:17:47.948557    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-k5bkg"
Oct 31 01:17:47.949302 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:17:47.949286    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-k5bkg"
Oct 31 01:17:48.004357 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:17:48.004289    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:17:52.819403 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:17:52.819362    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:17:53.621454 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:17:53.621419    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:17:54.270467 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:17:54.270435    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:17:54.644227 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:17:54.644128    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:17:58.108745 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:17:58.108707    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:18:00.166003 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:18:00.165968    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:18:02.416291 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:18:02.416246    4964 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-baacee/premium-w1-baacee-kserve-69cc8c6456-7ngq9"]
Oct 31 01:18:02.419567 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:18:02.419543    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-baacee/premium-w1-baacee-kserve-69cc8c6456-7ngq9"
Oct 31 01:18:02.520810 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:18:02.520778    4964 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-baacee\"/\"kube-root-ca.crt\""
Oct 31 01:18:02.520810 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:18:02.520782    4964 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-baacee\"/\"premium-w1-baacee-kserve-self-signed-certs\""
Oct 31 01:18:02.520810 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:18:02.520790    4964 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-baacee\"/\"openshift-service-ca.crt\""
Oct 31 01:18:02.521063 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:18:02.520842    4964 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-baacee\"/\"default-dockercfg-k6jvm\""
Oct 31 01:18:02.523374 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:18:02.523354    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-baacee/premium-w1-baacee-kserve-69cc8c6456-7ngq9"]
Oct 31 01:18:02.545630 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:18:02.545599    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/15028ba0-c888-4780-b770-6acb2d03d70a-dshm\") pod \"premium-w1-baacee-kserve-69cc8c6456-7ngq9\" (UID: \"15028ba0-c888-4780-b770-6acb2d03d70a\") " pod="e2e-models-e2e-worker-w1-baacee/premium-w1-baacee-kserve-69cc8c6456-7ngq9"
Oct 31 01:18:02.545772 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:18:02.545656    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j8mwp\" (UniqueName: \"kubernetes.io/projected/15028ba0-c888-4780-b770-6acb2d03d70a-kube-api-access-j8mwp\") pod \"premium-w1-baacee-kserve-69cc8c6456-7ngq9\" (UID: \"15028ba0-c888-4780-b770-6acb2d03d70a\") " pod="e2e-models-e2e-worker-w1-baacee/premium-w1-baacee-kserve-69cc8c6456-7ngq9"
Oct 31 01:18:02.545817 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:18:02.545778    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/15028ba0-c888-4780-b770-6acb2d03d70a-model-cache\") pod \"premium-w1-baacee-kserve-69cc8c6456-7ngq9\" (UID: \"15028ba0-c888-4780-b770-6acb2d03d70a\") " pod="e2e-models-e2e-worker-w1-baacee/premium-w1-baacee-kserve-69cc8c6456-7ngq9"
Oct 31 01:18:02.545817 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:18:02.545799    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/15028ba0-c888-4780-b770-6acb2d03d70a-tmp-dir\") pod \"premium-w1-baacee-kserve-69cc8c6456-7ngq9\" (UID: \"15028ba0-c888-4780-b770-6acb2d03d70a\") " pod="e2e-models-e2e-worker-w1-baacee/premium-w1-baacee-kserve-69cc8c6456-7ngq9"
Oct 31 01:18:02.545897 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:18:02.545817    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/15028ba0-c888-4780-b770-6acb2d03d70a-home\") pod \"premium-w1-baacee-kserve-69cc8c6456-7ngq9\" (UID: \"15028ba0-c888-4780-b770-6acb2d03d70a\") " pod="e2e-models-e2e-worker-w1-baacee/premium-w1-baacee-kserve-69cc8c6456-7ngq9"
Oct 31 01:18:02.545897 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:18:02.545890    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/15028ba0-c888-4780-b770-6acb2d03d70a-tls-certs\") pod \"premium-w1-baacee-kserve-69cc8c6456-7ngq9\" (UID: \"15028ba0-c888-4780-b770-6acb2d03d70a\") " pod="e2e-models-e2e-worker-w1-baacee/premium-w1-baacee-kserve-69cc8c6456-7ngq9"
Oct 31 01:18:02.558666 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:18:02.558634    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:18:02.646848 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:18:02.646817    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j8mwp\" (UniqueName: \"kubernetes.io/projected/15028ba0-c888-4780-b770-6acb2d03d70a-kube-api-access-j8mwp\") pod \"premium-w1-baacee-kserve-69cc8c6456-7ngq9\" (UID: \"15028ba0-c888-4780-b770-6acb2d03d70a\") " pod="e2e-models-e2e-worker-w1-baacee/premium-w1-baacee-kserve-69cc8c6456-7ngq9"
Oct 31 01:18:02.647037 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:18:02.646896    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/15028ba0-c888-4780-b770-6acb2d03d70a-model-cache\") pod \"premium-w1-baacee-kserve-69cc8c6456-7ngq9\" (UID: \"15028ba0-c888-4780-b770-6acb2d03d70a\") " pod="e2e-models-e2e-worker-w1-baacee/premium-w1-baacee-kserve-69cc8c6456-7ngq9"
Oct 31 01:18:02.647037 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:18:02.646917    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/15028ba0-c888-4780-b770-6acb2d03d70a-tmp-dir\") pod \"premium-w1-baacee-kserve-69cc8c6456-7ngq9\" (UID: \"15028ba0-c888-4780-b770-6acb2d03d70a\") " pod="e2e-models-e2e-worker-w1-baacee/premium-w1-baacee-kserve-69cc8c6456-7ngq9"
Oct 31 01:18:02.647037 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:18:02.647024    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/15028ba0-c888-4780-b770-6acb2d03d70a-home\") pod \"premium-w1-baacee-kserve-69cc8c6456-7ngq9\" (UID: \"15028ba0-c888-4780-b770-6acb2d03d70a\") " pod="e2e-models-e2e-worker-w1-baacee/premium-w1-baacee-kserve-69cc8c6456-7ngq9"
Oct 31 01:18:02.647202 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:18:02.647118    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/15028ba0-c888-4780-b770-6acb2d03d70a-tls-certs\") pod \"premium-w1-baacee-kserve-69cc8c6456-7ngq9\" (UID: \"15028ba0-c888-4780-b770-6acb2d03d70a\") " pod="e2e-models-e2e-worker-w1-baacee/premium-w1-baacee-kserve-69cc8c6456-7ngq9"
Oct 31 01:18:02.647202 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:18:02.647168    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/15028ba0-c888-4780-b770-6acb2d03d70a-dshm\") pod \"premium-w1-baacee-kserve-69cc8c6456-7ngq9\" (UID: \"15028ba0-c888-4780-b770-6acb2d03d70a\") " pod="e2e-models-e2e-worker-w1-baacee/premium-w1-baacee-kserve-69cc8c6456-7ngq9"
Oct 31 01:18:02.647369 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:18:02.647302    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/15028ba0-c888-4780-b770-6acb2d03d70a-tmp-dir\") pod \"premium-w1-baacee-kserve-69cc8c6456-7ngq9\" (UID: \"15028ba0-c888-4780-b770-6acb2d03d70a\") " pod="e2e-models-e2e-worker-w1-baacee/premium-w1-baacee-kserve-69cc8c6456-7ngq9"
Oct 31 01:18:02.647435 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:18:02.647376    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/15028ba0-c888-4780-b770-6acb2d03d70a-model-cache\") pod \"premium-w1-baacee-kserve-69cc8c6456-7ngq9\" (UID: \"15028ba0-c888-4780-b770-6acb2d03d70a\") " pod="e2e-models-e2e-worker-w1-baacee/premium-w1-baacee-kserve-69cc8c6456-7ngq9"
Oct 31 01:18:02.647473 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:18:02.647453    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/15028ba0-c888-4780-b770-6acb2d03d70a-home\") pod \"premium-w1-baacee-kserve-69cc8c6456-7ngq9\" (UID: \"15028ba0-c888-4780-b770-6acb2d03d70a\") " pod="e2e-models-e2e-worker-w1-baacee/premium-w1-baacee-kserve-69cc8c6456-7ngq9"
Oct 31 01:18:02.649567 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:18:02.649547    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/15028ba0-c888-4780-b770-6acb2d03d70a-dshm\") pod \"premium-w1-baacee-kserve-69cc8c6456-7ngq9\" (UID: \"15028ba0-c888-4780-b770-6acb2d03d70a\") " pod="e2e-models-e2e-worker-w1-baacee/premium-w1-baacee-kserve-69cc8c6456-7ngq9"
Oct 31 01:18:02.649816 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:18:02.649798    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/15028ba0-c888-4780-b770-6acb2d03d70a-tls-certs\") pod \"premium-w1-baacee-kserve-69cc8c6456-7ngq9\" (UID: \"15028ba0-c888-4780-b770-6acb2d03d70a\") " pod="e2e-models-e2e-worker-w1-baacee/premium-w1-baacee-kserve-69cc8c6456-7ngq9"
Oct 31 01:18:02.704697 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:18:02.704622    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j8mwp\" (UniqueName: \"kubernetes.io/projected/15028ba0-c888-4780-b770-6acb2d03d70a-kube-api-access-j8mwp\") pod \"premium-w1-baacee-kserve-69cc8c6456-7ngq9\" (UID: \"15028ba0-c888-4780-b770-6acb2d03d70a\") " pod="e2e-models-e2e-worker-w1-baacee/premium-w1-baacee-kserve-69cc8c6456-7ngq9"
Oct 31 01:18:02.730495 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:18:02.730466    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-baacee/premium-w1-baacee-kserve-69cc8c6456-7ngq9"
Oct 31 01:18:02.923600 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:18:02.923559    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:18:03.108691 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:18:03.108662    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-baacee/premium-w1-baacee-kserve-69cc8c6456-7ngq9"]
Oct 31 01:18:03.110754 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:18:03.110724    4964 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod15028ba0_c888_4780_b770_6acb2d03d70a.slice/crio-188ad780121fb18f71f1a0cddf3b70c8d3365e501e55a1e2c993554d84e5b1b2 WatchSource:0}: Error finding container 188ad780121fb18f71f1a0cddf3b70c8d3365e501e55a1e2c993554d84e5b1b2: Status 404 returned error can't find the container with id 188ad780121fb18f71f1a0cddf3b70c8d3365e501e55a1e2c993554d84e5b1b2
Oct 31 01:18:03.638548 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:18:03.638512    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-baacee/premium-w1-baacee-kserve-69cc8c6456-7ngq9" event={"ID":"15028ba0-c888-4780-b770-6acb2d03d70a","Type":"ContainerStarted","Data":"d05e6b37a0478286c4234afe2c8f31db97c13a366644531daa3f5569b46a2c63"}
Oct 31 01:18:03.638548 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:18:03.638553    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-baacee/premium-w1-baacee-kserve-69cc8c6456-7ngq9" event={"ID":"15028ba0-c888-4780-b770-6acb2d03d70a","Type":"ContainerStarted","Data":"188ad780121fb18f71f1a0cddf3b70c8d3365e501e55a1e2c993554d84e5b1b2"}
Oct 31 01:18:03.731068 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:18:03.731034    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-baacee/premium-w1-baacee-kserve-69cc8c6456-7ngq9"
Oct 31 01:18:03.754986 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:18:03.754930    4964 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-baacee/premium-w1-baacee-kserve-69cc8c6456-7ngq9" podStartSLOduration=1.7549152019999998 podStartE2EDuration="1.754915202s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:18:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:18:03.752589103 +0000 UTC m=+852.229155484" watchObservedRunningTime="2026-10-31 01:18:03.754915202 +0000 UTC m=+852.231481583"
Oct 31 01:18:05.473972 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:18:05.473926    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:18:07.819103 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:18:07.819070    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:18:11.238622 ip-10-0-133-197 kubenswrapper[4964]: E1031 01:18:11.238586    4964 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q56x5" containerName="extractor"
Oct 31 01:18:11.634975 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:18:11.634851    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:18:12.700544 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:18:12.699991    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:18:12.730977 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:18:12.730949    4964 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-baacee/premium-w1-baacee-kserve-69cc8c6456-7ngq9"
Oct 31 01:18:12.748267 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:18:12.748216    4964 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-baacee/premium-w1-baacee-kserve-69cc8c6456-7ngq9"
Oct 31 01:18:13.707186 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:18:13.707156    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-baacee/premium-w1-baacee-kserve-69cc8c6456-7ngq9"
Oct 31 01:18:17.500271 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:18:17.500237    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:18:19.136539 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:18:19.136500    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:18:22.337258 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:18:22.337224    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:18:27.208369 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:18:27.208310    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:18:32.016020 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:18:32.015987    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:18:36.823759 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:18:36.823607    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:18:41.236728 ip-10-0-133-197 kubenswrapper[4964]: E1031 01:18:41.236691    4964 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q56x5" containerName="extractor"
Oct 31 01:18:41.611848 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:18:41.611749    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:18:45.943546 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:18:45.943514    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:18:46.404458 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:18:46.404420    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:18:47.342242 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:18:47.342212    4964 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-00e840/premium-w2-00e840-kserve-6f6d876787-cv7w8"]
Oct 31 01:18:47.346206 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:18:47.346184    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-00e840/premium-w2-00e840-kserve-6f6d876787-cv7w8"
Oct 31 01:18:47.398626 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:18:47.398594    4964 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-00e840\"/\"kube-root-ca.crt\""
Oct 31 01:18:47.398626 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:18:47.398599    4964 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-00e840\"/\"openshift-service-ca.crt\""
Oct 31 01:18:47.448008 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:18:47.447975    4964 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-00e840\"/\"default-dockercfg-bx7rh\""
Oct 31 01:18:47.448168 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:18:47.448055    4964 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-00e840\"/\"premium-w2-00e840-kserve-self-signed-certs\""
Oct 31 01:18:47.450822 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:18:47.450801    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-00e840/premium-w2-00e840-kserve-6f6d876787-cv7w8"]
Oct 31 01:18:47.472002 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:18:47.471967    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:18:47.496605 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:18:47.496571    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x6tqs\" (UniqueName: \"kubernetes.io/projected/b8c12d5f-8ada-4c53-880c-446f5f0b5998-kube-api-access-x6tqs\") pod \"premium-w2-00e840-kserve-6f6d876787-cv7w8\" (UID: \"b8c12d5f-8ada-4c53-880c-446f5f0b5998\") " pod="e2e-models-e2e-worker-w2-00e840/premium-w2-00e840-kserve-6f6d876787-cv7w8"
Oct 31 01:18:47.496801 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:18:47.496628    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b8c12d5f-8ada-4c53-880c-446f5f0b5998-tmp-dir\") pod \"premium-w2-00e840-kserve-6f6d876787-cv7w8\" (UID: \"b8c12d5f-8ada-4c53-880c-446f5f0b5998\") " pod="e2e-models-e2e-worker-w2-00e840/premium-w2-00e840-kserve-6f6d876787-cv7w8"
Oct 31 01:18:47.496801 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:18:47.496745    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b8c12d5f-8ada-4c53-880c-446f5f0b5998-tls-certs\") pod \"premium-w2-00e840-kserve-6f6d876787-cv7w8\" (UID: \"b8c12d5f-8ada-4c53-880c-446f5f0b5998\") " pod="e2e-models-e2e-worker-w2-00e840/premium-w2-00e840-kserve-6f6d876787-cv7w8"
Oct 31 01:18:47.496896 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:18:47.496834    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b8c12d5f-8ada-4c53-880c-446f5f0b5998-model-cache\") pod \"premium-w2-00e840-kserve-6f6d876787-cv7w8\" (UID: \"b8c12d5f-8ada-4c53-880c-446f5f0b5998\") " pod="e2e-models-e2e-worker-w2-00e840/premium-w2-00e840-kserve-6f6d876787-cv7w8"
Oct 31 01:18:47.496967 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:18:47.496950    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b8c12d5f-8ada-4c53-880c-446f5f0b5998-home\") pod \"premium-w2-00e840-kserve-6f6d876787-cv7w8\" (UID: \"b8c12d5f-8ada-4c53-880c-446f5f0b5998\") " pod="e2e-models-e2e-worker-w2-00e840/premium-w2-00e840-kserve-6f6d876787-cv7w8"
Oct 31 01:18:47.497014 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:18:47.496998    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b8c12d5f-8ada-4c53-880c-446f5f0b5998-dshm\") pod \"premium-w2-00e840-kserve-6f6d876787-cv7w8\" (UID: \"b8c12d5f-8ada-4c53-880c-446f5f0b5998\") " pod="e2e-models-e2e-worker-w2-00e840/premium-w2-00e840-kserve-6f6d876787-cv7w8"
Oct 31 01:18:47.597845 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:18:47.597758    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b8c12d5f-8ada-4c53-880c-446f5f0b5998-dshm\") pod \"premium-w2-00e840-kserve-6f6d876787-cv7w8\" (UID: \"b8c12d5f-8ada-4c53-880c-446f5f0b5998\") " pod="e2e-models-e2e-worker-w2-00e840/premium-w2-00e840-kserve-6f6d876787-cv7w8"
Oct 31 01:18:47.597845 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:18:47.597810    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x6tqs\" (UniqueName: \"kubernetes.io/projected/b8c12d5f-8ada-4c53-880c-446f5f0b5998-kube-api-access-x6tqs\") pod \"premium-w2-00e840-kserve-6f6d876787-cv7w8\" (UID: \"b8c12d5f-8ada-4c53-880c-446f5f0b5998\") " pod="e2e-models-e2e-worker-w2-00e840/premium-w2-00e840-kserve-6f6d876787-cv7w8"
Oct 31 01:18:47.598084 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:18:47.597864    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b8c12d5f-8ada-4c53-880c-446f5f0b5998-tmp-dir\") pod \"premium-w2-00e840-kserve-6f6d876787-cv7w8\" (UID: \"b8c12d5f-8ada-4c53-880c-446f5f0b5998\") " pod="e2e-models-e2e-worker-w2-00e840/premium-w2-00e840-kserve-6f6d876787-cv7w8"
Oct 31 01:18:47.598084 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:18:47.598008    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b8c12d5f-8ada-4c53-880c-446f5f0b5998-tls-certs\") pod \"premium-w2-00e840-kserve-6f6d876787-cv7w8\" (UID: \"b8c12d5f-8ada-4c53-880c-446f5f0b5998\") " pod="e2e-models-e2e-worker-w2-00e840/premium-w2-00e840-kserve-6f6d876787-cv7w8"
Oct 31 01:18:47.598084 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:18:47.598058    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b8c12d5f-8ada-4c53-880c-446f5f0b5998-model-cache\") pod \"premium-w2-00e840-kserve-6f6d876787-cv7w8\" (UID: \"b8c12d5f-8ada-4c53-880c-446f5f0b5998\") " pod="e2e-models-e2e-worker-w2-00e840/premium-w2-00e840-kserve-6f6d876787-cv7w8"
Oct 31 01:18:47.598247 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:18:47.598158    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b8c12d5f-8ada-4c53-880c-446f5f0b5998-home\") pod \"premium-w2-00e840-kserve-6f6d876787-cv7w8\" (UID: \"b8c12d5f-8ada-4c53-880c-446f5f0b5998\") " pod="e2e-models-e2e-worker-w2-00e840/premium-w2-00e840-kserve-6f6d876787-cv7w8"
Oct 31 01:18:47.598247 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:18:47.598232    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b8c12d5f-8ada-4c53-880c-446f5f0b5998-tmp-dir\") pod \"premium-w2-00e840-kserve-6f6d876787-cv7w8\" (UID: \"b8c12d5f-8ada-4c53-880c-446f5f0b5998\") " pod="e2e-models-e2e-worker-w2-00e840/premium-w2-00e840-kserve-6f6d876787-cv7w8"
Oct 31 01:18:47.598493 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:18:47.598469    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b8c12d5f-8ada-4c53-880c-446f5f0b5998-model-cache\") pod \"premium-w2-00e840-kserve-6f6d876787-cv7w8\" (UID: \"b8c12d5f-8ada-4c53-880c-446f5f0b5998\") " pod="e2e-models-e2e-worker-w2-00e840/premium-w2-00e840-kserve-6f6d876787-cv7w8"
Oct 31 01:18:47.598563 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:18:47.598487    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b8c12d5f-8ada-4c53-880c-446f5f0b5998-home\") pod \"premium-w2-00e840-kserve-6f6d876787-cv7w8\" (UID: \"b8c12d5f-8ada-4c53-880c-446f5f0b5998\") " pod="e2e-models-e2e-worker-w2-00e840/premium-w2-00e840-kserve-6f6d876787-cv7w8"
Oct 31 01:18:47.600359 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:18:47.600337    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b8c12d5f-8ada-4c53-880c-446f5f0b5998-dshm\") pod \"premium-w2-00e840-kserve-6f6d876787-cv7w8\" (UID: \"b8c12d5f-8ada-4c53-880c-446f5f0b5998\") " pod="e2e-models-e2e-worker-w2-00e840/premium-w2-00e840-kserve-6f6d876787-cv7w8"
Oct 31 01:18:47.600730 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:18:47.600705    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b8c12d5f-8ada-4c53-880c-446f5f0b5998-tls-certs\") pod \"premium-w2-00e840-kserve-6f6d876787-cv7w8\" (UID: \"b8c12d5f-8ada-4c53-880c-446f5f0b5998\") " pod="e2e-models-e2e-worker-w2-00e840/premium-w2-00e840-kserve-6f6d876787-cv7w8"
Oct 31 01:18:47.659464 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:18:47.659433    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x6tqs\" (UniqueName: \"kubernetes.io/projected/b8c12d5f-8ada-4c53-880c-446f5f0b5998-kube-api-access-x6tqs\") pod \"premium-w2-00e840-kserve-6f6d876787-cv7w8\" (UID: \"b8c12d5f-8ada-4c53-880c-446f5f0b5998\") " pod="e2e-models-e2e-worker-w2-00e840/premium-w2-00e840-kserve-6f6d876787-cv7w8"
Oct 31 01:18:47.960013 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:18:47.959922    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-00e840/premium-w2-00e840-kserve-6f6d876787-cv7w8"
Oct 31 01:18:48.341402 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:18:48.341377    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-00e840/premium-w2-00e840-kserve-6f6d876787-cv7w8"]
Oct 31 01:18:48.342983 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:18:48.342962    4964 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb8c12d5f_8ada_4c53_880c_446f5f0b5998.slice/crio-14575c0f4994b4384b6969b59771c8cda6f4b7295fa78094d091b57b9e9607ea WatchSource:0}: Error finding container 14575c0f4994b4384b6969b59771c8cda6f4b7295fa78094d091b57b9e9607ea: Status 404 returned error can't find the container with id 14575c0f4994b4384b6969b59771c8cda6f4b7295fa78094d091b57b9e9607ea
Oct 31 01:18:48.921090 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:18:48.921057    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-00e840/premium-w2-00e840-kserve-6f6d876787-cv7w8" event={"ID":"b8c12d5f-8ada-4c53-880c-446f5f0b5998","Type":"ContainerStarted","Data":"3fd1dcaf3ee3ea29a6b8b325d1d184ff9277baa98504e22d1a089dae3b345baa"}
Oct 31 01:18:48.921090 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:18:48.921098    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-00e840/premium-w2-00e840-kserve-6f6d876787-cv7w8" event={"ID":"b8c12d5f-8ada-4c53-880c-446f5f0b5998","Type":"ContainerStarted","Data":"14575c0f4994b4384b6969b59771c8cda6f4b7295fa78094d091b57b9e9607ea"}
Oct 31 01:18:48.960650 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:18:48.960620    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-00e840/premium-w2-00e840-kserve-6f6d876787-cv7w8"
Oct 31 01:18:49.038472 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:18:49.038415    4964 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-00e840/premium-w2-00e840-kserve-6f6d876787-cv7w8" podStartSLOduration=2.038397167 podStartE2EDuration="2.038397167s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:18:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:18:49.037220316 +0000 UTC m=+897.513786697" watchObservedRunningTime="2026-10-31 01:18:49.038397167 +0000 UTC m=+897.514963551"
Oct 31 01:18:51.210814 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:18:51.210724    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:18:51.871690 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:18:51.871658    4964 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-9257n_f0c0ada7-7688-4dc2-97fc-813e4d0c4ef3/console-operator/2.log"
Oct 31 01:18:51.873655 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:18:51.873633    4964 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-9257n_f0c0ada7-7688-4dc2-97fc-813e4d0c4ef3/console-operator/2.log"
Oct 31 01:18:56.058866 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:18:56.058830    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:18:57.960661 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:18:57.960626    4964 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-00e840/premium-w2-00e840-kserve-6f6d876787-cv7w8"
Oct 31 01:18:57.974759 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:18:57.974736    4964 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-00e840/premium-w2-00e840-kserve-6f6d876787-cv7w8"
Oct 31 01:18:57.986873 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:18:57.986841    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-00e840/premium-w2-00e840-kserve-6f6d876787-cv7w8"
Oct 31 01:18:58.185016 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:18:58.184976    4964 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-jn9hq"]
Oct 31 01:18:58.185385 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:18:58.185357    4964 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-jn9hq" podUID="23f1b6ef-5cf9-4d1a-afdd-75a0c723727c" containerName="istio-proxy" containerID="cri-o://132299ecb89272458db55c532ab4f33a4f465573a9e338b86e64a0500bcca42f" gracePeriod=30
Oct 31 01:19:00.493885 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:19:00.493857    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:19:00.816178 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:19:00.816085    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:19:03.863821 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:19:03.863792    4964 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-jn9hq"
Oct 31 01:19:03.977411 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:19:03.977376    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/23f1b6ef-5cf9-4d1a-afdd-75a0c723727c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/23f1b6ef-5cf9-4d1a-afdd-75a0c723727c-istio-podinfo\") pod \"23f1b6ef-5cf9-4d1a-afdd-75a0c723727c\" (UID: \"23f1b6ef-5cf9-4d1a-afdd-75a0c723727c\") "
Oct 31 01:19:03.977609 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:19:03.977459    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/23f1b6ef-5cf9-4d1a-afdd-75a0c723727c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/23f1b6ef-5cf9-4d1a-afdd-75a0c723727c-credential-socket\") pod \"23f1b6ef-5cf9-4d1a-afdd-75a0c723727c\" (UID: \"23f1b6ef-5cf9-4d1a-afdd-75a0c723727c\") "
Oct 31 01:19:03.977609 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:19:03.977501    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/23f1b6ef-5cf9-4d1a-afdd-75a0c723727c-kube-api-access-5vj4m\" (UniqueName: \"kubernetes.io/projected/23f1b6ef-5cf9-4d1a-afdd-75a0c723727c-kube-api-access-5vj4m\") pod \"23f1b6ef-5cf9-4d1a-afdd-75a0c723727c\" (UID: \"23f1b6ef-5cf9-4d1a-afdd-75a0c723727c\") "
Oct 31 01:19:03.977609 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:19:03.977529    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/23f1b6ef-5cf9-4d1a-afdd-75a0c723727c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/23f1b6ef-5cf9-4d1a-afdd-75a0c723727c-istio-data\") pod \"23f1b6ef-5cf9-4d1a-afdd-75a0c723727c\" (UID: \"23f1b6ef-5cf9-4d1a-afdd-75a0c723727c\") "
Oct 31 01:19:03.977609 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:19:03.977600    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/23f1b6ef-5cf9-4d1a-afdd-75a0c723727c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/23f1b6ef-5cf9-4d1a-afdd-75a0c723727c-workload-socket\") pod \"23f1b6ef-5cf9-4d1a-afdd-75a0c723727c\" (UID: \"23f1b6ef-5cf9-4d1a-afdd-75a0c723727c\") "
Oct 31 01:19:03.977821 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:19:03.977736    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/23f1b6ef-5cf9-4d1a-afdd-75a0c723727c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/23f1b6ef-5cf9-4d1a-afdd-75a0c723727c-istio-envoy\") pod \"23f1b6ef-5cf9-4d1a-afdd-75a0c723727c\" (UID: \"23f1b6ef-5cf9-4d1a-afdd-75a0c723727c\") "
Oct 31 01:19:03.977821 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:19:03.977765    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/23f1b6ef-5cf9-4d1a-afdd-75a0c723727c-istio-token\" (UniqueName: \"kubernetes.io/projected/23f1b6ef-5cf9-4d1a-afdd-75a0c723727c-istio-token\") pod \"23f1b6ef-5cf9-4d1a-afdd-75a0c723727c\" (UID: \"23f1b6ef-5cf9-4d1a-afdd-75a0c723727c\") "
Oct 31 01:19:03.977918 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:19:03.977797    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23f1b6ef-5cf9-4d1a-afdd-75a0c723727c-credential-socket" pod "23f1b6ef-5cf9-4d1a-afdd-75a0c723727c" (UID: "23f1b6ef-5cf9-4d1a-afdd-75a0c723727c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:19:03.977918 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:19:03.977843    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/23f1b6ef-5cf9-4d1a-afdd-75a0c723727c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/23f1b6ef-5cf9-4d1a-afdd-75a0c723727c-istiod-ca-cert\") pod \"23f1b6ef-5cf9-4d1a-afdd-75a0c723727c\" (UID: \"23f1b6ef-5cf9-4d1a-afdd-75a0c723727c\") "
Oct 31 01:19:03.977918 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:19:03.977891    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/23f1b6ef-5cf9-4d1a-afdd-75a0c723727c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/23f1b6ef-5cf9-4d1a-afdd-75a0c723727c-workload-certs\") pod \"23f1b6ef-5cf9-4d1a-afdd-75a0c723727c\" (UID: \"23f1b6ef-5cf9-4d1a-afdd-75a0c723727c\") "
Oct 31 01:19:03.978067 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:19:03.977946    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23f1b6ef-5cf9-4d1a-afdd-75a0c723727c-workload-socket" pod "23f1b6ef-5cf9-4d1a-afdd-75a0c723727c" (UID: "23f1b6ef-5cf9-4d1a-afdd-75a0c723727c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:19:03.978067 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:19:03.978009    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23f1b6ef-5cf9-4d1a-afdd-75a0c723727c-istio-data" pod "23f1b6ef-5cf9-4d1a-afdd-75a0c723727c" (UID: "23f1b6ef-5cf9-4d1a-afdd-75a0c723727c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:19:03.978172 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:19:03.978120    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/23f1b6ef-5cf9-4d1a-afdd-75a0c723727c-istiod-ca-cert" pod "23f1b6ef-5cf9-4d1a-afdd-75a0c723727c" (UID: "23f1b6ef-5cf9-4d1a-afdd-75a0c723727c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:19:03.978251 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:19:03.978219    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23f1b6ef-5cf9-4d1a-afdd-75a0c723727c-workload-certs" pod "23f1b6ef-5cf9-4d1a-afdd-75a0c723727c" (UID: "23f1b6ef-5cf9-4d1a-afdd-75a0c723727c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:19:03.978662 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:19:03.978571    4964 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/23f1b6ef-5cf9-4d1a-afdd-75a0c723727c-istiod-ca-cert\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:19:03.978662 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:19:03.978596    4964 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/23f1b6ef-5cf9-4d1a-afdd-75a0c723727c-workload-certs\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:19:03.978662 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:19:03.978613    4964 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/23f1b6ef-5cf9-4d1a-afdd-75a0c723727c-credential-socket\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:19:03.978662 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:19:03.978626    4964 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/23f1b6ef-5cf9-4d1a-afdd-75a0c723727c-istio-data\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:19:03.978662 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:19:03.978640    4964 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/23f1b6ef-5cf9-4d1a-afdd-75a0c723727c-workload-socket\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:19:03.980165 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:19:03.980139    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/23f1b6ef-5cf9-4d1a-afdd-75a0c723727c-istio-podinfo" pod "23f1b6ef-5cf9-4d1a-afdd-75a0c723727c" (UID: "23f1b6ef-5cf9-4d1a-afdd-75a0c723727c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 01:19:03.980293 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:19:03.980195    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23f1b6ef-5cf9-4d1a-afdd-75a0c723727c-istio-envoy" pod "23f1b6ef-5cf9-4d1a-afdd-75a0c723727c" (UID: "23f1b6ef-5cf9-4d1a-afdd-75a0c723727c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:19:03.980382 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:19:03.980335    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/23f1b6ef-5cf9-4d1a-afdd-75a0c723727c-istio-token" pod "23f1b6ef-5cf9-4d1a-afdd-75a0c723727c" (UID: "23f1b6ef-5cf9-4d1a-afdd-75a0c723727c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:19:03.980520 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:19:03.980501    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/23f1b6ef-5cf9-4d1a-afdd-75a0c723727c-kube-api-access-5vj4m" pod "23f1b6ef-5cf9-4d1a-afdd-75a0c723727c" (UID: "23f1b6ef-5cf9-4d1a-afdd-75a0c723727c"). InnerVolumeSpecName "kube-api-access-5vj4m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:19:04.014719 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:19:04.014682    4964 generic.go:356] "Generic (PLEG): container finished" podID="23f1b6ef-5cf9-4d1a-afdd-75a0c723727c" containerID="132299ecb89272458db55c532ab4f33a4f465573a9e338b86e64a0500bcca42f" exitCode=0
Oct 31 01:19:04.014875 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:19:04.014747    4964 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-jn9hq"
Oct 31 01:19:04.014875 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:19:04.014768    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-jn9hq" event={"ID":"23f1b6ef-5cf9-4d1a-afdd-75a0c723727c","Type":"ContainerDied","Data":"132299ecb89272458db55c532ab4f33a4f465573a9e338b86e64a0500bcca42f"}
Oct 31 01:19:04.014875 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:19:04.014836    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-jn9hq" event={"ID":"23f1b6ef-5cf9-4d1a-afdd-75a0c723727c","Type":"ContainerDied","Data":"008b15c6027a0d7553cfeb88e439b14100336de2a7c2c03295baaa3f53e2f155"}
Oct 31 01:19:04.014875 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:19:04.014865    4964 scope.go:122] "RemoveContainer" containerID="132299ecb89272458db55c532ab4f33a4f465573a9e338b86e64a0500bcca42f"
Oct 31 01:19:04.024815 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:19:04.024791    4964 scope.go:122] "RemoveContainer" containerID="b8cc5e4779b99a864ec1494eabe62d70b306a08294a77c9d1066df055e523b2c"
Oct 31 01:19:04.040149 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:19:04.040053    4964 scope.go:122] "RemoveContainer" containerID="132299ecb89272458db55c532ab4f33a4f465573a9e338b86e64a0500bcca42f"
Oct 31 01:19:04.040538 ip-10-0-133-197 kubenswrapper[4964]: E1031 01:19:04.040510    4964 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"132299ecb89272458db55c532ab4f33a4f465573a9e338b86e64a0500bcca42f\": container with ID starting with 132299ecb89272458db55c532ab4f33a4f465573a9e338b86e64a0500bcca42f not found: ID does not exist" containerID="132299ecb89272458db55c532ab4f33a4f465573a9e338b86e64a0500bcca42f"
Oct 31 01:19:04.040645 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:19:04.040549    4964 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"132299ecb89272458db55c532ab4f33a4f465573a9e338b86e64a0500bcca42f"} err="failed to get container status \"132299ecb89272458db55c532ab4f33a4f465573a9e338b86e64a0500bcca42f\": rpc error: code = NotFound desc = could not find container \"132299ecb89272458db55c532ab4f33a4f465573a9e338b86e64a0500bcca42f\": container with ID starting with 132299ecb89272458db55c532ab4f33a4f465573a9e338b86e64a0500bcca42f not found: ID does not exist"
Oct 31 01:19:04.040645 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:19:04.040573    4964 scope.go:122] "RemoveContainer" containerID="b8cc5e4779b99a864ec1494eabe62d70b306a08294a77c9d1066df055e523b2c"
Oct 31 01:19:04.040956 ip-10-0-133-197 kubenswrapper[4964]: E1031 01:19:04.040933    4964 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b8cc5e4779b99a864ec1494eabe62d70b306a08294a77c9d1066df055e523b2c\": container with ID starting with b8cc5e4779b99a864ec1494eabe62d70b306a08294a77c9d1066df055e523b2c not found: ID does not exist" containerID="b8cc5e4779b99a864ec1494eabe62d70b306a08294a77c9d1066df055e523b2c"
Oct 31 01:19:04.041047 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:19:04.040964    4964 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b8cc5e4779b99a864ec1494eabe62d70b306a08294a77c9d1066df055e523b2c"} err="failed to get container status \"b8cc5e4779b99a864ec1494eabe62d70b306a08294a77c9d1066df055e523b2c\": rpc error: code = NotFound desc = could not find container \"b8cc5e4779b99a864ec1494eabe62d70b306a08294a77c9d1066df055e523b2c\": container with ID starting with b8cc5e4779b99a864ec1494eabe62d70b306a08294a77c9d1066df055e523b2c not found: ID does not exist"
Oct 31 01:19:04.079661 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:19:04.079635    4964 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/23f1b6ef-5cf9-4d1a-afdd-75a0c723727c-istio-envoy\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:19:04.079661 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:19:04.079653    4964 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/23f1b6ef-5cf9-4d1a-afdd-75a0c723727c-istio-token\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:19:04.079661 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:19:04.079663    4964 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/23f1b6ef-5cf9-4d1a-afdd-75a0c723727c-istio-podinfo\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:19:04.079882 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:19:04.079677    4964 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5vj4m\" (UniqueName: \"kubernetes.io/projected/23f1b6ef-5cf9-4d1a-afdd-75a0c723727c-kube-api-access-5vj4m\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:19:04.143466 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:19:04.143440    4964 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-jn9hq"]
Oct 31 01:19:04.148044 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:19:04.148020    4964 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-jn9hq"]
Oct 31 01:19:05.713373 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:19:05.713343    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:19:05.776540 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:19:05.776509    4964 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="23f1b6ef-5cf9-4d1a-afdd-75a0c723727c" path="/var/lib/kubelet/pods/23f1b6ef-5cf9-4d1a-afdd-75a0c723727c/volumes"
Oct 31 01:19:05.897077 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:19:05.897044    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:19:07.612729 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:19:07.612697    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:19:07.753096 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:19:07.753061    4964 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-7jd55"]
Oct 31 01:19:07.753581 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:19:07.753566    4964 cpu_manager.go:414] "containerMap: removing container" podUID="23f1b6ef-5cf9-4d1a-afdd-75a0c723727c" containerName="istio-proxy"
Oct 31 01:19:07.754051 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:19:07.753582    4964 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="23f1b6ef-5cf9-4d1a-afdd-75a0c723727c" containerName="istio-proxy"
Oct 31 01:19:07.754051 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:19:07.753680    4964 memory_manager.go:365] "RemoveStaleState removing state" podUID="23f1b6ef-5cf9-4d1a-afdd-75a0c723727c" containerName="istio-proxy"
Oct 31 01:19:07.756896 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:19:07.756881    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-7jd55"
Oct 31 01:19:07.808880 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:19:07.808857    4964 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-a8e157-openshift-default-dockercfg-f24hn\""
Oct 31 01:19:07.821786 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:19:07.821753    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fd5d3644-7596-44e1-844f-fdc3c68b5021-istiod-ca-cert\") pod \"e2e-worker-w3-a8e157-openshift-default-7fb96689c7-7jd55\" (UID: \"fd5d3644-7596-44e1-844f-fdc3c68b5021\") " pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-7jd55"
Oct 31 01:19:07.821919 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:19:07.821825    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fd5d3644-7596-44e1-844f-fdc3c68b5021-istio-envoy\") pod \"e2e-worker-w3-a8e157-openshift-default-7fb96689c7-7jd55\" (UID: \"fd5d3644-7596-44e1-844f-fdc3c68b5021\") " pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-7jd55"
Oct 31 01:19:07.821919 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:19:07.821860    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd5d3644-7596-44e1-844f-fdc3c68b5021-workload-socket\") pod \"e2e-worker-w3-a8e157-openshift-default-7fb96689c7-7jd55\" (UID: \"fd5d3644-7596-44e1-844f-fdc3c68b5021\") " pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-7jd55"
Oct 31 01:19:07.822045 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:19:07.821930    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fd5d3644-7596-44e1-844f-fdc3c68b5021-istio-data\") pod \"e2e-worker-w3-a8e157-openshift-default-7fb96689c7-7jd55\" (UID: \"fd5d3644-7596-44e1-844f-fdc3c68b5021\") " pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-7jd55"
Oct 31 01:19:07.822045 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:19:07.821954    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fd5d3644-7596-44e1-844f-fdc3c68b5021-istio-token\") pod \"e2e-worker-w3-a8e157-openshift-default-7fb96689c7-7jd55\" (UID: \"fd5d3644-7596-44e1-844f-fdc3c68b5021\") " pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-7jd55"
Oct 31 01:19:07.822045 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:19:07.821987    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fd5d3644-7596-44e1-844f-fdc3c68b5021-istio-podinfo\") pod \"e2e-worker-w3-a8e157-openshift-default-7fb96689c7-7jd55\" (UID: \"fd5d3644-7596-44e1-844f-fdc3c68b5021\") " pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-7jd55"
Oct 31 01:19:07.822045 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:19:07.822026    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hlbtc\" (UniqueName: \"kubernetes.io/projected/fd5d3644-7596-44e1-844f-fdc3c68b5021-kube-api-access-hlbtc\") pod \"e2e-worker-w3-a8e157-openshift-default-7fb96689c7-7jd55\" (UID: \"fd5d3644-7596-44e1-844f-fdc3c68b5021\") " pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-7jd55"
Oct 31 01:19:07.822240 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:19:07.822075    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fd5d3644-7596-44e1-844f-fdc3c68b5021-workload-certs\") pod \"e2e-worker-w3-a8e157-openshift-default-7fb96689c7-7jd55\" (UID: \"fd5d3644-7596-44e1-844f-fdc3c68b5021\") " pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-7jd55"
Oct 31 01:19:07.822240 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:19:07.822134    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd5d3644-7596-44e1-844f-fdc3c68b5021-credential-socket\") pod \"e2e-worker-w3-a8e157-openshift-default-7fb96689c7-7jd55\" (UID: \"fd5d3644-7596-44e1-844f-fdc3c68b5021\") " pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-7jd55"
Oct 31 01:19:07.862029 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:19:07.862000    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-7jd55"]
Oct 31 01:19:07.923111 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:19:07.923039    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fd5d3644-7596-44e1-844f-fdc3c68b5021-istiod-ca-cert\") pod \"e2e-worker-w3-a8e157-openshift-default-7fb96689c7-7jd55\" (UID: \"fd5d3644-7596-44e1-844f-fdc3c68b5021\") " pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-7jd55"
Oct 31 01:19:07.923111 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:19:07.923090    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fd5d3644-7596-44e1-844f-fdc3c68b5021-istio-envoy\") pod \"e2e-worker-w3-a8e157-openshift-default-7fb96689c7-7jd55\" (UID: \"fd5d3644-7596-44e1-844f-fdc3c68b5021\") " pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-7jd55"
Oct 31 01:19:07.923358 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:19:07.923115    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd5d3644-7596-44e1-844f-fdc3c68b5021-workload-socket\") pod \"e2e-worker-w3-a8e157-openshift-default-7fb96689c7-7jd55\" (UID: \"fd5d3644-7596-44e1-844f-fdc3c68b5021\") " pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-7jd55"
Oct 31 01:19:07.923358 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:19:07.923163    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fd5d3644-7596-44e1-844f-fdc3c68b5021-istio-data\") pod \"e2e-worker-w3-a8e157-openshift-default-7fb96689c7-7jd55\" (UID: \"fd5d3644-7596-44e1-844f-fdc3c68b5021\") " pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-7jd55"
Oct 31 01:19:07.923358 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:19:07.923179    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fd5d3644-7596-44e1-844f-fdc3c68b5021-istio-token\") pod \"e2e-worker-w3-a8e157-openshift-default-7fb96689c7-7jd55\" (UID: \"fd5d3644-7596-44e1-844f-fdc3c68b5021\") " pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-7jd55"
Oct 31 01:19:07.923358 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:19:07.923201    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fd5d3644-7596-44e1-844f-fdc3c68b5021-istio-podinfo\") pod \"e2e-worker-w3-a8e157-openshift-default-7fb96689c7-7jd55\" (UID: \"fd5d3644-7596-44e1-844f-fdc3c68b5021\") " pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-7jd55"
Oct 31 01:19:07.923358 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:19:07.923231    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hlbtc\" (UniqueName: \"kubernetes.io/projected/fd5d3644-7596-44e1-844f-fdc3c68b5021-kube-api-access-hlbtc\") pod \"e2e-worker-w3-a8e157-openshift-default-7fb96689c7-7jd55\" (UID: \"fd5d3644-7596-44e1-844f-fdc3c68b5021\") " pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-7jd55"
Oct 31 01:19:07.923358 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:19:07.923281    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fd5d3644-7596-44e1-844f-fdc3c68b5021-workload-certs\") pod \"e2e-worker-w3-a8e157-openshift-default-7fb96689c7-7jd55\" (UID: \"fd5d3644-7596-44e1-844f-fdc3c68b5021\") " pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-7jd55"
Oct 31 01:19:07.923358 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:19:07.923357    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd5d3644-7596-44e1-844f-fdc3c68b5021-credential-socket\") pod \"e2e-worker-w3-a8e157-openshift-default-7fb96689c7-7jd55\" (UID: \"fd5d3644-7596-44e1-844f-fdc3c68b5021\") " pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-7jd55"
Oct 31 01:19:07.923727 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:19:07.923557    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fd5d3644-7596-44e1-844f-fdc3c68b5021-istio-data\") pod \"e2e-worker-w3-a8e157-openshift-default-7fb96689c7-7jd55\" (UID: \"fd5d3644-7596-44e1-844f-fdc3c68b5021\") " pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-7jd55"
Oct 31 01:19:07.923727 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:19:07.923700    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd5d3644-7596-44e1-844f-fdc3c68b5021-workload-socket\") pod \"e2e-worker-w3-a8e157-openshift-default-7fb96689c7-7jd55\" (UID: \"fd5d3644-7596-44e1-844f-fdc3c68b5021\") " pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-7jd55"
Oct 31 01:19:07.923820 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:19:07.923768    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fd5d3644-7596-44e1-844f-fdc3c68b5021-workload-certs\") pod \"e2e-worker-w3-a8e157-openshift-default-7fb96689c7-7jd55\" (UID: \"fd5d3644-7596-44e1-844f-fdc3c68b5021\") " pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-7jd55"
Oct 31 01:19:07.923855 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:19:07.923811    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fd5d3644-7596-44e1-844f-fdc3c68b5021-istiod-ca-cert\") pod \"e2e-worker-w3-a8e157-openshift-default-7fb96689c7-7jd55\" (UID: \"fd5d3644-7596-44e1-844f-fdc3c68b5021\") " pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-7jd55"
Oct 31 01:19:07.923982 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:19:07.923964    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd5d3644-7596-44e1-844f-fdc3c68b5021-credential-socket\") pod \"e2e-worker-w3-a8e157-openshift-default-7fb96689c7-7jd55\" (UID: \"fd5d3644-7596-44e1-844f-fdc3c68b5021\") " pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-7jd55"
Oct 31 01:19:07.925846 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:19:07.925821    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fd5d3644-7596-44e1-844f-fdc3c68b5021-istio-envoy\") pod \"e2e-worker-w3-a8e157-openshift-default-7fb96689c7-7jd55\" (UID: \"fd5d3644-7596-44e1-844f-fdc3c68b5021\") " pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-7jd55"
Oct 31 01:19:07.925987 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:19:07.925965    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fd5d3644-7596-44e1-844f-fdc3c68b5021-istio-podinfo\") pod \"e2e-worker-w3-a8e157-openshift-default-7fb96689c7-7jd55\" (UID: \"fd5d3644-7596-44e1-844f-fdc3c68b5021\") " pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-7jd55"
Oct 31 01:19:07.979917 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:19:07.979887    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fd5d3644-7596-44e1-844f-fdc3c68b5021-istio-token\") pod \"e2e-worker-w3-a8e157-openshift-default-7fb96689c7-7jd55\" (UID: \"fd5d3644-7596-44e1-844f-fdc3c68b5021\") " pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-7jd55"
Oct 31 01:19:07.980044 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:19:07.979985    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hlbtc\" (UniqueName: \"kubernetes.io/projected/fd5d3644-7596-44e1-844f-fdc3c68b5021-kube-api-access-hlbtc\") pod \"e2e-worker-w3-a8e157-openshift-default-7fb96689c7-7jd55\" (UID: \"fd5d3644-7596-44e1-844f-fdc3c68b5021\") " pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-7jd55"
Oct 31 01:19:08.069940 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:19:08.069912    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-7jd55"
Oct 31 01:19:08.500642 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:19:08.500602    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-7jd55"]
Oct 31 01:19:08.538265 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:19:08.538236    4964 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd5d3644_7596_44e1_844f_fdc3c68b5021.slice/crio-1292dcba10258abc0014748ad6d7e9069b2eda01c02ef390c96a9b4925246d40 WatchSource:0}: Error finding container 1292dcba10258abc0014748ad6d7e9069b2eda01c02ef390c96a9b4925246d40: Status 404 returned error can't find the container with id 1292dcba10258abc0014748ad6d7e9069b2eda01c02ef390c96a9b4925246d40
Oct 31 01:19:08.540486 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:19:08.540451    4964 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 01:19:08.540575 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:19:08.540521    4964 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 01:19:08.540575 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:19:08.540553    4964 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 01:19:09.047743 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:19:09.047708    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-7jd55" event={"ID":"fd5d3644-7596-44e1-844f-fdc3c68b5021","Type":"ContainerStarted","Data":"6a604a4e640a88052a646705ff4256ad69d192ed642c00387523baa5279161f6"}
Oct 31 01:19:09.048250 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:19:09.047762    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-7jd55" event={"ID":"fd5d3644-7596-44e1-844f-fdc3c68b5021","Type":"ContainerStarted","Data":"1292dcba10258abc0014748ad6d7e9069b2eda01c02ef390c96a9b4925246d40"}
Oct 31 01:19:09.071156 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:19:09.071129    4964 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-7jd55"
Oct 31 01:19:09.072634 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:19:09.072606    4964 patch_prober.go:29] interesting pod/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-7jd55 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" start-of-body=
Oct 31 01:19:09.072794 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:19:09.072665    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-7jd55" podUID="fd5d3644-7596-44e1-844f-fdc3c68b5021" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused"
Oct 31 01:19:09.166727 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:19:09.166673    4964 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-7jd55" podStartSLOduration=2.166659459 podStartE2EDuration="2.166659459s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:19:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:19:09.164923633 +0000 UTC m=+917.641490014" watchObservedRunningTime="2026-10-31 01:19:09.166659459 +0000 UTC m=+917.643225838"
Oct 31 01:19:10.071220 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:19:10.071190    4964 patch_prober.go:29] interesting pod/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-7jd55 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" start-of-body=
Oct 31 01:19:10.071717 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:19:10.071256    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-7jd55" podUID="fd5d3644-7596-44e1-844f-fdc3c68b5021" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused"
Oct 31 01:19:10.513426 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:19:10.511841    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:19:11.070907 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:19:11.070866    4964 patch_prober.go:29] interesting pod/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-7jd55 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" start-of-body=
Oct 31 01:19:11.071148 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:19:11.070933    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-7jd55" podUID="fd5d3644-7596-44e1-844f-fdc3c68b5021" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused"
Oct 31 01:19:11.243813 ip-10-0-133-197 kubenswrapper[4964]: E1031 01:19:11.243759    4964 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q56x5" containerName="extractor"
Oct 31 01:19:12.070407 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:19:12.070367    4964 patch_prober.go:29] interesting pod/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-7jd55 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" start-of-body=
Oct 31 01:19:12.070587 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:19:12.070424    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-7jd55" podUID="fd5d3644-7596-44e1-844f-fdc3c68b5021" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused"
Oct 31 01:19:13.070909 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:19:13.070869    4964 patch_prober.go:29] interesting pod/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-7jd55 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" start-of-body=
Oct 31 01:19:13.071361 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:19:13.070927    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-7jd55" podUID="fd5d3644-7596-44e1-844f-fdc3c68b5021" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused"
Oct 31 01:19:14.070693 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:19:14.070659    4964 patch_prober.go:29] interesting pod/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-7jd55 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" start-of-body=
Oct 31 01:19:14.070867 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:19:14.070713    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-7jd55" podUID="fd5d3644-7596-44e1-844f-fdc3c68b5021" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused"
Oct 31 01:19:15.070701 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:19:15.070660    4964 patch_prober.go:29] interesting pod/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-7jd55 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" start-of-body=
Oct 31 01:19:15.071173 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:19:15.070730    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-7jd55" podUID="fd5d3644-7596-44e1-844f-fdc3c68b5021" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused"
Oct 31 01:19:15.336721 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:19:15.336629    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:19:16.075063 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:19:16.075031    4964 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-7jd55"
Oct 31 01:19:16.075530 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:19:16.075507    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-7jd55"
Oct 31 01:19:16.076254 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:19:16.076236    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-7jd55"
Oct 31 01:19:20.188583 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:19:20.188546    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:19:25.012693 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:19:25.012660    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:19:25.192386 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:19:25.192337    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:19:29.992671 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:19:29.992641    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:19:32.823277 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:19:32.823242    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:19:34.500365 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:19:34.500304    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:19:34.911102 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:19:34.910993    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:19:39.675757 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:19:39.675722    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:19:41.241145 ip-10-0-133-197 kubenswrapper[4964]: E1031 01:19:41.241106    4964 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q56x5" containerName="extractor"
Oct 31 01:19:44.570274 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:19:44.570229    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:19:49.397680 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:19:49.397643    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:19:54.213460 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:19:54.213413    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:19:58.977385 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:19:58.977348    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:20:03.808638 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:20:03.808601    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:20:04.376114 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:20:04.376070    4964 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-jhg75"]
Oct 31 01:20:04.376605 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:20:04.376589    4964 cpu_manager.go:414] "containerMap: removing container" podUID="23f1b6ef-5cf9-4d1a-afdd-75a0c723727c" containerName="istio-proxy"
Oct 31 01:20:04.376605 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:20:04.376605    4964 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="23f1b6ef-5cf9-4d1a-afdd-75a0c723727c" containerName="istio-proxy"
Oct 31 01:20:04.376756 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:20:04.376699    4964 memory_manager.go:365] "RemoveStaleState removing state" podUID="23f1b6ef-5cf9-4d1a-afdd-75a0c723727c" containerName="istio-proxy"
Oct 31 01:20:04.381358 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:20:04.381339    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-jhg75"
Oct 31 01:20:04.433213 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:20:04.433190    4964 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-057e2d-openshift-default-dockercfg-2bvxk\""
Oct 31 01:20:04.443776 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:20:04.443748    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e190985a-446c-409d-87bb-7926a945e43c-istio-podinfo\") pod \"e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-jhg75\" (UID: \"e190985a-446c-409d-87bb-7926a945e43c\") " pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-jhg75"
Oct 31 01:20:04.443888 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:20:04.443809    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e190985a-446c-409d-87bb-7926a945e43c-credential-socket\") pod \"e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-jhg75\" (UID: \"e190985a-446c-409d-87bb-7926a945e43c\") " pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-jhg75"
Oct 31 01:20:04.443940 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:20:04.443905    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e190985a-446c-409d-87bb-7926a945e43c-istio-envoy\") pod \"e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-jhg75\" (UID: \"e190985a-446c-409d-87bb-7926a945e43c\") " pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-jhg75"
Oct 31 01:20:04.443983 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:20:04.443940    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e190985a-446c-409d-87bb-7926a945e43c-istio-token\") pod \"e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-jhg75\" (UID: \"e190985a-446c-409d-87bb-7926a945e43c\") " pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-jhg75"
Oct 31 01:20:04.444024 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:20:04.444007    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e190985a-446c-409d-87bb-7926a945e43c-workload-certs\") pod \"e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-jhg75\" (UID: \"e190985a-446c-409d-87bb-7926a945e43c\") " pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-jhg75"
Oct 31 01:20:04.444078 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:20:04.444032    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e190985a-446c-409d-87bb-7926a945e43c-istiod-ca-cert\") pod \"e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-jhg75\" (UID: \"e190985a-446c-409d-87bb-7926a945e43c\") " pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-jhg75"
Oct 31 01:20:04.444167 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:20:04.444152    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e190985a-446c-409d-87bb-7926a945e43c-istio-data\") pod \"e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-jhg75\" (UID: \"e190985a-446c-409d-87bb-7926a945e43c\") " pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-jhg75"
Oct 31 01:20:04.444226 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:20:04.444197    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e190985a-446c-409d-87bb-7926a945e43c-workload-socket\") pod \"e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-jhg75\" (UID: \"e190985a-446c-409d-87bb-7926a945e43c\") " pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-jhg75"
Oct 31 01:20:04.444329 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:20:04.444271    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ldhx6\" (UniqueName: \"kubernetes.io/projected/e190985a-446c-409d-87bb-7926a945e43c-kube-api-access-ldhx6\") pod \"e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-jhg75\" (UID: \"e190985a-446c-409d-87bb-7926a945e43c\") " pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-jhg75"
Oct 31 01:20:04.485172 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:20:04.485145    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-jhg75"]
Oct 31 01:20:04.544941 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:20:04.544906    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e190985a-446c-409d-87bb-7926a945e43c-credential-socket\") pod \"e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-jhg75\" (UID: \"e190985a-446c-409d-87bb-7926a945e43c\") " pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-jhg75"
Oct 31 01:20:04.544941 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:20:04.544945    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e190985a-446c-409d-87bb-7926a945e43c-istio-envoy\") pod \"e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-jhg75\" (UID: \"e190985a-446c-409d-87bb-7926a945e43c\") " pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-jhg75"
Oct 31 01:20:04.545216 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:20:04.544962    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e190985a-446c-409d-87bb-7926a945e43c-istio-token\") pod \"e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-jhg75\" (UID: \"e190985a-446c-409d-87bb-7926a945e43c\") " pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-jhg75"
Oct 31 01:20:04.545216 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:20:04.544989    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e190985a-446c-409d-87bb-7926a945e43c-workload-certs\") pod \"e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-jhg75\" (UID: \"e190985a-446c-409d-87bb-7926a945e43c\") " pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-jhg75"
Oct 31 01:20:04.545216 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:20:04.545011    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e190985a-446c-409d-87bb-7926a945e43c-istiod-ca-cert\") pod \"e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-jhg75\" (UID: \"e190985a-446c-409d-87bb-7926a945e43c\") " pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-jhg75"
Oct 31 01:20:04.545216 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:20:04.545086    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e190985a-446c-409d-87bb-7926a945e43c-istio-data\") pod \"e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-jhg75\" (UID: \"e190985a-446c-409d-87bb-7926a945e43c\") " pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-jhg75"
Oct 31 01:20:04.545216 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:20:04.545118    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e190985a-446c-409d-87bb-7926a945e43c-workload-socket\") pod \"e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-jhg75\" (UID: \"e190985a-446c-409d-87bb-7926a945e43c\") " pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-jhg75"
Oct 31 01:20:04.545216 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:20:04.545156    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ldhx6\" (UniqueName: \"kubernetes.io/projected/e190985a-446c-409d-87bb-7926a945e43c-kube-api-access-ldhx6\") pod \"e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-jhg75\" (UID: \"e190985a-446c-409d-87bb-7926a945e43c\") " pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-jhg75"
Oct 31 01:20:04.545597 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:20:04.545253    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e190985a-446c-409d-87bb-7926a945e43c-istio-podinfo\") pod \"e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-jhg75\" (UID: \"e190985a-446c-409d-87bb-7926a945e43c\") " pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-jhg75"
Oct 31 01:20:04.545597 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:20:04.545382    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e190985a-446c-409d-87bb-7926a945e43c-credential-socket\") pod \"e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-jhg75\" (UID: \"e190985a-446c-409d-87bb-7926a945e43c\") " pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-jhg75"
Oct 31 01:20:04.545735 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:20:04.545713    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e190985a-446c-409d-87bb-7926a945e43c-istio-data\") pod \"e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-jhg75\" (UID: \"e190985a-446c-409d-87bb-7926a945e43c\") " pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-jhg75"
Oct 31 01:20:04.545857 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:20:04.545815    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e190985a-446c-409d-87bb-7926a945e43c-workload-certs\") pod \"e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-jhg75\" (UID: \"e190985a-446c-409d-87bb-7926a945e43c\") " pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-jhg75"
Oct 31 01:20:04.545857 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:20:04.545835    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e190985a-446c-409d-87bb-7926a945e43c-workload-socket\") pod \"e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-jhg75\" (UID: \"e190985a-446c-409d-87bb-7926a945e43c\") " pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-jhg75"
Oct 31 01:20:04.546869 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:20:04.546844    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e190985a-446c-409d-87bb-7926a945e43c-istiod-ca-cert\") pod \"e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-jhg75\" (UID: \"e190985a-446c-409d-87bb-7926a945e43c\") " pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-jhg75"
Oct 31 01:20:04.548008 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:20:04.547987    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e190985a-446c-409d-87bb-7926a945e43c-istio-envoy\") pod \"e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-jhg75\" (UID: \"e190985a-446c-409d-87bb-7926a945e43c\") " pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-jhg75"
Oct 31 01:20:04.548088 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:20:04.548024    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e190985a-446c-409d-87bb-7926a945e43c-istio-podinfo\") pod \"e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-jhg75\" (UID: \"e190985a-446c-409d-87bb-7926a945e43c\") " pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-jhg75"
Oct 31 01:20:04.601720 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:20:04.601695    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ldhx6\" (UniqueName: \"kubernetes.io/projected/e190985a-446c-409d-87bb-7926a945e43c-kube-api-access-ldhx6\") pod \"e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-jhg75\" (UID: \"e190985a-446c-409d-87bb-7926a945e43c\") " pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-jhg75"
Oct 31 01:20:04.601861 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:20:04.601807    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e190985a-446c-409d-87bb-7926a945e43c-istio-token\") pod \"e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-jhg75\" (UID: \"e190985a-446c-409d-87bb-7926a945e43c\") " pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-jhg75"
Oct 31 01:20:04.695282 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:20:04.695203    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-jhg75"
Oct 31 01:20:05.013564 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:20:05.013533    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:20:05.128224 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:20:05.128202    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-jhg75"]
Oct 31 01:20:05.129850 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:20:05.129824    4964 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode190985a_446c_409d_87bb_7926a945e43c.slice/crio-07b42df801c0b2ac6b7af98d3f2a78cdea57542ce466ad9ae025fa72af34b375 WatchSource:0}: Error finding container 07b42df801c0b2ac6b7af98d3f2a78cdea57542ce466ad9ae025fa72af34b375: Status 404 returned error can't find the container with id 07b42df801c0b2ac6b7af98d3f2a78cdea57542ce466ad9ae025fa72af34b375
Oct 31 01:20:05.132075 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:20:05.132045    4964 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 01:20:05.132138 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:20:05.132109    4964 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 01:20:05.132175 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:20:05.132135    4964 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 01:20:05.227780 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:20:05.227744    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:20:05.384593 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:20:05.384508    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-jhg75" event={"ID":"e190985a-446c-409d-87bb-7926a945e43c","Type":"ContainerStarted","Data":"e85cb26c6dce5fd1a4789974b9eccfbd2810499bb7f59de8a345a59164f6e2f2"}
Oct 31 01:20:05.384593 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:20:05.384573    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-jhg75" event={"ID":"e190985a-446c-409d-87bb-7926a945e43c","Type":"ContainerStarted","Data":"07b42df801c0b2ac6b7af98d3f2a78cdea57542ce466ad9ae025fa72af34b375"}
Oct 31 01:20:05.462889 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:20:05.462854    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:20:05.509776 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:20:05.509716    4964 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-jhg75" podStartSLOduration=1.509701661 podStartE2EDuration="1.509701661s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:20:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:20:05.507586747 +0000 UTC m=+973.984153127" watchObservedRunningTime="2026-10-31 01:20:05.509701661 +0000 UTC m=+973.986268040"
Oct 31 01:20:05.698415 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:20:05.695398    4964 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-jhg75"
Oct 31 01:20:06.186100 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:20:06.186040    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:20:06.711341 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:20:06.696893    4964 patch_prober.go:29] interesting pod/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-jhg75 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body=
Oct 31 01:20:06.711341 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:20:06.696947    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-jhg75" podUID="e190985a-446c-409d-87bb-7926a945e43c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused"
Oct 31 01:20:07.698357 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:20:07.698288    4964 patch_prober.go:29] interesting pod/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-jhg75 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body=
Oct 31 01:20:07.698937 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:20:07.698901    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-jhg75" podUID="e190985a-446c-409d-87bb-7926a945e43c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused"
Oct 31 01:20:08.633843 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:20:08.633805    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:20:08.716486 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:20:08.716452    4964 patch_prober.go:29] interesting pod/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-jhg75 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body=
Oct 31 01:20:08.717051 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:20:08.717024    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-jhg75" podUID="e190985a-446c-409d-87bb-7926a945e43c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused"
Oct 31 01:20:09.424563 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:20:09.424527    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:20:09.712391 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:20:09.705426    4964 patch_prober.go:29] interesting pod/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-jhg75 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body=
Oct 31 01:20:09.712391 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:20:09.705502    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-jhg75" podUID="e190985a-446c-409d-87bb-7926a945e43c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused"
Oct 31 01:20:10.702368 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:20:10.697214    4964 patch_prober.go:29] interesting pod/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-jhg75 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body=
Oct 31 01:20:10.702368 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:20:10.697280    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-jhg75" podUID="e190985a-446c-409d-87bb-7926a945e43c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused"
Oct 31 01:20:11.237999 ip-10-0-133-197 kubenswrapper[4964]: E1031 01:20:11.237964    4964 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q56x5" containerName="extractor"
Oct 31 01:20:12.006376 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:20:12.005535    4964 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-jhg75"
Oct 31 01:20:12.007049 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:20:12.007001    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-jhg75"
Oct 31 01:20:12.007625 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:20:12.007605    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-jhg75"
Oct 31 01:20:13.427087 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:20:13.427044    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:20:18.202085 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:20:18.202048    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:20:23.133841 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:20:23.133799    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:20:27.909333 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:20:27.909288    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:20:32.718289 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:20:32.718253    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:20:36.744544 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:20:36.744510    4964 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-55449dd944-gv2xm"]
Oct 31 01:20:36.748502 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:20:36.748487    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-55449dd944-gv2xm"
Oct 31 01:20:36.851534 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:20:36.851491    4964 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"routing-overlay\""
Oct 31 01:20:36.851534 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:20:36.851490    4964 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-plugins\""
Oct 31 01:20:36.851534 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:20:36.851521    4964 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"openshift-service-ca.crt\""
Oct 31 01:20:36.851534 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:20:36.851493    4964 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-2d1088576145\""
Oct 31 01:20:36.851813 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:20:36.851575    4964 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-dockercfg-7qstt\""
Oct 31 01:20:36.851813 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:20:36.851624    4964 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"kube-root-ca.crt\""
Oct 31 01:20:36.854224 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:20:36.854204    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-55449dd944-gv2xm"]
Oct 31 01:20:36.925124 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:20:36.925095    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5805382b-6a6c-425e-8de5-5863827f10f3-plugins-config-volume\") pod \"payload-processing-external-model-55449dd944-gv2xm\" (UID: \"5805382b-6a6c-425e-8de5-5863827f10f3\") " pod="models-as-a-service/payload-processing-external-model-55449dd944-gv2xm"
Oct 31 01:20:36.925124 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:20:36.925123    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l8ktf\" (UniqueName: \"kubernetes.io/projected/5805382b-6a6c-425e-8de5-5863827f10f3-kube-api-access-l8ktf\") pod \"payload-processing-external-model-55449dd944-gv2xm\" (UID: \"5805382b-6a6c-425e-8de5-5863827f10f3\") " pod="models-as-a-service/payload-processing-external-model-55449dd944-gv2xm"
Oct 31 01:20:36.925400 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:20:36.925283    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/5805382b-6a6c-425e-8de5-5863827f10f3-routing-overlay-volume\") pod \"payload-processing-external-model-55449dd944-gv2xm\" (UID: \"5805382b-6a6c-425e-8de5-5863827f10f3\") " pod="models-as-a-service/payload-processing-external-model-55449dd944-gv2xm"
Oct 31 01:20:36.925400 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:20:36.925347    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/5805382b-6a6c-425e-8de5-5863827f10f3-provider-credentials\") pod \"payload-processing-external-model-55449dd944-gv2xm\" (UID: \"5805382b-6a6c-425e-8de5-5863827f10f3\") " pod="models-as-a-service/payload-processing-external-model-55449dd944-gv2xm"
Oct 31 01:20:37.026749 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:20:37.026655    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/5805382b-6a6c-425e-8de5-5863827f10f3-routing-overlay-volume\") pod \"payload-processing-external-model-55449dd944-gv2xm\" (UID: \"5805382b-6a6c-425e-8de5-5863827f10f3\") " pod="models-as-a-service/payload-processing-external-model-55449dd944-gv2xm"
Oct 31 01:20:37.026749 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:20:37.026703    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/5805382b-6a6c-425e-8de5-5863827f10f3-provider-credentials\") pod \"payload-processing-external-model-55449dd944-gv2xm\" (UID: \"5805382b-6a6c-425e-8de5-5863827f10f3\") " pod="models-as-a-service/payload-processing-external-model-55449dd944-gv2xm"
Oct 31 01:20:37.026960 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:20:37.026835    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5805382b-6a6c-425e-8de5-5863827f10f3-plugins-config-volume\") pod \"payload-processing-external-model-55449dd944-gv2xm\" (UID: \"5805382b-6a6c-425e-8de5-5863827f10f3\") " pod="models-as-a-service/payload-processing-external-model-55449dd944-gv2xm"
Oct 31 01:20:37.026960 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:20:37.026861    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l8ktf\" (UniqueName: \"kubernetes.io/projected/5805382b-6a6c-425e-8de5-5863827f10f3-kube-api-access-l8ktf\") pod \"payload-processing-external-model-55449dd944-gv2xm\" (UID: \"5805382b-6a6c-425e-8de5-5863827f10f3\") " pod="models-as-a-service/payload-processing-external-model-55449dd944-gv2xm"
Oct 31 01:20:37.027503 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:20:37.027473    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/5805382b-6a6c-425e-8de5-5863827f10f3-routing-overlay-volume\") pod \"payload-processing-external-model-55449dd944-gv2xm\" (UID: \"5805382b-6a6c-425e-8de5-5863827f10f3\") " pod="models-as-a-service/payload-processing-external-model-55449dd944-gv2xm"
Oct 31 01:20:37.027726 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:20:37.027705    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5805382b-6a6c-425e-8de5-5863827f10f3-plugins-config-volume\") pod \"payload-processing-external-model-55449dd944-gv2xm\" (UID: \"5805382b-6a6c-425e-8de5-5863827f10f3\") " pod="models-as-a-service/payload-processing-external-model-55449dd944-gv2xm"
Oct 31 01:20:37.029998 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:20:37.029971    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/5805382b-6a6c-425e-8de5-5863827f10f3-provider-credentials\") pod \"payload-processing-external-model-55449dd944-gv2xm\" (UID: \"5805382b-6a6c-425e-8de5-5863827f10f3\") " pod="models-as-a-service/payload-processing-external-model-55449dd944-gv2xm"
Oct 31 01:20:37.083405 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:20:37.083378    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l8ktf\" (UniqueName: \"kubernetes.io/projected/5805382b-6a6c-425e-8de5-5863827f10f3-kube-api-access-l8ktf\") pod \"payload-processing-external-model-55449dd944-gv2xm\" (UID: \"5805382b-6a6c-425e-8de5-5863827f10f3\") " pod="models-as-a-service/payload-processing-external-model-55449dd944-gv2xm"
Oct 31 01:20:37.361885 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:20:37.361802    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-55449dd944-gv2xm"
Oct 31 01:20:37.509603 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:20:37.509561    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:20:37.743987 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:20:37.743965    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-55449dd944-gv2xm"]
Oct 31 01:20:37.746307 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:20:37.746281    4964 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5805382b_6a6c_425e_8de5_5863827f10f3.slice/crio-90218266578c5b5302d11a50110e49705528d19b4443653b8f5f4a8339fc10dc WatchSource:0}: Error finding container 90218266578c5b5302d11a50110e49705528d19b4443653b8f5f4a8339fc10dc: Status 404 returned error can't find the container with id 90218266578c5b5302d11a50110e49705528d19b4443653b8f5f4a8339fc10dc
Oct 31 01:20:38.521333 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:20:38.521279    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:20:38.616584 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:20:38.616549    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-55449dd944-gv2xm" event={"ID":"5805382b-6a6c-425e-8de5-5863827f10f3","Type":"ContainerStarted","Data":"5b6a1435b07c82f8a8115b4551ebc1f38dd34a3d45f552a542029cd1c93b7dbf"}
Oct 31 01:20:38.616782 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:20:38.616597    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-55449dd944-gv2xm" event={"ID":"5805382b-6a6c-425e-8de5-5863827f10f3","Type":"ContainerStarted","Data":"90218266578c5b5302d11a50110e49705528d19b4443653b8f5f4a8339fc10dc"}
Oct 31 01:20:38.616782 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:20:38.616741    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-55449dd944-gv2xm"
Oct 31 01:20:38.698837 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:20:38.698807    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:20:38.733236 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:20:38.732647    4964 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-55449dd944-gv2xm" podStartSLOduration=2.732625936 podStartE2EDuration="2.732625936s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:20:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:20:38.730368493 +0000 UTC m=+1007.206934874" watchObservedRunningTime="2026-10-31 01:20:38.732625936 +0000 UTC m=+1007.209192318"
Oct 31 01:20:41.239225 ip-10-0-133-197 kubenswrapper[4964]: E1031 01:20:41.239180    4964 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q56x5" containerName="extractor"
Oct 31 01:20:42.307697 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:20:42.307651    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:20:44.023236 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:20:44.023201    4964 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-klw8t"]
Oct 31 01:20:44.028379 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:20:44.028358    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-klw8t"
Oct 31 01:20:44.072927 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:20:44.072898    4964 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-b4hfr"]
Oct 31 01:20:44.079042 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:20:44.079011    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-b4hfr"
Oct 31 01:20:44.096631 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:20:44.096608    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a8754d1f-9656-460d-879e-c8d3f917492e-istio-token\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-b4hfr\" (UID: \"a8754d1f-9656-460d-879e-c8d3f917492e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-b4hfr"
Oct 31 01:20:44.096734 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:20:44.096639    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6zgqj\" (UniqueName: \"kubernetes.io/projected/a8754d1f-9656-460d-879e-c8d3f917492e-kube-api-access-6zgqj\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-b4hfr\" (UID: \"a8754d1f-9656-460d-879e-c8d3f917492e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-b4hfr"
Oct 31 01:20:44.096734 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:20:44.096664    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c15fa68c-09a2-453a-81c1-687a3ce2b87a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-klw8t\" (UID: \"c15fa68c-09a2-453a-81c1-687a3ce2b87a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-klw8t"
Oct 31 01:20:44.096829 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:20:44.096757    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8754d1f-9656-460d-879e-c8d3f917492e-credential-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-b4hfr\" (UID: \"a8754d1f-9656-460d-879e-c8d3f917492e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-b4hfr"
Oct 31 01:20:44.096829 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:20:44.096808    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c15fa68c-09a2-453a-81c1-687a3ce2b87a-istio-data\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-klw8t\" (UID: \"c15fa68c-09a2-453a-81c1-687a3ce2b87a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-klw8t"
Oct 31 01:20:44.096898 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:20:44.096843    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a8754d1f-9656-460d-879e-c8d3f917492e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-b4hfr\" (UID: \"a8754d1f-9656-460d-879e-c8d3f917492e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-b4hfr"
Oct 31 01:20:44.096931 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:20:44.096907    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a8754d1f-9656-460d-879e-c8d3f917492e-workload-certs\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-b4hfr\" (UID: \"a8754d1f-9656-460d-879e-c8d3f917492e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-b4hfr"
Oct 31 01:20:44.096964 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:20:44.096942    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c15fa68c-09a2-453a-81c1-687a3ce2b87a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-klw8t\" (UID: \"c15fa68c-09a2-453a-81c1-687a3ce2b87a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-klw8t"
Oct 31 01:20:44.096999 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:20:44.096964    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c15fa68c-09a2-453a-81c1-687a3ce2b87a-istio-token\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-klw8t\" (UID: \"c15fa68c-09a2-453a-81c1-687a3ce2b87a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-klw8t"
Oct 31 01:20:44.096999 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:20:44.096987    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a8754d1f-9656-460d-879e-c8d3f917492e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-b4hfr\" (UID: \"a8754d1f-9656-460d-879e-c8d3f917492e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-b4hfr"
Oct 31 01:20:44.097097 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:20:44.097083    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a8754d1f-9656-460d-879e-c8d3f917492e-istio-data\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-b4hfr\" (UID: \"a8754d1f-9656-460d-879e-c8d3f917492e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-b4hfr"
Oct 31 01:20:44.097139 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:20:44.097127    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8754d1f-9656-460d-879e-c8d3f917492e-workload-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-b4hfr\" (UID: \"a8754d1f-9656-460d-879e-c8d3f917492e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-b4hfr"
Oct 31 01:20:44.097172 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:20:44.097157    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zxjc6\" (UniqueName: \"kubernetes.io/projected/c15fa68c-09a2-453a-81c1-687a3ce2b87a-kube-api-access-zxjc6\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-klw8t\" (UID: \"c15fa68c-09a2-453a-81c1-687a3ce2b87a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-klw8t"
Oct 31 01:20:44.097207 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:20:44.097194    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a8754d1f-9656-460d-879e-c8d3f917492e-istio-envoy\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-b4hfr\" (UID: \"a8754d1f-9656-460d-879e-c8d3f917492e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-b4hfr"
Oct 31 01:20:44.097294 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:20:44.097270    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c15fa68c-09a2-453a-81c1-687a3ce2b87a-workload-certs\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-klw8t\" (UID: \"c15fa68c-09a2-453a-81c1-687a3ce2b87a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-klw8t"
Oct 31 01:20:44.097436 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:20:44.097307    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c15fa68c-09a2-453a-81c1-687a3ce2b87a-istio-envoy\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-klw8t\" (UID: \"c15fa68c-09a2-453a-81c1-687a3ce2b87a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-klw8t"
Oct 31 01:20:44.097436 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:20:44.097363    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c15fa68c-09a2-453a-81c1-687a3ce2b87a-workload-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-klw8t\" (UID: \"c15fa68c-09a2-453a-81c1-687a3ce2b87a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-klw8t"
Oct 31 01:20:44.097436 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:20:44.097389    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c15fa68c-09a2-453a-81c1-687a3ce2b87a-credential-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-klw8t\" (UID: \"c15fa68c-09a2-453a-81c1-687a3ce2b87a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-klw8t"
Oct 31 01:20:44.128713 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:20:44.128686    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-klw8t"]
Oct 31 01:20:44.182938 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:20:44.182908    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-b4hfr"]
Oct 31 01:20:44.198166 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:20:44.198139    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c15fa68c-09a2-453a-81c1-687a3ce2b87a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-klw8t\" (UID: \"c15fa68c-09a2-453a-81c1-687a3ce2b87a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-klw8t"
Oct 31 01:20:44.198373 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:20:44.198181    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c15fa68c-09a2-453a-81c1-687a3ce2b87a-istio-token\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-klw8t\" (UID: \"c15fa68c-09a2-453a-81c1-687a3ce2b87a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-klw8t"
Oct 31 01:20:44.198373 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:20:44.198219    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a8754d1f-9656-460d-879e-c8d3f917492e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-b4hfr\" (UID: \"a8754d1f-9656-460d-879e-c8d3f917492e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-b4hfr"
Oct 31 01:20:44.198373 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:20:44.198245    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a8754d1f-9656-460d-879e-c8d3f917492e-istio-data\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-b4hfr\" (UID: \"a8754d1f-9656-460d-879e-c8d3f917492e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-b4hfr"
Oct 31 01:20:44.198373 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:20:44.198275    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8754d1f-9656-460d-879e-c8d3f917492e-workload-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-b4hfr\" (UID: \"a8754d1f-9656-460d-879e-c8d3f917492e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-b4hfr"
Oct 31 01:20:44.198373 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:20:44.198335    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zxjc6\" (UniqueName: \"kubernetes.io/projected/c15fa68c-09a2-453a-81c1-687a3ce2b87a-kube-api-access-zxjc6\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-klw8t\" (UID: \"c15fa68c-09a2-453a-81c1-687a3ce2b87a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-klw8t"
Oct 31 01:20:44.198684 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:20:44.198398    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a8754d1f-9656-460d-879e-c8d3f917492e-istio-envoy\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-b4hfr\" (UID: \"a8754d1f-9656-460d-879e-c8d3f917492e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-b4hfr"
Oct 31 01:20:44.198684 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:20:44.198445    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c15fa68c-09a2-453a-81c1-687a3ce2b87a-workload-certs\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-klw8t\" (UID: \"c15fa68c-09a2-453a-81c1-687a3ce2b87a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-klw8t"
Oct 31 01:20:44.198684 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:20:44.198470    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c15fa68c-09a2-453a-81c1-687a3ce2b87a-istio-envoy\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-klw8t\" (UID: \"c15fa68c-09a2-453a-81c1-687a3ce2b87a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-klw8t"
Oct 31 01:20:44.198684 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:20:44.198503    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c15fa68c-09a2-453a-81c1-687a3ce2b87a-workload-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-klw8t\" (UID: \"c15fa68c-09a2-453a-81c1-687a3ce2b87a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-klw8t"
Oct 31 01:20:44.198684 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:20:44.198529    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c15fa68c-09a2-453a-81c1-687a3ce2b87a-credential-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-klw8t\" (UID: \"c15fa68c-09a2-453a-81c1-687a3ce2b87a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-klw8t"
Oct 31 01:20:44.198684 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:20:44.198560    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a8754d1f-9656-460d-879e-c8d3f917492e-istio-token\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-b4hfr\" (UID: \"a8754d1f-9656-460d-879e-c8d3f917492e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-b4hfr"
Oct 31 01:20:44.198684 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:20:44.198587    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6zgqj\" (UniqueName: \"kubernetes.io/projected/a8754d1f-9656-460d-879e-c8d3f917492e-kube-api-access-6zgqj\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-b4hfr\" (UID: \"a8754d1f-9656-460d-879e-c8d3f917492e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-b4hfr"
Oct 31 01:20:44.198684 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:20:44.198616    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c15fa68c-09a2-453a-81c1-687a3ce2b87a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-klw8t\" (UID: \"c15fa68c-09a2-453a-81c1-687a3ce2b87a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-klw8t"
Oct 31 01:20:44.198684 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:20:44.198668    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8754d1f-9656-460d-879e-c8d3f917492e-credential-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-b4hfr\" (UID: \"a8754d1f-9656-460d-879e-c8d3f917492e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-b4hfr"
Oct 31 01:20:44.199150 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:20:44.198708    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c15fa68c-09a2-453a-81c1-687a3ce2b87a-istio-data\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-klw8t\" (UID: \"c15fa68c-09a2-453a-81c1-687a3ce2b87a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-klw8t"
Oct 31 01:20:44.199150 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:20:44.198751    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a8754d1f-9656-460d-879e-c8d3f917492e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-b4hfr\" (UID: \"a8754d1f-9656-460d-879e-c8d3f917492e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-b4hfr"
Oct 31 01:20:44.199150 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:20:44.198784    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a8754d1f-9656-460d-879e-c8d3f917492e-workload-certs\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-b4hfr\" (UID: \"a8754d1f-9656-460d-879e-c8d3f917492e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-b4hfr"
Oct 31 01:20:44.199150 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:20:44.199005    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c15fa68c-09a2-453a-81c1-687a3ce2b87a-workload-certs\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-klw8t\" (UID: \"c15fa68c-09a2-453a-81c1-687a3ce2b87a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-klw8t"
Oct 31 01:20:44.199150 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:20:44.199062    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a8754d1f-9656-460d-879e-c8d3f917492e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-b4hfr\" (UID: \"a8754d1f-9656-460d-879e-c8d3f917492e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-b4hfr"
Oct 31 01:20:44.199444 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:20:44.199278    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a8754d1f-9656-460d-879e-c8d3f917492e-istio-data\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-b4hfr\" (UID: \"a8754d1f-9656-460d-879e-c8d3f917492e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-b4hfr"
Oct 31 01:20:44.199444 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:20:44.199408    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c15fa68c-09a2-453a-81c1-687a3ce2b87a-credential-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-klw8t\" (UID: \"c15fa68c-09a2-453a-81c1-687a3ce2b87a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-klw8t"
Oct 31 01:20:44.199614 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:20:44.199589    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8754d1f-9656-460d-879e-c8d3f917492e-workload-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-b4hfr\" (UID: \"a8754d1f-9656-460d-879e-c8d3f917492e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-b4hfr"
Oct 31 01:20:44.199866 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:20:44.199841    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8754d1f-9656-460d-879e-c8d3f917492e-credential-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-b4hfr\" (UID: \"a8754d1f-9656-460d-879e-c8d3f917492e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-b4hfr"
Oct 31 01:20:44.200581 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:20:44.200554    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c15fa68c-09a2-453a-81c1-687a3ce2b87a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-klw8t\" (UID: \"c15fa68c-09a2-453a-81c1-687a3ce2b87a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-klw8t"
Oct 31 01:20:44.201164 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:20:44.200927    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a8754d1f-9656-460d-879e-c8d3f917492e-workload-certs\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-b4hfr\" (UID: \"a8754d1f-9656-460d-879e-c8d3f917492e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-b4hfr"
Oct 31 01:20:44.201295 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:20:44.201032    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c15fa68c-09a2-453a-81c1-687a3ce2b87a-istio-data\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-klw8t\" (UID: \"c15fa68c-09a2-453a-81c1-687a3ce2b87a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-klw8t"
Oct 31 01:20:44.201411 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:20:44.201120    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c15fa68c-09a2-453a-81c1-687a3ce2b87a-workload-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-klw8t\" (UID: \"c15fa68c-09a2-453a-81c1-687a3ce2b87a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-klw8t"
Oct 31 01:20:44.201665 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:20:44.201629    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c15fa68c-09a2-453a-81c1-687a3ce2b87a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-klw8t\" (UID: \"c15fa68c-09a2-453a-81c1-687a3ce2b87a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-klw8t"
Oct 31 01:20:44.202108 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:20:44.202079    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c15fa68c-09a2-453a-81c1-687a3ce2b87a-istio-envoy\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-klw8t\" (UID: \"c15fa68c-09a2-453a-81c1-687a3ce2b87a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-klw8t"
Oct 31 01:20:44.202479 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:20:44.202461    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a8754d1f-9656-460d-879e-c8d3f917492e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-b4hfr\" (UID: \"a8754d1f-9656-460d-879e-c8d3f917492e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-b4hfr"
Oct 31 01:20:44.203882 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:20:44.203846    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a8754d1f-9656-460d-879e-c8d3f917492e-istio-envoy\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-b4hfr\" (UID: \"a8754d1f-9656-460d-879e-c8d3f917492e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-b4hfr"
Oct 31 01:20:44.256705 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:20:44.256668    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a8754d1f-9656-460d-879e-c8d3f917492e-istio-token\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-b4hfr\" (UID: \"a8754d1f-9656-460d-879e-c8d3f917492e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-b4hfr"
Oct 31 01:20:44.257647 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:20:44.257626    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zxjc6\" (UniqueName: \"kubernetes.io/projected/c15fa68c-09a2-453a-81c1-687a3ce2b87a-kube-api-access-zxjc6\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-klw8t\" (UID: \"c15fa68c-09a2-453a-81c1-687a3ce2b87a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-klw8t"
Oct 31 01:20:44.257765 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:20:44.257699    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c15fa68c-09a2-453a-81c1-687a3ce2b87a-istio-token\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-klw8t\" (UID: \"c15fa68c-09a2-453a-81c1-687a3ce2b87a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-klw8t"
Oct 31 01:20:44.283549 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:20:44.283484    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6zgqj\" (UniqueName: \"kubernetes.io/projected/a8754d1f-9656-460d-879e-c8d3f917492e-kube-api-access-6zgqj\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-b4hfr\" (UID: \"a8754d1f-9656-460d-879e-c8d3f917492e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-b4hfr"
Oct 31 01:20:44.342609 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:20:44.342581    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-klw8t"
Oct 31 01:20:44.393980 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:20:44.393943    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-b4hfr"
Oct 31 01:20:44.782647 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:20:44.782612    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-klw8t"]
Oct 31 01:20:44.819176 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:20:44.819145    4964 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc15fa68c_09a2_453a_81c1_687a3ce2b87a.slice/crio-ab45a736141df472f2e05c78d578cd7a8c95c71386fc02ac51abaee6d39949c0 WatchSource:0}: Error finding container ab45a736141df472f2e05c78d578cd7a8c95c71386fc02ac51abaee6d39949c0: Status 404 returned error can't find the container with id ab45a736141df472f2e05c78d578cd7a8c95c71386fc02ac51abaee6d39949c0
Oct 31 01:20:44.821418 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:20:44.821387    4964 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 01:20:44.821506 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:20:44.821464    4964 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 01:20:44.821506 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:20:44.821493    4964 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 01:20:44.836914 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:20:44.836893    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-b4hfr"]
Oct 31 01:20:44.837688 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:20:44.837660    4964 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8754d1f_9656_460d_879e_c8d3f917492e.slice/crio-bb319f0db707e4ba7769ed4b949f6b30fcb4dbc9626e3f7732a8bb6406599362 WatchSource:0}: Error finding container bb319f0db707e4ba7769ed4b949f6b30fcb4dbc9626e3f7732a8bb6406599362: Status 404 returned error can't find the container with id bb319f0db707e4ba7769ed4b949f6b30fcb4dbc9626e3f7732a8bb6406599362
Oct 31 01:20:44.839910 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:20:44.839877    4964 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 01:20:44.839985 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:20:44.839954    4964 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 01:20:44.840037 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:20:44.839988    4964 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 01:20:44.845019 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:20:44.844974    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:20:45.665098 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:20:45.665061    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-klw8t" event={"ID":"c15fa68c-09a2-453a-81c1-687a3ce2b87a","Type":"ContainerStarted","Data":"2ed0bd1cfed13a4fab558bde12f03065ff30db31105d26f8092c7fdb2935252b"}
Oct 31 01:20:45.665584 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:20:45.665127    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-klw8t" event={"ID":"c15fa68c-09a2-453a-81c1-687a3ce2b87a","Type":"ContainerStarted","Data":"ab45a736141df472f2e05c78d578cd7a8c95c71386fc02ac51abaee6d39949c0"}
Oct 31 01:20:45.666961 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:20:45.666938    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-b4hfr" event={"ID":"a8754d1f-9656-460d-879e-c8d3f917492e","Type":"ContainerStarted","Data":"f487c9e5f1066adefa450e1d49631a6c1cd66173a7c3e2ab10c9f9dbd71debf2"}
Oct 31 01:20:45.667067 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:20:45.666981    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-b4hfr" event={"ID":"a8754d1f-9656-460d-879e-c8d3f917492e","Type":"ContainerStarted","Data":"bb319f0db707e4ba7769ed4b949f6b30fcb4dbc9626e3f7732a8bb6406599362"}
Oct 31 01:20:45.907344 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:20:45.905848    4964 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-klw8t" podStartSLOduration=2.905825945 podStartE2EDuration="2.905825945s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:20:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:20:45.843212281 +0000 UTC m=+1014.319778662" watchObservedRunningTime="2026-10-31 01:20:45.905825945 +0000 UTC m=+1014.382392326"
Oct 31 01:20:45.980994 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:20:45.980842    4964 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-b4hfr" podStartSLOduration=2.98082307 podStartE2EDuration="2.98082307s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:20:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:20:45.980710269 +0000 UTC m=+1014.457276650" watchObservedRunningTime="2026-10-31 01:20:45.98082307 +0000 UTC m=+1014.457389451"
Oct 31 01:20:46.345862 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:20:46.345774    4964 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-klw8t"
Oct 31 01:20:46.347880 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:20:46.347853    4964 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-klw8t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body=
Oct 31 01:20:46.348022 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:20:46.347902    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-klw8t" podUID="c15fa68c-09a2-453a-81c1-687a3ce2b87a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused"
Oct 31 01:20:46.394764 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:20:46.394731    4964 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-b4hfr"
Oct 31 01:20:46.416964 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:20:46.416466    4964 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-b4hfr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body=
Oct 31 01:20:46.416964 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:20:46.416527    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-b4hfr" podUID="a8754d1f-9656-460d-879e-c8d3f917492e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused"
Oct 31 01:20:47.145107 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:20:47.145071    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:20:47.344340 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:20:47.343702    4964 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-klw8t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body=
Oct 31 01:20:47.344340 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:20:47.343761    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-klw8t" podUID="c15fa68c-09a2-453a-81c1-687a3ce2b87a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused"
Oct 31 01:20:47.411807 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:20:47.411717    4964 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-b4hfr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body=
Oct 31 01:20:47.412046 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:20:47.412018    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-b4hfr" podUID="a8754d1f-9656-460d-879e-c8d3f917492e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused"
Oct 31 01:20:48.347016 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:20:48.346970    4964 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-klw8t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body=
Oct 31 01:20:48.347535 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:20:48.347040    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-klw8t" podUID="c15fa68c-09a2-453a-81c1-687a3ce2b87a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused"
Oct 31 01:20:48.397349 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:20:48.396715    4964 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-b4hfr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body=
Oct 31 01:20:48.397349 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:20:48.396772    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-b4hfr" podUID="a8754d1f-9656-460d-879e-c8d3f917492e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused"
Oct 31 01:20:49.047832 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:20:49.047799    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:20:49.344399 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:20:49.343730    4964 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-klw8t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body=
Oct 31 01:20:49.344399 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:20:49.343790    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-klw8t" podUID="c15fa68c-09a2-453a-81c1-687a3ce2b87a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused"
Oct 31 01:20:49.394782 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:20:49.394749    4964 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-b4hfr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body=
Oct 31 01:20:49.395415 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:20:49.395384    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-b4hfr" podUID="a8754d1f-9656-460d-879e-c8d3f917492e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused"
Oct 31 01:20:49.636576 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:20:49.636478    4964 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-55449dd944-gv2xm" podUID="5805382b-6a6c-425e-8de5-5863827f10f3" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 01:20:50.343910 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:20:50.343864    4964 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-klw8t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body=
Oct 31 01:20:50.344086 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:20:50.343941    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-klw8t" podUID="c15fa68c-09a2-453a-81c1-687a3ce2b87a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused"
Oct 31 01:20:50.394777 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:20:50.394740    4964 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-b4hfr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body=
Oct 31 01:20:50.395075 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:20:50.395041    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-b4hfr" podUID="a8754d1f-9656-460d-879e-c8d3f917492e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused"
Oct 31 01:20:51.343572 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:20:51.343539    4964 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-klw8t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body=
Oct 31 01:20:51.343766 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:20:51.343600    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-klw8t" podUID="c15fa68c-09a2-453a-81c1-687a3ce2b87a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused"
Oct 31 01:20:51.395089 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:20:51.395052    4964 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-b4hfr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body=
Oct 31 01:20:51.395537 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:20:51.395107    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-b4hfr" podUID="a8754d1f-9656-460d-879e-c8d3f917492e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused"
Oct 31 01:20:52.019005 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:20:52.018933    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:20:52.343306 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:20:52.343216    4964 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-klw8t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body=
Oct 31 01:20:52.343306 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:20:52.343280    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-klw8t" podUID="c15fa68c-09a2-453a-81c1-687a3ce2b87a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused"
Oct 31 01:20:52.394601 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:20:52.394564    4964 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-b4hfr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body=
Oct 31 01:20:52.394775 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:20:52.394626    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-b4hfr" podUID="a8754d1f-9656-460d-879e-c8d3f917492e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused"
Oct 31 01:20:53.469179 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:20:53.469135    4964 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-klw8t"
Oct 31 01:20:53.469672 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:20:53.469548    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-klw8t"
Oct 31 01:20:53.470640 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:20:53.470414    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-klw8t"
Oct 31 01:20:53.528749 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:20:53.528717    4964 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-b4hfr"
Oct 31 01:20:53.529468 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:20:53.529448    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-b4hfr"
Oct 31 01:20:53.530819 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:20:53.530805    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-b4hfr"
Oct 31 01:20:56.828379 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:20:56.828344    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:20:59.627302 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:20:59.627262    4964 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-55449dd944-gv2xm" podUID="5805382b-6a6c-425e-8de5-5863827f10f3" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 01:21:01.839985 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:21:01.839676    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:21:05.843515 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:21:05.843479    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:21:06.515088 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:21:06.515049    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:21:06.734517 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:21:06.734479    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:21:09.626622 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:21:09.626586    4964 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-55449dd944-gv2xm" podUID="5805382b-6a6c-425e-8de5-5863827f10f3" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 01:21:11.237618 ip-10-0-133-197 kubenswrapper[4964]: E1031 01:21:11.237583    4964 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q56x5" containerName="extractor"
Oct 31 01:21:11.601292 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:21:11.601170    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:21:14.543272 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:21:14.543237    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:21:15.256390 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:21:15.256350    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:21:16.421355 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:21:16.421305    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:21:19.627011 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:21:19.626968    4964 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-55449dd944-gv2xm" podUID="5805382b-6a6c-425e-8de5-5863827f10f3" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 01:21:21.311733 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:21:21.311693    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:21:26.145212 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:21:26.145181    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:21:29.626694 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:21:29.626657    4964 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-55449dd944-gv2xm" podUID="5805382b-6a6c-425e-8de5-5863827f10f3" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 01:21:30.996601 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:21:30.996565    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:21:34.821901 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:21:34.821869    4964 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-a8e157/distinct-w3-a8e157-kserve-549559dd5c-t29tw"]
Oct 31 01:21:34.827203 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:21:34.827175    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-a8e157/distinct-w3-a8e157-kserve-549559dd5c-t29tw"
Oct 31 01:21:34.928831 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:21:34.928801    4964 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-a8e157\"/\"kube-root-ca.crt\""
Oct 31 01:21:34.929031 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:21:34.928902    4964 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-a8e157\"/\"openshift-service-ca.crt\""
Oct 31 01:21:34.929031 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:21:34.928911    4964 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-a8e157\"/\"distinct-w3-a8e157-kserve-self-signed-certs\""
Oct 31 01:21:34.929031 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:21:34.928993    4964 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-a8e157\"/\"default-dockercfg-rlpbk\""
Oct 31 01:21:34.929192 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:21:34.929102    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4b0fd818-20c2-4dd6-9846-f361c236dbd9-dshm\") pod \"distinct-w3-a8e157-kserve-549559dd5c-t29tw\" (UID: \"4b0fd818-20c2-4dd6-9846-f361c236dbd9\") " pod="e2e-models-e2e-worker-w3-a8e157/distinct-w3-a8e157-kserve-549559dd5c-t29tw"
Oct 31 01:21:34.929192 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:21:34.929127    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4b0fd818-20c2-4dd6-9846-f361c236dbd9-model-cache\") pod \"distinct-w3-a8e157-kserve-549559dd5c-t29tw\" (UID: \"4b0fd818-20c2-4dd6-9846-f361c236dbd9\") " pod="e2e-models-e2e-worker-w3-a8e157/distinct-w3-a8e157-kserve-549559dd5c-t29tw"
Oct 31 01:21:34.929297 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:21:34.929251    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4b0fd818-20c2-4dd6-9846-f361c236dbd9-tmp-dir\") pod \"distinct-w3-a8e157-kserve-549559dd5c-t29tw\" (UID: \"4b0fd818-20c2-4dd6-9846-f361c236dbd9\") " pod="e2e-models-e2e-worker-w3-a8e157/distinct-w3-a8e157-kserve-549559dd5c-t29tw"
Oct 31 01:21:34.929375 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:21:34.929299    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4b0fd818-20c2-4dd6-9846-f361c236dbd9-home\") pod \"distinct-w3-a8e157-kserve-549559dd5c-t29tw\" (UID: \"4b0fd818-20c2-4dd6-9846-f361c236dbd9\") " pod="e2e-models-e2e-worker-w3-a8e157/distinct-w3-a8e157-kserve-549559dd5c-t29tw"
Oct 31 01:21:34.929375 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:21:34.929352    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4b0fd818-20c2-4dd6-9846-f361c236dbd9-tls-certs\") pod \"distinct-w3-a8e157-kserve-549559dd5c-t29tw\" (UID: \"4b0fd818-20c2-4dd6-9846-f361c236dbd9\") " pod="e2e-models-e2e-worker-w3-a8e157/distinct-w3-a8e157-kserve-549559dd5c-t29tw"
Oct 31 01:21:34.929541 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:21:34.929435    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nlhpz\" (UniqueName: \"kubernetes.io/projected/4b0fd818-20c2-4dd6-9846-f361c236dbd9-kube-api-access-nlhpz\") pod \"distinct-w3-a8e157-kserve-549559dd5c-t29tw\" (UID: \"4b0fd818-20c2-4dd6-9846-f361c236dbd9\") " pod="e2e-models-e2e-worker-w3-a8e157/distinct-w3-a8e157-kserve-549559dd5c-t29tw"
Oct 31 01:21:34.932415 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:21:34.932394    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-a8e157/distinct-w3-a8e157-kserve-549559dd5c-t29tw"]
Oct 31 01:21:34.961863 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:21:34.961835    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:21:35.030457 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:21:35.030427    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4b0fd818-20c2-4dd6-9846-f361c236dbd9-dshm\") pod \"distinct-w3-a8e157-kserve-549559dd5c-t29tw\" (UID: \"4b0fd818-20c2-4dd6-9846-f361c236dbd9\") " pod="e2e-models-e2e-worker-w3-a8e157/distinct-w3-a8e157-kserve-549559dd5c-t29tw"
Oct 31 01:21:35.030457 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:21:35.030458    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4b0fd818-20c2-4dd6-9846-f361c236dbd9-model-cache\") pod \"distinct-w3-a8e157-kserve-549559dd5c-t29tw\" (UID: \"4b0fd818-20c2-4dd6-9846-f361c236dbd9\") " pod="e2e-models-e2e-worker-w3-a8e157/distinct-w3-a8e157-kserve-549559dd5c-t29tw"
Oct 31 01:21:35.030712 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:21:35.030661    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4b0fd818-20c2-4dd6-9846-f361c236dbd9-tmp-dir\") pod \"distinct-w3-a8e157-kserve-549559dd5c-t29tw\" (UID: \"4b0fd818-20c2-4dd6-9846-f361c236dbd9\") " pod="e2e-models-e2e-worker-w3-a8e157/distinct-w3-a8e157-kserve-549559dd5c-t29tw"
Oct 31 01:21:35.030712 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:21:35.030703    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4b0fd818-20c2-4dd6-9846-f361c236dbd9-home\") pod \"distinct-w3-a8e157-kserve-549559dd5c-t29tw\" (UID: \"4b0fd818-20c2-4dd6-9846-f361c236dbd9\") " pod="e2e-models-e2e-worker-w3-a8e157/distinct-w3-a8e157-kserve-549559dd5c-t29tw"
Oct 31 01:21:35.030825 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:21:35.030736    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4b0fd818-20c2-4dd6-9846-f361c236dbd9-tls-certs\") pod \"distinct-w3-a8e157-kserve-549559dd5c-t29tw\" (UID: \"4b0fd818-20c2-4dd6-9846-f361c236dbd9\") " pod="e2e-models-e2e-worker-w3-a8e157/distinct-w3-a8e157-kserve-549559dd5c-t29tw"
Oct 31 01:21:35.030880 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:21:35.030838    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nlhpz\" (UniqueName: \"kubernetes.io/projected/4b0fd818-20c2-4dd6-9846-f361c236dbd9-kube-api-access-nlhpz\") pod \"distinct-w3-a8e157-kserve-549559dd5c-t29tw\" (UID: \"4b0fd818-20c2-4dd6-9846-f361c236dbd9\") " pod="e2e-models-e2e-worker-w3-a8e157/distinct-w3-a8e157-kserve-549559dd5c-t29tw"
Oct 31 01:21:35.030936 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:21:35.030918    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4b0fd818-20c2-4dd6-9846-f361c236dbd9-model-cache\") pod \"distinct-w3-a8e157-kserve-549559dd5c-t29tw\" (UID: \"4b0fd818-20c2-4dd6-9846-f361c236dbd9\") " pod="e2e-models-e2e-worker-w3-a8e157/distinct-w3-a8e157-kserve-549559dd5c-t29tw"
Oct 31 01:21:35.031062 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:21:35.031041    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4b0fd818-20c2-4dd6-9846-f361c236dbd9-home\") pod \"distinct-w3-a8e157-kserve-549559dd5c-t29tw\" (UID: \"4b0fd818-20c2-4dd6-9846-f361c236dbd9\") " pod="e2e-models-e2e-worker-w3-a8e157/distinct-w3-a8e157-kserve-549559dd5c-t29tw"
Oct 31 01:21:35.031287 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:21:35.031264    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4b0fd818-20c2-4dd6-9846-f361c236dbd9-tmp-dir\") pod \"distinct-w3-a8e157-kserve-549559dd5c-t29tw\" (UID: \"4b0fd818-20c2-4dd6-9846-f361c236dbd9\") " pod="e2e-models-e2e-worker-w3-a8e157/distinct-w3-a8e157-kserve-549559dd5c-t29tw"
Oct 31 01:21:35.033019 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:21:35.032997    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4b0fd818-20c2-4dd6-9846-f361c236dbd9-dshm\") pod \"distinct-w3-a8e157-kserve-549559dd5c-t29tw\" (UID: \"4b0fd818-20c2-4dd6-9846-f361c236dbd9\") " pod="e2e-models-e2e-worker-w3-a8e157/distinct-w3-a8e157-kserve-549559dd5c-t29tw"
Oct 31 01:21:35.033401 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:21:35.033382    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4b0fd818-20c2-4dd6-9846-f361c236dbd9-tls-certs\") pod \"distinct-w3-a8e157-kserve-549559dd5c-t29tw\" (UID: \"4b0fd818-20c2-4dd6-9846-f361c236dbd9\") " pod="e2e-models-e2e-worker-w3-a8e157/distinct-w3-a8e157-kserve-549559dd5c-t29tw"
Oct 31 01:21:35.091827 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:21:35.091773    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nlhpz\" (UniqueName: \"kubernetes.io/projected/4b0fd818-20c2-4dd6-9846-f361c236dbd9-kube-api-access-nlhpz\") pod \"distinct-w3-a8e157-kserve-549559dd5c-t29tw\" (UID: \"4b0fd818-20c2-4dd6-9846-f361c236dbd9\") " pod="e2e-models-e2e-worker-w3-a8e157/distinct-w3-a8e157-kserve-549559dd5c-t29tw"
Oct 31 01:21:35.140637 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:21:35.140611    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-a8e157/distinct-w3-a8e157-kserve-549559dd5c-t29tw"
Oct 31 01:21:35.526806 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:21:35.526784    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-a8e157/distinct-w3-a8e157-kserve-549559dd5c-t29tw"]
Oct 31 01:21:35.528298 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:21:35.528264    4964 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4b0fd818_20c2_4dd6_9846_f361c236dbd9.slice/crio-d578446f99c916b93ae99a425267165a338fb47f8e574b3115cdd0215f78c5c4 WatchSource:0}: Error finding container d578446f99c916b93ae99a425267165a338fb47f8e574b3115cdd0215f78c5c4: Status 404 returned error can't find the container with id d578446f99c916b93ae99a425267165a338fb47f8e574b3115cdd0215f78c5c4
Oct 31 01:21:35.916632 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:21:35.916508    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:21:36.024735 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:21:36.024701    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-a8e157/distinct-w3-a8e157-kserve-549559dd5c-t29tw" event={"ID":"4b0fd818-20c2-4dd6-9846-f361c236dbd9","Type":"ContainerStarted","Data":"0aad8fa9926e3f2093e2880f7603cac39c6ce924e62dc29d804a5c96f7cae641"}
Oct 31 01:21:36.024932 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:21:36.024744    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-a8e157/distinct-w3-a8e157-kserve-549559dd5c-t29tw" event={"ID":"4b0fd818-20c2-4dd6-9846-f361c236dbd9","Type":"ContainerStarted","Data":"d578446f99c916b93ae99a425267165a338fb47f8e574b3115cdd0215f78c5c4"}
Oct 31 01:21:36.139285 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:21:36.139227    4964 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-a8e157/distinct-w3-a8e157-kserve-549559dd5c-t29tw" podStartSLOduration=2.139212357 podStartE2EDuration="2.139212357s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:21:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:21:36.138729916 +0000 UTC m=+1064.615296304" watchObservedRunningTime="2026-10-31 01:21:36.139212357 +0000 UTC m=+1064.615778727"
Oct 31 01:21:36.150278 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:21:36.141021    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-a8e157/distinct-w3-a8e157-kserve-549559dd5c-t29tw"
Oct 31 01:21:39.362436 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:21:39.362363    4964 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-55449dd944-gv2xm"]
Oct 31 01:21:39.363797 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:21:39.363768    4964 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-55449dd944-gv2xm" podUID="5805382b-6a6c-425e-8de5-5863827f10f3" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 01:21:39.778548 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:21:39.778520    4964 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-8f6b5b/e2e-conflict-a-8bd02cd2-kserve-7c55c85bd6-rwlfr"]
Oct 31 01:21:39.782188 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:21:39.782171    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-8f6b5b/e2e-conflict-a-8bd02cd2-kserve-7c55c85bd6-rwlfr"
Oct 31 01:21:39.863844 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:21:39.863816    4964 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-8f6b5b\"/\"kube-root-ca.crt\""
Oct 31 01:21:39.884154 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:21:39.884125    4964 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-8f6b5b\"/\"default-dockercfg-28mfd\""
Oct 31 01:21:39.884362 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:21:39.884244    4964 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-8f6b5b\"/\"openshift-service-ca.crt\""
Oct 31 01:21:39.884362 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:21:39.884244    4964 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-8f6b5b\"/\"e2e-conflict-a-8bd02cd2-kserve-self-signed-certs\""
Oct 31 01:21:39.886652 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:21:39.886611    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/766d96dd-2ba9-4216-9388-7a29c6aaf42c-dshm\") pod \"e2e-conflict-a-8bd02cd2-kserve-7c55c85bd6-rwlfr\" (UID: \"766d96dd-2ba9-4216-9388-7a29c6aaf42c\") " pod="e2e-models-e2e-worker-w4-8f6b5b/e2e-conflict-a-8bd02cd2-kserve-7c55c85bd6-rwlfr"
Oct 31 01:21:39.886652 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:21:39.886641    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/766d96dd-2ba9-4216-9388-7a29c6aaf42c-model-cache\") pod \"e2e-conflict-a-8bd02cd2-kserve-7c55c85bd6-rwlfr\" (UID: \"766d96dd-2ba9-4216-9388-7a29c6aaf42c\") " pod="e2e-models-e2e-worker-w4-8f6b5b/e2e-conflict-a-8bd02cd2-kserve-7c55c85bd6-rwlfr"
Oct 31 01:21:39.886823 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:21:39.886669    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/766d96dd-2ba9-4216-9388-7a29c6aaf42c-home\") pod \"e2e-conflict-a-8bd02cd2-kserve-7c55c85bd6-rwlfr\" (UID: \"766d96dd-2ba9-4216-9388-7a29c6aaf42c\") " pod="e2e-models-e2e-worker-w4-8f6b5b/e2e-conflict-a-8bd02cd2-kserve-7c55c85bd6-rwlfr"
Oct 31 01:21:39.886823 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:21:39.886738    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/766d96dd-2ba9-4216-9388-7a29c6aaf42c-tls-certs\") pod \"e2e-conflict-a-8bd02cd2-kserve-7c55c85bd6-rwlfr\" (UID: \"766d96dd-2ba9-4216-9388-7a29c6aaf42c\") " pod="e2e-models-e2e-worker-w4-8f6b5b/e2e-conflict-a-8bd02cd2-kserve-7c55c85bd6-rwlfr"
Oct 31 01:21:39.886910 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:21:39.886848    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vsbkz\" (UniqueName: \"kubernetes.io/projected/766d96dd-2ba9-4216-9388-7a29c6aaf42c-kube-api-access-vsbkz\") pod \"e2e-conflict-a-8bd02cd2-kserve-7c55c85bd6-rwlfr\" (UID: \"766d96dd-2ba9-4216-9388-7a29c6aaf42c\") " pod="e2e-models-e2e-worker-w4-8f6b5b/e2e-conflict-a-8bd02cd2-kserve-7c55c85bd6-rwlfr"
Oct 31 01:21:39.886910 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:21:39.886879    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/766d96dd-2ba9-4216-9388-7a29c6aaf42c-tmp-dir\") pod \"e2e-conflict-a-8bd02cd2-kserve-7c55c85bd6-rwlfr\" (UID: \"766d96dd-2ba9-4216-9388-7a29c6aaf42c\") " pod="e2e-models-e2e-worker-w4-8f6b5b/e2e-conflict-a-8bd02cd2-kserve-7c55c85bd6-rwlfr"
Oct 31 01:21:39.887237 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:21:39.887219    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-8f6b5b/e2e-conflict-a-8bd02cd2-kserve-7c55c85bd6-rwlfr"]
Oct 31 01:21:39.929681 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:21:39.929648    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:21:39.988469 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:21:39.988437    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/766d96dd-2ba9-4216-9388-7a29c6aaf42c-dshm\") pod \"e2e-conflict-a-8bd02cd2-kserve-7c55c85bd6-rwlfr\" (UID: \"766d96dd-2ba9-4216-9388-7a29c6aaf42c\") " pod="e2e-models-e2e-worker-w4-8f6b5b/e2e-conflict-a-8bd02cd2-kserve-7c55c85bd6-rwlfr"
Oct 31 01:21:39.988469 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:21:39.988471    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/766d96dd-2ba9-4216-9388-7a29c6aaf42c-model-cache\") pod \"e2e-conflict-a-8bd02cd2-kserve-7c55c85bd6-rwlfr\" (UID: \"766d96dd-2ba9-4216-9388-7a29c6aaf42c\") " pod="e2e-models-e2e-worker-w4-8f6b5b/e2e-conflict-a-8bd02cd2-kserve-7c55c85bd6-rwlfr"
Oct 31 01:21:39.988729 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:21:39.988502    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/766d96dd-2ba9-4216-9388-7a29c6aaf42c-home\") pod \"e2e-conflict-a-8bd02cd2-kserve-7c55c85bd6-rwlfr\" (UID: \"766d96dd-2ba9-4216-9388-7a29c6aaf42c\") " pod="e2e-models-e2e-worker-w4-8f6b5b/e2e-conflict-a-8bd02cd2-kserve-7c55c85bd6-rwlfr"
Oct 31 01:21:39.988729 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:21:39.988526    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/766d96dd-2ba9-4216-9388-7a29c6aaf42c-tls-certs\") pod \"e2e-conflict-a-8bd02cd2-kserve-7c55c85bd6-rwlfr\" (UID: \"766d96dd-2ba9-4216-9388-7a29c6aaf42c\") " pod="e2e-models-e2e-worker-w4-8f6b5b/e2e-conflict-a-8bd02cd2-kserve-7c55c85bd6-rwlfr"
Oct 31 01:21:39.988729 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:21:39.988617    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vsbkz\" (UniqueName: \"kubernetes.io/projected/766d96dd-2ba9-4216-9388-7a29c6aaf42c-kube-api-access-vsbkz\") pod \"e2e-conflict-a-8bd02cd2-kserve-7c55c85bd6-rwlfr\" (UID: \"766d96dd-2ba9-4216-9388-7a29c6aaf42c\") " pod="e2e-models-e2e-worker-w4-8f6b5b/e2e-conflict-a-8bd02cd2-kserve-7c55c85bd6-rwlfr"
Oct 31 01:21:39.988729 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:21:39.988654    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/766d96dd-2ba9-4216-9388-7a29c6aaf42c-tmp-dir\") pod \"e2e-conflict-a-8bd02cd2-kserve-7c55c85bd6-rwlfr\" (UID: \"766d96dd-2ba9-4216-9388-7a29c6aaf42c\") " pod="e2e-models-e2e-worker-w4-8f6b5b/e2e-conflict-a-8bd02cd2-kserve-7c55c85bd6-rwlfr"
Oct 31 01:21:39.988943 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:21:39.988913    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/766d96dd-2ba9-4216-9388-7a29c6aaf42c-home\") pod \"e2e-conflict-a-8bd02cd2-kserve-7c55c85bd6-rwlfr\" (UID: \"766d96dd-2ba9-4216-9388-7a29c6aaf42c\") " pod="e2e-models-e2e-worker-w4-8f6b5b/e2e-conflict-a-8bd02cd2-kserve-7c55c85bd6-rwlfr"
Oct 31 01:21:39.988997 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:21:39.988969    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/766d96dd-2ba9-4216-9388-7a29c6aaf42c-model-cache\") pod \"e2e-conflict-a-8bd02cd2-kserve-7c55c85bd6-rwlfr\" (UID: \"766d96dd-2ba9-4216-9388-7a29c6aaf42c\") " pod="e2e-models-e2e-worker-w4-8f6b5b/e2e-conflict-a-8bd02cd2-kserve-7c55c85bd6-rwlfr"
Oct 31 01:21:39.989055 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:21:39.989038    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/766d96dd-2ba9-4216-9388-7a29c6aaf42c-tmp-dir\") pod \"e2e-conflict-a-8bd02cd2-kserve-7c55c85bd6-rwlfr\" (UID: \"766d96dd-2ba9-4216-9388-7a29c6aaf42c\") " pod="e2e-models-e2e-worker-w4-8f6b5b/e2e-conflict-a-8bd02cd2-kserve-7c55c85bd6-rwlfr"
Oct 31 01:21:39.991170 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:21:39.991149    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/766d96dd-2ba9-4216-9388-7a29c6aaf42c-dshm\") pod \"e2e-conflict-a-8bd02cd2-kserve-7c55c85bd6-rwlfr\" (UID: \"766d96dd-2ba9-4216-9388-7a29c6aaf42c\") " pod="e2e-models-e2e-worker-w4-8f6b5b/e2e-conflict-a-8bd02cd2-kserve-7c55c85bd6-rwlfr"
Oct 31 01:21:39.991294 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:21:39.991270    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/766d96dd-2ba9-4216-9388-7a29c6aaf42c-tls-certs\") pod \"e2e-conflict-a-8bd02cd2-kserve-7c55c85bd6-rwlfr\" (UID: \"766d96dd-2ba9-4216-9388-7a29c6aaf42c\") " pod="e2e-models-e2e-worker-w4-8f6b5b/e2e-conflict-a-8bd02cd2-kserve-7c55c85bd6-rwlfr"
Oct 31 01:21:40.044874 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:21:40.044812    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vsbkz\" (UniqueName: \"kubernetes.io/projected/766d96dd-2ba9-4216-9388-7a29c6aaf42c-kube-api-access-vsbkz\") pod \"e2e-conflict-a-8bd02cd2-kserve-7c55c85bd6-rwlfr\" (UID: \"766d96dd-2ba9-4216-9388-7a29c6aaf42c\") " pod="e2e-models-e2e-worker-w4-8f6b5b/e2e-conflict-a-8bd02cd2-kserve-7c55c85bd6-rwlfr"
Oct 31 01:21:40.096454 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:21:40.096421    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-8f6b5b/e2e-conflict-a-8bd02cd2-kserve-7c55c85bd6-rwlfr"
Oct 31 01:21:40.472728 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:21:40.472701    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-8f6b5b/e2e-conflict-a-8bd02cd2-kserve-7c55c85bd6-rwlfr"]
Oct 31 01:21:40.474449 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:21:40.474421    4964 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod766d96dd_2ba9_4216_9388_7a29c6aaf42c.slice/crio-9aae13c0c963abccb67f26a39bf482885a8470ac1d3d37eb8fe2da1e4aa504cb WatchSource:0}: Error finding container 9aae13c0c963abccb67f26a39bf482885a8470ac1d3d37eb8fe2da1e4aa504cb: Status 404 returned error can't find the container with id 9aae13c0c963abccb67f26a39bf482885a8470ac1d3d37eb8fe2da1e4aa504cb
Oct 31 01:21:40.715600 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:21:40.715554    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:21:41.061306 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:21:41.061274    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-8f6b5b/e2e-conflict-a-8bd02cd2-kserve-7c55c85bd6-rwlfr" event={"ID":"766d96dd-2ba9-4216-9388-7a29c6aaf42c","Type":"ContainerStarted","Data":"6bf86c13591c3c1f4557f5340eb804a56d5aa4e2055fe659cbfb4a52450faeb9"}
Oct 31 01:21:41.061477 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:21:41.061312    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-8f6b5b/e2e-conflict-a-8bd02cd2-kserve-7c55c85bd6-rwlfr" event={"ID":"766d96dd-2ba9-4216-9388-7a29c6aaf42c","Type":"ContainerStarted","Data":"9aae13c0c963abccb67f26a39bf482885a8470ac1d3d37eb8fe2da1e4aa504cb"}
Oct 31 01:21:41.096975 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:21:41.096950    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-8f6b5b/e2e-conflict-a-8bd02cd2-kserve-7c55c85bd6-rwlfr"
Oct 31 01:21:41.177289 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:21:41.177234    4964 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-8f6b5b/e2e-conflict-a-8bd02cd2-kserve-7c55c85bd6-rwlfr" podStartSLOduration=2.1772214 podStartE2EDuration="2.1772214s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:21:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:21:41.174527062 +0000 UTC m=+1069.651093443" watchObservedRunningTime="2026-10-31 01:21:41.1772214 +0000 UTC m=+1069.653787779"
Oct 31 01:21:41.237615 ip-10-0-133-197 kubenswrapper[4964]: E1031 01:21:41.237581    4964 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q56x5" containerName="extractor"
Oct 31 01:21:44.363750 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:21:44.363705    4964 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-55449dd944-gv2xm" podUID="5805382b-6a6c-425e-8de5-5863827f10f3" containerName="payload-processing" containerID="cri-o://5b6a1435b07c82f8a8115b4551ebc1f38dd34a3d45f552a542029cd1c93b7dbf" gracePeriod=25
Oct 31 01:21:44.649512 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:21:44.649482    4964 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-55449dd944-gv2xm"
Oct 31 01:21:44.745857 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:21:44.745827    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5805382b-6a6c-425e-8de5-5863827f10f3-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/5805382b-6a6c-425e-8de5-5863827f10f3-routing-overlay-volume\") pod \"5805382b-6a6c-425e-8de5-5863827f10f3\" (UID: \"5805382b-6a6c-425e-8de5-5863827f10f3\") "
Oct 31 01:21:44.746008 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:21:44.745990    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5805382b-6a6c-425e-8de5-5863827f10f3-kube-api-access-l8ktf\" (UniqueName: \"kubernetes.io/projected/5805382b-6a6c-425e-8de5-5863827f10f3-kube-api-access-l8ktf\") pod \"5805382b-6a6c-425e-8de5-5863827f10f3\" (UID: \"5805382b-6a6c-425e-8de5-5863827f10f3\") "
Oct 31 01:21:44.746084 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:21:44.746041    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5805382b-6a6c-425e-8de5-5863827f10f3-provider-credentials\" (UniqueName: \"kubernetes.io/projected/5805382b-6a6c-425e-8de5-5863827f10f3-provider-credentials\") pod \"5805382b-6a6c-425e-8de5-5863827f10f3\" (UID: \"5805382b-6a6c-425e-8de5-5863827f10f3\") "
Oct 31 01:21:44.746140 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:21:44.746085    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5805382b-6a6c-425e-8de5-5863827f10f3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5805382b-6a6c-425e-8de5-5863827f10f3-plugins-config-volume\") pod \"5805382b-6a6c-425e-8de5-5863827f10f3\" (UID: \"5805382b-6a6c-425e-8de5-5863827f10f3\") "
Oct 31 01:21:44.746241 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:21:44.746215    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5805382b-6a6c-425e-8de5-5863827f10f3-routing-overlay-volume" pod "5805382b-6a6c-425e-8de5-5863827f10f3" (UID: "5805382b-6a6c-425e-8de5-5863827f10f3"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:21:44.746551 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:21:44.746526    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5805382b-6a6c-425e-8de5-5863827f10f3-plugins-config-volume" pod "5805382b-6a6c-425e-8de5-5863827f10f3" (UID: "5805382b-6a6c-425e-8de5-5863827f10f3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:21:44.746825 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:21:44.746535    4964 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/5805382b-6a6c-425e-8de5-5863827f10f3-routing-overlay-volume\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:21:44.748345 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:21:44.748305    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5805382b-6a6c-425e-8de5-5863827f10f3-provider-credentials" pod "5805382b-6a6c-425e-8de5-5863827f10f3" (UID: "5805382b-6a6c-425e-8de5-5863827f10f3"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:21:44.748453 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:21:44.748433    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5805382b-6a6c-425e-8de5-5863827f10f3-kube-api-access-l8ktf" pod "5805382b-6a6c-425e-8de5-5863827f10f3" (UID: "5805382b-6a6c-425e-8de5-5863827f10f3"). InnerVolumeSpecName "kube-api-access-l8ktf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:21:44.847890 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:21:44.847858    4964 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l8ktf\" (UniqueName: \"kubernetes.io/projected/5805382b-6a6c-425e-8de5-5863827f10f3-kube-api-access-l8ktf\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:21:44.847890 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:21:44.847887    4964 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/5805382b-6a6c-425e-8de5-5863827f10f3-provider-credentials\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:21:44.848105 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:21:44.847902    4964 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5805382b-6a6c-425e-8de5-5863827f10f3-plugins-config-volume\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:21:45.089138 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:21:45.089104    4964 generic.go:356] "Generic (PLEG): container finished" podID="5805382b-6a6c-425e-8de5-5863827f10f3" containerID="5b6a1435b07c82f8a8115b4551ebc1f38dd34a3d45f552a542029cd1c93b7dbf" exitCode=0
Oct 31 01:21:45.089309 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:21:45.089152    4964 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-55449dd944-gv2xm"
Oct 31 01:21:45.089309 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:21:45.089186    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-55449dd944-gv2xm" event={"ID":"5805382b-6a6c-425e-8de5-5863827f10f3","Type":"ContainerDied","Data":"5b6a1435b07c82f8a8115b4551ebc1f38dd34a3d45f552a542029cd1c93b7dbf"}
Oct 31 01:21:45.089309 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:21:45.089239    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-55449dd944-gv2xm" event={"ID":"5805382b-6a6c-425e-8de5-5863827f10f3","Type":"ContainerDied","Data":"90218266578c5b5302d11a50110e49705528d19b4443653b8f5f4a8339fc10dc"}
Oct 31 01:21:45.089309 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:21:45.089262    4964 scope.go:122] "RemoveContainer" containerID="5b6a1435b07c82f8a8115b4551ebc1f38dd34a3d45f552a542029cd1c93b7dbf"
Oct 31 01:21:45.099355 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:21:45.099310    4964 scope.go:122] "RemoveContainer" containerID="5b6a1435b07c82f8a8115b4551ebc1f38dd34a3d45f552a542029cd1c93b7dbf"
Oct 31 01:21:45.099818 ip-10-0-133-197 kubenswrapper[4964]: E1031 01:21:45.099723    4964 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5b6a1435b07c82f8a8115b4551ebc1f38dd34a3d45f552a542029cd1c93b7dbf\": container with ID starting with 5b6a1435b07c82f8a8115b4551ebc1f38dd34a3d45f552a542029cd1c93b7dbf not found: ID does not exist" containerID="5b6a1435b07c82f8a8115b4551ebc1f38dd34a3d45f552a542029cd1c93b7dbf"
Oct 31 01:21:45.099818 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:21:45.099758    4964 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5b6a1435b07c82f8a8115b4551ebc1f38dd34a3d45f552a542029cd1c93b7dbf"} err="failed to get container status \"5b6a1435b07c82f8a8115b4551ebc1f38dd34a3d45f552a542029cd1c93b7dbf\": rpc error: code = NotFound desc = could not find container \"5b6a1435b07c82f8a8115b4551ebc1f38dd34a3d45f552a542029cd1c93b7dbf\": container with ID starting with 5b6a1435b07c82f8a8115b4551ebc1f38dd34a3d45f552a542029cd1c93b7dbf not found: ID does not exist"
Oct 31 01:21:45.141206 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:21:45.141183    4964 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-a8e157/distinct-w3-a8e157-kserve-549559dd5c-t29tw"
Oct 31 01:21:45.154204 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:21:45.154181    4964 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-a8e157/distinct-w3-a8e157-kserve-549559dd5c-t29tw"
Oct 31 01:21:45.221181 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:21:45.221145    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:21:45.255950 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:21:45.255925    4964 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-55449dd944-gv2xm"]
Oct 31 01:21:45.306417 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:21:45.306384    4964 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-55449dd944-gv2xm"]
Oct 31 01:21:45.529889 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:21:45.529845    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:21:45.776777 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:21:45.776747    4964 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5805382b-6a6c-425e-8de5-5863827f10f3" path="/var/lib/kubelet/pods/5805382b-6a6c-425e-8de5-5863827f10f3/volumes"
Oct 31 01:21:46.108438 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:21:46.108411    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-a8e157/distinct-w3-a8e157-kserve-549559dd5c-t29tw"
Oct 31 01:21:50.097176 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:21:50.097143    4964 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-8f6b5b/e2e-conflict-a-8bd02cd2-kserve-7c55c85bd6-rwlfr"
Oct 31 01:21:50.112643 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:21:50.112613    4964 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-8f6b5b/e2e-conflict-a-8bd02cd2-kserve-7c55c85bd6-rwlfr"
Oct 31 01:21:50.185061 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:21:50.185029    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-8f6b5b/e2e-conflict-a-8bd02cd2-kserve-7c55c85bd6-rwlfr"
Oct 31 01:21:50.408107 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:21:50.408026    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:21:50.587507 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:21:50.587470    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:21:55.391733 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:21:55.391689    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:22:00.327486 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:22:00.327448    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:22:02.295863 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:22:02.295833    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:22:02.483526 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:22:02.483492    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:22:05.121947 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:22:05.121904    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:22:06.177465 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:22:06.177422    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:22:06.858740 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:22:06.858704    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:22:09.164806 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:22:09.164762    4964 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-dtxgt"]
Oct 31 01:22:09.165400 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:22:09.165383    4964 cpu_manager.go:414] "containerMap: removing container" podUID="5805382b-6a6c-425e-8de5-5863827f10f3" containerName="payload-processing"
Oct 31 01:22:09.165400 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:22:09.165400    4964 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5805382b-6a6c-425e-8de5-5863827f10f3" containerName="payload-processing"
Oct 31 01:22:09.165510 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:22:09.165496    4964 memory_manager.go:365] "RemoveStaleState removing state" podUID="5805382b-6a6c-425e-8de5-5863827f10f3" containerName="payload-processing"
Oct 31 01:22:09.171951 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:22:09.170221    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-dtxgt"
Oct 31 01:22:09.195759 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:22:09.195730    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9feacee-50cf-472f-94c7-9d15afb5aefc-credential-socket\") pod \"e2e-worker-w3-a8e157-openshift-default-7fb96689c7-dtxgt\" (UID: \"b9feacee-50cf-472f-94c7-9d15afb5aefc\") " pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-dtxgt"
Oct 31 01:22:09.195890 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:22:09.195781    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b9feacee-50cf-472f-94c7-9d15afb5aefc-istio-data\") pod \"e2e-worker-w3-a8e157-openshift-default-7fb96689c7-dtxgt\" (UID: \"b9feacee-50cf-472f-94c7-9d15afb5aefc\") " pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-dtxgt"
Oct 31 01:22:09.195890 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:22:09.195803    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b9feacee-50cf-472f-94c7-9d15afb5aefc-istio-podinfo\") pod \"e2e-worker-w3-a8e157-openshift-default-7fb96689c7-dtxgt\" (UID: \"b9feacee-50cf-472f-94c7-9d15afb5aefc\") " pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-dtxgt"
Oct 31 01:22:09.196015 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:22:09.195947    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vfpzx\" (UniqueName: \"kubernetes.io/projected/b9feacee-50cf-472f-94c7-9d15afb5aefc-kube-api-access-vfpzx\") pod \"e2e-worker-w3-a8e157-openshift-default-7fb96689c7-dtxgt\" (UID: \"b9feacee-50cf-472f-94c7-9d15afb5aefc\") " pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-dtxgt"
Oct 31 01:22:09.196015 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:22:09.195998    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b9feacee-50cf-472f-94c7-9d15afb5aefc-workload-certs\") pod \"e2e-worker-w3-a8e157-openshift-default-7fb96689c7-dtxgt\" (UID: \"b9feacee-50cf-472f-94c7-9d15afb5aefc\") " pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-dtxgt"
Oct 31 01:22:09.196122 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:22:09.196074    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9feacee-50cf-472f-94c7-9d15afb5aefc-workload-socket\") pod \"e2e-worker-w3-a8e157-openshift-default-7fb96689c7-dtxgt\" (UID: \"b9feacee-50cf-472f-94c7-9d15afb5aefc\") " pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-dtxgt"
Oct 31 01:22:09.196122 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:22:09.196100    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b9feacee-50cf-472f-94c7-9d15afb5aefc-istio-token\") pod \"e2e-worker-w3-a8e157-openshift-default-7fb96689c7-dtxgt\" (UID: \"b9feacee-50cf-472f-94c7-9d15afb5aefc\") " pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-dtxgt"
Oct 31 01:22:09.196206 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:22:09.196194    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b9feacee-50cf-472f-94c7-9d15afb5aefc-istiod-ca-cert\") pod \"e2e-worker-w3-a8e157-openshift-default-7fb96689c7-dtxgt\" (UID: \"b9feacee-50cf-472f-94c7-9d15afb5aefc\") " pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-dtxgt"
Oct 31 01:22:09.196240 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:22:09.196217    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b9feacee-50cf-472f-94c7-9d15afb5aefc-istio-envoy\") pod \"e2e-worker-w3-a8e157-openshift-default-7fb96689c7-dtxgt\" (UID: \"b9feacee-50cf-472f-94c7-9d15afb5aefc\") " pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-dtxgt"
Oct 31 01:22:09.275573 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:22:09.275542    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-dtxgt"]
Oct 31 01:22:09.297376 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:22:09.297347    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9feacee-50cf-472f-94c7-9d15afb5aefc-workload-socket\") pod \"e2e-worker-w3-a8e157-openshift-default-7fb96689c7-dtxgt\" (UID: \"b9feacee-50cf-472f-94c7-9d15afb5aefc\") " pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-dtxgt"
Oct 31 01:22:09.297376 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:22:09.297381    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b9feacee-50cf-472f-94c7-9d15afb5aefc-istio-token\") pod \"e2e-worker-w3-a8e157-openshift-default-7fb96689c7-dtxgt\" (UID: \"b9feacee-50cf-472f-94c7-9d15afb5aefc\") " pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-dtxgt"
Oct 31 01:22:09.297644 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:22:09.297421    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b9feacee-50cf-472f-94c7-9d15afb5aefc-istiod-ca-cert\") pod \"e2e-worker-w3-a8e157-openshift-default-7fb96689c7-dtxgt\" (UID: \"b9feacee-50cf-472f-94c7-9d15afb5aefc\") " pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-dtxgt"
Oct 31 01:22:09.297644 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:22:09.297445    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b9feacee-50cf-472f-94c7-9d15afb5aefc-istio-envoy\") pod \"e2e-worker-w3-a8e157-openshift-default-7fb96689c7-dtxgt\" (UID: \"b9feacee-50cf-472f-94c7-9d15afb5aefc\") " pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-dtxgt"
Oct 31 01:22:09.297644 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:22:09.297521    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9feacee-50cf-472f-94c7-9d15afb5aefc-credential-socket\") pod \"e2e-worker-w3-a8e157-openshift-default-7fb96689c7-dtxgt\" (UID: \"b9feacee-50cf-472f-94c7-9d15afb5aefc\") " pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-dtxgt"
Oct 31 01:22:09.297644 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:22:09.297563    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b9feacee-50cf-472f-94c7-9d15afb5aefc-istio-data\") pod \"e2e-worker-w3-a8e157-openshift-default-7fb96689c7-dtxgt\" (UID: \"b9feacee-50cf-472f-94c7-9d15afb5aefc\") " pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-dtxgt"
Oct 31 01:22:09.297644 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:22:09.297594    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b9feacee-50cf-472f-94c7-9d15afb5aefc-istio-podinfo\") pod \"e2e-worker-w3-a8e157-openshift-default-7fb96689c7-dtxgt\" (UID: \"b9feacee-50cf-472f-94c7-9d15afb5aefc\") " pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-dtxgt"
Oct 31 01:22:09.297644 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:22:09.297639    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vfpzx\" (UniqueName: \"kubernetes.io/projected/b9feacee-50cf-472f-94c7-9d15afb5aefc-kube-api-access-vfpzx\") pod \"e2e-worker-w3-a8e157-openshift-default-7fb96689c7-dtxgt\" (UID: \"b9feacee-50cf-472f-94c7-9d15afb5aefc\") " pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-dtxgt"
Oct 31 01:22:09.297972 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:22:09.297680    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b9feacee-50cf-472f-94c7-9d15afb5aefc-workload-certs\") pod \"e2e-worker-w3-a8e157-openshift-default-7fb96689c7-dtxgt\" (UID: \"b9feacee-50cf-472f-94c7-9d15afb5aefc\") " pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-dtxgt"
Oct 31 01:22:09.297972 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:22:09.297832    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9feacee-50cf-472f-94c7-9d15afb5aefc-workload-socket\") pod \"e2e-worker-w3-a8e157-openshift-default-7fb96689c7-dtxgt\" (UID: \"b9feacee-50cf-472f-94c7-9d15afb5aefc\") " pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-dtxgt"
Oct 31 01:22:09.298158 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:22:09.298131    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9feacee-50cf-472f-94c7-9d15afb5aefc-credential-socket\") pod \"e2e-worker-w3-a8e157-openshift-default-7fb96689c7-dtxgt\" (UID: \"b9feacee-50cf-472f-94c7-9d15afb5aefc\") " pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-dtxgt"
Oct 31 01:22:09.298488 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:22:09.298399    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b9feacee-50cf-472f-94c7-9d15afb5aefc-istio-data\") pod \"e2e-worker-w3-a8e157-openshift-default-7fb96689c7-dtxgt\" (UID: \"b9feacee-50cf-472f-94c7-9d15afb5aefc\") " pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-dtxgt"
Oct 31 01:22:09.298566 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:22:09.298512    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b9feacee-50cf-472f-94c7-9d15afb5aefc-workload-certs\") pod \"e2e-worker-w3-a8e157-openshift-default-7fb96689c7-dtxgt\" (UID: \"b9feacee-50cf-472f-94c7-9d15afb5aefc\") " pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-dtxgt"
Oct 31 01:22:09.298793 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:22:09.298774    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b9feacee-50cf-472f-94c7-9d15afb5aefc-istiod-ca-cert\") pod \"e2e-worker-w3-a8e157-openshift-default-7fb96689c7-dtxgt\" (UID: \"b9feacee-50cf-472f-94c7-9d15afb5aefc\") " pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-dtxgt"
Oct 31 01:22:09.300093 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:22:09.300065    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b9feacee-50cf-472f-94c7-9d15afb5aefc-istio-envoy\") pod \"e2e-worker-w3-a8e157-openshift-default-7fb96689c7-dtxgt\" (UID: \"b9feacee-50cf-472f-94c7-9d15afb5aefc\") " pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-dtxgt"
Oct 31 01:22:09.300891 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:22:09.300866    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b9feacee-50cf-472f-94c7-9d15afb5aefc-istio-podinfo\") pod \"e2e-worker-w3-a8e157-openshift-default-7fb96689c7-dtxgt\" (UID: \"b9feacee-50cf-472f-94c7-9d15afb5aefc\") " pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-dtxgt"
Oct 31 01:22:09.355524 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:22:09.355493    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b9feacee-50cf-472f-94c7-9d15afb5aefc-istio-token\") pod \"e2e-worker-w3-a8e157-openshift-default-7fb96689c7-dtxgt\" (UID: \"b9feacee-50cf-472f-94c7-9d15afb5aefc\") " pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-dtxgt"
Oct 31 01:22:09.355735 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:22:09.355710    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vfpzx\" (UniqueName: \"kubernetes.io/projected/b9feacee-50cf-472f-94c7-9d15afb5aefc-kube-api-access-vfpzx\") pod \"e2e-worker-w3-a8e157-openshift-default-7fb96689c7-dtxgt\" (UID: \"b9feacee-50cf-472f-94c7-9d15afb5aefc\") " pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-dtxgt"
Oct 31 01:22:09.487947 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:22:09.487917    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-dtxgt"
Oct 31 01:22:09.902683 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:22:09.902598    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:22:09.920579 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:22:09.920550    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-dtxgt"]
Oct 31 01:22:09.923091 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:22:09.923061    4964 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb9feacee_50cf_472f_94c7_9d15afb5aefc.slice/crio-8b988eea33693eb4b3dfbdacf2ef776c0ff40b055525ad3bdc0648b141d8c098 WatchSource:0}: Error finding container 8b988eea33693eb4b3dfbdacf2ef776c0ff40b055525ad3bdc0648b141d8c098: Status 404 returned error can't find the container with id 8b988eea33693eb4b3dfbdacf2ef776c0ff40b055525ad3bdc0648b141d8c098
Oct 31 01:22:09.926562 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:22:09.926527    4964 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 01:22:09.926692 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:22:09.926619    4964 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 01:22:09.926692 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:22:09.926686    4964 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 01:22:10.304951 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:22:10.304915    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-dtxgt" event={"ID":"b9feacee-50cf-472f-94c7-9d15afb5aefc","Type":"ContainerStarted","Data":"09baf1c6431b481e836ba2ce93d08e410a6af1927ad72a9341e7d4460ce74c1e"}
Oct 31 01:22:10.305406 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:22:10.304985    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-dtxgt" event={"ID":"b9feacee-50cf-472f-94c7-9d15afb5aefc","Type":"ContainerStarted","Data":"8b988eea33693eb4b3dfbdacf2ef776c0ff40b055525ad3bdc0648b141d8c098"}
Oct 31 01:22:10.435464 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:22:10.433191    4964 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-dtxgt" podStartSLOduration=1.433169371 podStartE2EDuration="1.433169371s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:22:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:22:10.428603199 +0000 UTC m=+1098.905169579" watchObservedRunningTime="2026-10-31 01:22:10.433169371 +0000 UTC m=+1098.909735753"
Oct 31 01:22:10.488638 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:22:10.488602    4964 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-dtxgt"
Oct 31 01:22:10.489970 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:22:10.489945    4964 patch_prober.go:29] interesting pod/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-dtxgt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body=
Oct 31 01:22:10.490101 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:22:10.489991    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-dtxgt" podUID="b9feacee-50cf-472f-94c7-9d15afb5aefc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused"
Oct 31 01:22:11.246598 ip-10-0-133-197 kubenswrapper[4964]: E1031 01:22:11.246565    4964 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q56x5" containerName="extractor"
Oct 31 01:22:11.488799 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:22:11.488766    4964 patch_prober.go:29] interesting pod/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-dtxgt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body=
Oct 31 01:22:11.489297 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:22:11.489272    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-dtxgt" podUID="b9feacee-50cf-472f-94c7-9d15afb5aefc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused"
Oct 31 01:22:12.488586 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:22:12.488550    4964 patch_prober.go:29] interesting pod/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-dtxgt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body=
Oct 31 01:22:12.488739 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:22:12.488603    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-dtxgt" podUID="b9feacee-50cf-472f-94c7-9d15afb5aefc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused"
Oct 31 01:22:13.489083 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:22:13.489052    4964 patch_prober.go:29] interesting pod/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-dtxgt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body=
Oct 31 01:22:13.489485 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:22:13.489131    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-dtxgt" podUID="b9feacee-50cf-472f-94c7-9d15afb5aefc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused"
Oct 31 01:22:14.488931 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:22:14.488890    4964 patch_prober.go:29] interesting pod/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-dtxgt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body=
Oct 31 01:22:14.489098 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:22:14.488951    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-dtxgt" podUID="b9feacee-50cf-472f-94c7-9d15afb5aefc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused"
Oct 31 01:22:14.970862 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:22:14.970823    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:22:15.488220 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:22:15.488194    4964 patch_prober.go:29] interesting pod/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-dtxgt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body=
Oct 31 01:22:15.488411 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:22:15.488255    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-dtxgt" podUID="b9feacee-50cf-472f-94c7-9d15afb5aefc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused"
Oct 31 01:22:16.489186 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:22:16.489147    4964 patch_prober.go:29] interesting pod/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-dtxgt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body=
Oct 31 01:22:16.489617 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:22:16.489231    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-dtxgt" podUID="b9feacee-50cf-472f-94c7-9d15afb5aefc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused"
Oct 31 01:22:17.493854 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:22:17.493828    4964 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-dtxgt"
Oct 31 01:22:17.494350 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:22:17.494270    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-dtxgt"
Oct 31 01:22:17.495079 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:22:17.495062    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-dtxgt"
Oct 31 01:22:19.908410 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:22:19.908372    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:22:24.703350 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:22:24.703245    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:22:29.601256 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:22:29.601216    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:22:34.411138 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:22:34.411104    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:22:37.489659 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:22:37.489628    4964 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-a8e157/distinct-2-w3-a8e157-kserve-9ddd45d6-ng7j7"]
Oct 31 01:22:37.493772 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:22:37.493752    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-a8e157/distinct-2-w3-a8e157-kserve-9ddd45d6-ng7j7"
Oct 31 01:22:37.535312 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:22:37.535287    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bc7a71df-c523-4590-91df-cf8fdf3ed19e-tmp-dir\") pod \"distinct-2-w3-a8e157-kserve-9ddd45d6-ng7j7\" (UID: \"bc7a71df-c523-4590-91df-cf8fdf3ed19e\") " pod="e2e-models-e2e-worker-w3-a8e157/distinct-2-w3-a8e157-kserve-9ddd45d6-ng7j7"
Oct 31 01:22:37.535455 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:22:37.535355    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bc7a71df-c523-4590-91df-cf8fdf3ed19e-home\") pod \"distinct-2-w3-a8e157-kserve-9ddd45d6-ng7j7\" (UID: \"bc7a71df-c523-4590-91df-cf8fdf3ed19e\") " pod="e2e-models-e2e-worker-w3-a8e157/distinct-2-w3-a8e157-kserve-9ddd45d6-ng7j7"
Oct 31 01:22:37.535455 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:22:37.535431    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-98r7b\" (UniqueName: \"kubernetes.io/projected/bc7a71df-c523-4590-91df-cf8fdf3ed19e-kube-api-access-98r7b\") pod \"distinct-2-w3-a8e157-kserve-9ddd45d6-ng7j7\" (UID: \"bc7a71df-c523-4590-91df-cf8fdf3ed19e\") " pod="e2e-models-e2e-worker-w3-a8e157/distinct-2-w3-a8e157-kserve-9ddd45d6-ng7j7"
Oct 31 01:22:37.535591 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:22:37.535572    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bc7a71df-c523-4590-91df-cf8fdf3ed19e-model-cache\") pod \"distinct-2-w3-a8e157-kserve-9ddd45d6-ng7j7\" (UID: \"bc7a71df-c523-4590-91df-cf8fdf3ed19e\") " pod="e2e-models-e2e-worker-w3-a8e157/distinct-2-w3-a8e157-kserve-9ddd45d6-ng7j7"
Oct 31 01:22:37.535627 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:22:37.535605    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bc7a71df-c523-4590-91df-cf8fdf3ed19e-tls-certs\") pod \"distinct-2-w3-a8e157-kserve-9ddd45d6-ng7j7\" (UID: \"bc7a71df-c523-4590-91df-cf8fdf3ed19e\") " pod="e2e-models-e2e-worker-w3-a8e157/distinct-2-w3-a8e157-kserve-9ddd45d6-ng7j7"
Oct 31 01:22:37.535627 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:22:37.535622    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bc7a71df-c523-4590-91df-cf8fdf3ed19e-dshm\") pod \"distinct-2-w3-a8e157-kserve-9ddd45d6-ng7j7\" (UID: \"bc7a71df-c523-4590-91df-cf8fdf3ed19e\") " pod="e2e-models-e2e-worker-w3-a8e157/distinct-2-w3-a8e157-kserve-9ddd45d6-ng7j7"
Oct 31 01:22:37.597049 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:22:37.597018    4964 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-a8e157\"/\"distinct-2-w3-a8e157-kserve-self-signed-certs\""
Oct 31 01:22:37.599915 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:22:37.599893    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-a8e157/distinct-2-w3-a8e157-kserve-9ddd45d6-ng7j7"]
Oct 31 01:22:37.636461 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:22:37.636433    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bc7a71df-c523-4590-91df-cf8fdf3ed19e-tls-certs\") pod \"distinct-2-w3-a8e157-kserve-9ddd45d6-ng7j7\" (UID: \"bc7a71df-c523-4590-91df-cf8fdf3ed19e\") " pod="e2e-models-e2e-worker-w3-a8e157/distinct-2-w3-a8e157-kserve-9ddd45d6-ng7j7"
Oct 31 01:22:37.636461 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:22:37.636463    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bc7a71df-c523-4590-91df-cf8fdf3ed19e-dshm\") pod \"distinct-2-w3-a8e157-kserve-9ddd45d6-ng7j7\" (UID: \"bc7a71df-c523-4590-91df-cf8fdf3ed19e\") " pod="e2e-models-e2e-worker-w3-a8e157/distinct-2-w3-a8e157-kserve-9ddd45d6-ng7j7"
Oct 31 01:22:37.636677 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:22:37.636556    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bc7a71df-c523-4590-91df-cf8fdf3ed19e-tmp-dir\") pod \"distinct-2-w3-a8e157-kserve-9ddd45d6-ng7j7\" (UID: \"bc7a71df-c523-4590-91df-cf8fdf3ed19e\") " pod="e2e-models-e2e-worker-w3-a8e157/distinct-2-w3-a8e157-kserve-9ddd45d6-ng7j7"
Oct 31 01:22:37.636677 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:22:37.636594    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bc7a71df-c523-4590-91df-cf8fdf3ed19e-home\") pod \"distinct-2-w3-a8e157-kserve-9ddd45d6-ng7j7\" (UID: \"bc7a71df-c523-4590-91df-cf8fdf3ed19e\") " pod="e2e-models-e2e-worker-w3-a8e157/distinct-2-w3-a8e157-kserve-9ddd45d6-ng7j7"
Oct 31 01:22:37.636677 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:22:37.636626    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-98r7b\" (UniqueName: \"kubernetes.io/projected/bc7a71df-c523-4590-91df-cf8fdf3ed19e-kube-api-access-98r7b\") pod \"distinct-2-w3-a8e157-kserve-9ddd45d6-ng7j7\" (UID: \"bc7a71df-c523-4590-91df-cf8fdf3ed19e\") " pod="e2e-models-e2e-worker-w3-a8e157/distinct-2-w3-a8e157-kserve-9ddd45d6-ng7j7"
Oct 31 01:22:37.636820 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:22:37.636717    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bc7a71df-c523-4590-91df-cf8fdf3ed19e-model-cache\") pod \"distinct-2-w3-a8e157-kserve-9ddd45d6-ng7j7\" (UID: \"bc7a71df-c523-4590-91df-cf8fdf3ed19e\") " pod="e2e-models-e2e-worker-w3-a8e157/distinct-2-w3-a8e157-kserve-9ddd45d6-ng7j7"
Oct 31 01:22:37.637088 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:22:37.637065    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bc7a71df-c523-4590-91df-cf8fdf3ed19e-tmp-dir\") pod \"distinct-2-w3-a8e157-kserve-9ddd45d6-ng7j7\" (UID: \"bc7a71df-c523-4590-91df-cf8fdf3ed19e\") " pod="e2e-models-e2e-worker-w3-a8e157/distinct-2-w3-a8e157-kserve-9ddd45d6-ng7j7"
Oct 31 01:22:37.637202 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:22:37.637109    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bc7a71df-c523-4590-91df-cf8fdf3ed19e-home\") pod \"distinct-2-w3-a8e157-kserve-9ddd45d6-ng7j7\" (UID: \"bc7a71df-c523-4590-91df-cf8fdf3ed19e\") " pod="e2e-models-e2e-worker-w3-a8e157/distinct-2-w3-a8e157-kserve-9ddd45d6-ng7j7"
Oct 31 01:22:37.637202 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:22:37.637121    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bc7a71df-c523-4590-91df-cf8fdf3ed19e-model-cache\") pod \"distinct-2-w3-a8e157-kserve-9ddd45d6-ng7j7\" (UID: \"bc7a71df-c523-4590-91df-cf8fdf3ed19e\") " pod="e2e-models-e2e-worker-w3-a8e157/distinct-2-w3-a8e157-kserve-9ddd45d6-ng7j7"
Oct 31 01:22:37.638941 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:22:37.638915    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bc7a71df-c523-4590-91df-cf8fdf3ed19e-dshm\") pod \"distinct-2-w3-a8e157-kserve-9ddd45d6-ng7j7\" (UID: \"bc7a71df-c523-4590-91df-cf8fdf3ed19e\") " pod="e2e-models-e2e-worker-w3-a8e157/distinct-2-w3-a8e157-kserve-9ddd45d6-ng7j7"
Oct 31 01:22:37.639276 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:22:37.639260    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bc7a71df-c523-4590-91df-cf8fdf3ed19e-tls-certs\") pod \"distinct-2-w3-a8e157-kserve-9ddd45d6-ng7j7\" (UID: \"bc7a71df-c523-4590-91df-cf8fdf3ed19e\") " pod="e2e-models-e2e-worker-w3-a8e157/distinct-2-w3-a8e157-kserve-9ddd45d6-ng7j7"
Oct 31 01:22:37.649708 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:22:37.649656    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:22:37.695333 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:22:37.695294    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-98r7b\" (UniqueName: \"kubernetes.io/projected/bc7a71df-c523-4590-91df-cf8fdf3ed19e-kube-api-access-98r7b\") pod \"distinct-2-w3-a8e157-kserve-9ddd45d6-ng7j7\" (UID: \"bc7a71df-c523-4590-91df-cf8fdf3ed19e\") " pod="e2e-models-e2e-worker-w3-a8e157/distinct-2-w3-a8e157-kserve-9ddd45d6-ng7j7"
Oct 31 01:22:37.806639 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:22:37.806550    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-a8e157/distinct-2-w3-a8e157-kserve-9ddd45d6-ng7j7"
Oct 31 01:22:38.186950 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:22:38.186924    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-a8e157/distinct-2-w3-a8e157-kserve-9ddd45d6-ng7j7"]
Oct 31 01:22:38.188862 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:22:38.188841    4964 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbc7a71df_c523_4590_91df_cf8fdf3ed19e.slice/crio-70d64fa2281f36b15e8541dd1793528c025e09a6fec3858757b6cad3fc84621a WatchSource:0}: Error finding container 70d64fa2281f36b15e8541dd1793528c025e09a6fec3858757b6cad3fc84621a: Status 404 returned error can't find the container with id 70d64fa2281f36b15e8541dd1793528c025e09a6fec3858757b6cad3fc84621a
Oct 31 01:22:38.505692 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:22:38.505653    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-a8e157/distinct-2-w3-a8e157-kserve-9ddd45d6-ng7j7" event={"ID":"bc7a71df-c523-4590-91df-cf8fdf3ed19e","Type":"ContainerStarted","Data":"cfcbc7b773c61d1d43ce734aabc9a090f19a1794c12f9dac57a6821f7fef7361"}
Oct 31 01:22:38.506206 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:22:38.505706    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-a8e157/distinct-2-w3-a8e157-kserve-9ddd45d6-ng7j7" event={"ID":"bc7a71df-c523-4590-91df-cf8fdf3ed19e","Type":"ContainerStarted","Data":"70d64fa2281f36b15e8541dd1793528c025e09a6fec3858757b6cad3fc84621a"}
Oct 31 01:22:38.622759 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:22:38.622695    4964 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-a8e157/distinct-2-w3-a8e157-kserve-9ddd45d6-ng7j7" podStartSLOduration=1.622679797 podStartE2EDuration="1.622679797s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:22:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:22:38.619664352 +0000 UTC m=+1127.096230734" watchObservedRunningTime="2026-10-31 01:22:38.622679797 +0000 UTC m=+1127.099246244"
Oct 31 01:22:38.807487 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:22:38.807405    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-a8e157/distinct-2-w3-a8e157-kserve-9ddd45d6-ng7j7"
Oct 31 01:22:39.333143 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:22:39.333083    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:22:41.237854 ip-10-0-133-197 kubenswrapper[4964]: E1031 01:22:41.237820    4964 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q56x5" containerName="extractor"
Oct 31 01:22:41.494202 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:22:41.494116    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:22:41.725518 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:22:41.725479    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:22:41.916001 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:22:41.915902    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:22:44.110814 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:22:44.110775    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:22:45.504154 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:22:45.504101    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:22:45.871883 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:22:45.871801    4964 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-75984d866d-k5g7h"]
Oct 31 01:22:45.876098 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:22:45.876076    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-75984d866d-k5g7h"
Oct 31 01:22:45.928578 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:22:45.928541    4964 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-plugins\""
Oct 31 01:22:45.928578 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:22:45.928541    4964 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"openshift-service-ca.crt\""
Oct 31 01:22:45.928578 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:22:45.928582    4964 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-4d206374657b\""
Oct 31 01:22:45.932364 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:22:45.932334    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-czj6h\" (UniqueName: \"kubernetes.io/projected/97691d17-0ce5-4ca4-885a-c092a0317ebb-kube-api-access-czj6h\") pod \"payload-processing-external-model-75984d866d-k5g7h\" (UID: \"97691d17-0ce5-4ca4-885a-c092a0317ebb\") " pod="models-as-a-service/payload-processing-external-model-75984d866d-k5g7h"
Oct 31 01:22:45.932536 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:22:45.932514    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/97691d17-0ce5-4ca4-885a-c092a0317ebb-provider-credentials\") pod \"payload-processing-external-model-75984d866d-k5g7h\" (UID: \"97691d17-0ce5-4ca4-885a-c092a0317ebb\") " pod="models-as-a-service/payload-processing-external-model-75984d866d-k5g7h"
Oct 31 01:22:45.932599 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:22:45.932583    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/97691d17-0ce5-4ca4-885a-c092a0317ebb-plugins-config-volume\") pod \"payload-processing-external-model-75984d866d-k5g7h\" (UID: \"97691d17-0ce5-4ca4-885a-c092a0317ebb\") " pod="models-as-a-service/payload-processing-external-model-75984d866d-k5g7h"
Oct 31 01:22:45.932659 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:22:45.932642    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/97691d17-0ce5-4ca4-885a-c092a0317ebb-routing-overlay-volume\") pod \"payload-processing-external-model-75984d866d-k5g7h\" (UID: \"97691d17-0ce5-4ca4-885a-c092a0317ebb\") " pod="models-as-a-service/payload-processing-external-model-75984d866d-k5g7h"
Oct 31 01:22:45.977892 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:22:45.977863    4964 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-1984e9af4453\""
Oct 31 01:22:45.978191 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:22:45.978172    4964 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-1dab50b5e299\""
Oct 31 01:22:45.978287 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:22:45.978174    4964 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"routing-overlay\""
Oct 31 01:22:45.978287 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:22:45.978183    4964 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"kube-root-ca.crt\""
Oct 31 01:22:45.978661 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:22:45.978639    4964 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-dockercfg-7qstt\""
Oct 31 01:22:45.983538 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:22:45.983511    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-75984d866d-k5g7h"]
Oct 31 01:22:46.034157 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:22:46.034124    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/97691d17-0ce5-4ca4-885a-c092a0317ebb-provider-credentials\") pod \"payload-processing-external-model-75984d866d-k5g7h\" (UID: \"97691d17-0ce5-4ca4-885a-c092a0317ebb\") " pod="models-as-a-service/payload-processing-external-model-75984d866d-k5g7h"
Oct 31 01:22:46.034390 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:22:46.034200    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/97691d17-0ce5-4ca4-885a-c092a0317ebb-plugins-config-volume\") pod \"payload-processing-external-model-75984d866d-k5g7h\" (UID: \"97691d17-0ce5-4ca4-885a-c092a0317ebb\") " pod="models-as-a-service/payload-processing-external-model-75984d866d-k5g7h"
Oct 31 01:22:46.034390 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:22:46.034225    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/97691d17-0ce5-4ca4-885a-c092a0317ebb-routing-overlay-volume\") pod \"payload-processing-external-model-75984d866d-k5g7h\" (UID: \"97691d17-0ce5-4ca4-885a-c092a0317ebb\") " pod="models-as-a-service/payload-processing-external-model-75984d866d-k5g7h"
Oct 31 01:22:46.034390 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:22:46.034292    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-czj6h\" (UniqueName: \"kubernetes.io/projected/97691d17-0ce5-4ca4-885a-c092a0317ebb-kube-api-access-czj6h\") pod \"payload-processing-external-model-75984d866d-k5g7h\" (UID: \"97691d17-0ce5-4ca4-885a-c092a0317ebb\") " pod="models-as-a-service/payload-processing-external-model-75984d866d-k5g7h"
Oct 31 01:22:46.035142 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:22:46.035108    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/97691d17-0ce5-4ca4-885a-c092a0317ebb-routing-overlay-volume\") pod \"payload-processing-external-model-75984d866d-k5g7h\" (UID: \"97691d17-0ce5-4ca4-885a-c092a0317ebb\") " pod="models-as-a-service/payload-processing-external-model-75984d866d-k5g7h"
Oct 31 01:22:46.035142 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:22:46.035130    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/97691d17-0ce5-4ca4-885a-c092a0317ebb-plugins-config-volume\") pod \"payload-processing-external-model-75984d866d-k5g7h\" (UID: \"97691d17-0ce5-4ca4-885a-c092a0317ebb\") " pod="models-as-a-service/payload-processing-external-model-75984d866d-k5g7h"
Oct 31 01:22:46.037031 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:22:46.037011    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/97691d17-0ce5-4ca4-885a-c092a0317ebb-provider-credentials\") pod \"payload-processing-external-model-75984d866d-k5g7h\" (UID: \"97691d17-0ce5-4ca4-885a-c092a0317ebb\") " pod="models-as-a-service/payload-processing-external-model-75984d866d-k5g7h"
Oct 31 01:22:46.090969 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:22:46.090932    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-czj6h\" (UniqueName: \"kubernetes.io/projected/97691d17-0ce5-4ca4-885a-c092a0317ebb-kube-api-access-czj6h\") pod \"payload-processing-external-model-75984d866d-k5g7h\" (UID: \"97691d17-0ce5-4ca4-885a-c092a0317ebb\") " pod="models-as-a-service/payload-processing-external-model-75984d866d-k5g7h"
Oct 31 01:22:46.192352 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:22:46.192247    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-75984d866d-k5g7h"
Oct 31 01:22:46.573747 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:22:46.573719    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-75984d866d-k5g7h"]
Oct 31 01:22:46.575739 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:22:46.575710    4964 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97691d17_0ce5_4ca4_885a_c092a0317ebb.slice/crio-1c1ebc0ddb738367ffc6a81d0165034cd95d828a472ccf86772b06fb4b39b53b WatchSource:0}: Error finding container 1c1ebc0ddb738367ffc6a81d0165034cd95d828a472ccf86772b06fb4b39b53b: Status 404 returned error can't find the container with id 1c1ebc0ddb738367ffc6a81d0165034cd95d828a472ccf86772b06fb4b39b53b
Oct 31 01:22:46.888310 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:22:46.888230    4964 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-baacee/distinct-w1-baacee-kserve-6c85cfb7d7-stkqq"]
Oct 31 01:22:46.892237 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:22:46.892220    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-baacee/distinct-w1-baacee-kserve-6c85cfb7d7-stkqq"
Oct 31 01:22:46.945989 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:22:46.945650    4964 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-baacee\"/\"distinct-w1-baacee-kserve-self-signed-certs\""
Oct 31 01:22:46.946280 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:22:46.946049    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/df6d0bdd-382d-49b7-879c-15cb5371246e-tmp-dir\") pod \"distinct-w1-baacee-kserve-6c85cfb7d7-stkqq\" (UID: \"df6d0bdd-382d-49b7-879c-15cb5371246e\") " pod="e2e-models-e2e-worker-w1-baacee/distinct-w1-baacee-kserve-6c85cfb7d7-stkqq"
Oct 31 01:22:46.946280 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:22:46.946108    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/df6d0bdd-382d-49b7-879c-15cb5371246e-model-cache\") pod \"distinct-w1-baacee-kserve-6c85cfb7d7-stkqq\" (UID: \"df6d0bdd-382d-49b7-879c-15cb5371246e\") " pod="e2e-models-e2e-worker-w1-baacee/distinct-w1-baacee-kserve-6c85cfb7d7-stkqq"
Oct 31 01:22:46.946280 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:22:46.946138    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/df6d0bdd-382d-49b7-879c-15cb5371246e-tls-certs\") pod \"distinct-w1-baacee-kserve-6c85cfb7d7-stkqq\" (UID: \"df6d0bdd-382d-49b7-879c-15cb5371246e\") " pod="e2e-models-e2e-worker-w1-baacee/distinct-w1-baacee-kserve-6c85cfb7d7-stkqq"
Oct 31 01:22:46.946280 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:22:46.946168    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/df6d0bdd-382d-49b7-879c-15cb5371246e-dshm\") pod \"distinct-w1-baacee-kserve-6c85cfb7d7-stkqq\" (UID: \"df6d0bdd-382d-49b7-879c-15cb5371246e\") " pod="e2e-models-e2e-worker-w1-baacee/distinct-w1-baacee-kserve-6c85cfb7d7-stkqq"
Oct 31 01:22:46.946280 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:22:46.946261    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wjk2j\" (UniqueName: \"kubernetes.io/projected/df6d0bdd-382d-49b7-879c-15cb5371246e-kube-api-access-wjk2j\") pod \"distinct-w1-baacee-kserve-6c85cfb7d7-stkqq\" (UID: \"df6d0bdd-382d-49b7-879c-15cb5371246e\") " pod="e2e-models-e2e-worker-w1-baacee/distinct-w1-baacee-kserve-6c85cfb7d7-stkqq"
Oct 31 01:22:46.946568 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:22:46.946292    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/df6d0bdd-382d-49b7-879c-15cb5371246e-home\") pod \"distinct-w1-baacee-kserve-6c85cfb7d7-stkqq\" (UID: \"df6d0bdd-382d-49b7-879c-15cb5371246e\") " pod="e2e-models-e2e-worker-w1-baacee/distinct-w1-baacee-kserve-6c85cfb7d7-stkqq"
Oct 31 01:22:46.950744 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:22:46.950723    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-baacee/distinct-w1-baacee-kserve-6c85cfb7d7-stkqq"]
Oct 31 01:22:47.019659 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:22:47.019629    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:22:47.047746 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:22:47.047714    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/df6d0bdd-382d-49b7-879c-15cb5371246e-model-cache\") pod \"distinct-w1-baacee-kserve-6c85cfb7d7-stkqq\" (UID: \"df6d0bdd-382d-49b7-879c-15cb5371246e\") " pod="e2e-models-e2e-worker-w1-baacee/distinct-w1-baacee-kserve-6c85cfb7d7-stkqq"
Oct 31 01:22:47.047940 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:22:47.047758    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/df6d0bdd-382d-49b7-879c-15cb5371246e-tls-certs\") pod \"distinct-w1-baacee-kserve-6c85cfb7d7-stkqq\" (UID: \"df6d0bdd-382d-49b7-879c-15cb5371246e\") " pod="e2e-models-e2e-worker-w1-baacee/distinct-w1-baacee-kserve-6c85cfb7d7-stkqq"
Oct 31 01:22:47.047940 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:22:47.047830    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/df6d0bdd-382d-49b7-879c-15cb5371246e-dshm\") pod \"distinct-w1-baacee-kserve-6c85cfb7d7-stkqq\" (UID: \"df6d0bdd-382d-49b7-879c-15cb5371246e\") " pod="e2e-models-e2e-worker-w1-baacee/distinct-w1-baacee-kserve-6c85cfb7d7-stkqq"
Oct 31 01:22:47.047940 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:22:47.047923    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wjk2j\" (UniqueName: \"kubernetes.io/projected/df6d0bdd-382d-49b7-879c-15cb5371246e-kube-api-access-wjk2j\") pod \"distinct-w1-baacee-kserve-6c85cfb7d7-stkqq\" (UID: \"df6d0bdd-382d-49b7-879c-15cb5371246e\") " pod="e2e-models-e2e-worker-w1-baacee/distinct-w1-baacee-kserve-6c85cfb7d7-stkqq"
Oct 31 01:22:47.048112 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:22:47.047947    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/df6d0bdd-382d-49b7-879c-15cb5371246e-home\") pod \"distinct-w1-baacee-kserve-6c85cfb7d7-stkqq\" (UID: \"df6d0bdd-382d-49b7-879c-15cb5371246e\") " pod="e2e-models-e2e-worker-w1-baacee/distinct-w1-baacee-kserve-6c85cfb7d7-stkqq"
Oct 31 01:22:47.048164 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:22:47.048117    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/df6d0bdd-382d-49b7-879c-15cb5371246e-tmp-dir\") pod \"distinct-w1-baacee-kserve-6c85cfb7d7-stkqq\" (UID: \"df6d0bdd-382d-49b7-879c-15cb5371246e\") " pod="e2e-models-e2e-worker-w1-baacee/distinct-w1-baacee-kserve-6c85cfb7d7-stkqq"
Oct 31 01:22:47.048467 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:22:47.048445    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/df6d0bdd-382d-49b7-879c-15cb5371246e-tmp-dir\") pod \"distinct-w1-baacee-kserve-6c85cfb7d7-stkqq\" (UID: \"df6d0bdd-382d-49b7-879c-15cb5371246e\") " pod="e2e-models-e2e-worker-w1-baacee/distinct-w1-baacee-kserve-6c85cfb7d7-stkqq"
Oct 31 01:22:47.048706 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:22:47.048686    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/df6d0bdd-382d-49b7-879c-15cb5371246e-home\") pod \"distinct-w1-baacee-kserve-6c85cfb7d7-stkqq\" (UID: \"df6d0bdd-382d-49b7-879c-15cb5371246e\") " pod="e2e-models-e2e-worker-w1-baacee/distinct-w1-baacee-kserve-6c85cfb7d7-stkqq"
Oct 31 01:22:47.048848 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:22:47.048190    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/df6d0bdd-382d-49b7-879c-15cb5371246e-model-cache\") pod \"distinct-w1-baacee-kserve-6c85cfb7d7-stkqq\" (UID: \"df6d0bdd-382d-49b7-879c-15cb5371246e\") " pod="e2e-models-e2e-worker-w1-baacee/distinct-w1-baacee-kserve-6c85cfb7d7-stkqq"
Oct 31 01:22:47.051114 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:22:47.051089    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/df6d0bdd-382d-49b7-879c-15cb5371246e-dshm\") pod \"distinct-w1-baacee-kserve-6c85cfb7d7-stkqq\" (UID: \"df6d0bdd-382d-49b7-879c-15cb5371246e\") " pod="e2e-models-e2e-worker-w1-baacee/distinct-w1-baacee-kserve-6c85cfb7d7-stkqq"
Oct 31 01:22:47.051238 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:22:47.051214    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/df6d0bdd-382d-49b7-879c-15cb5371246e-tls-certs\") pod \"distinct-w1-baacee-kserve-6c85cfb7d7-stkqq\" (UID: \"df6d0bdd-382d-49b7-879c-15cb5371246e\") " pod="e2e-models-e2e-worker-w1-baacee/distinct-w1-baacee-kserve-6c85cfb7d7-stkqq"
Oct 31 01:22:47.120812 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:22:47.120776    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wjk2j\" (UniqueName: \"kubernetes.io/projected/df6d0bdd-382d-49b7-879c-15cb5371246e-kube-api-access-wjk2j\") pod \"distinct-w1-baacee-kserve-6c85cfb7d7-stkqq\" (UID: \"df6d0bdd-382d-49b7-879c-15cb5371246e\") " pod="e2e-models-e2e-worker-w1-baacee/distinct-w1-baacee-kserve-6c85cfb7d7-stkqq"
Oct 31 01:22:47.207524 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:22:47.207436    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-baacee/distinct-w1-baacee-kserve-6c85cfb7d7-stkqq"
Oct 31 01:22:47.567152 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:22:47.567117    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-75984d866d-k5g7h" event={"ID":"97691d17-0ce5-4ca4-885a-c092a0317ebb","Type":"ContainerStarted","Data":"fd694fb4c4ed8385be4f1b2c91ae33c244af4fe12eff06bc99bc64e0889361bc"}
Oct 31 01:22:47.567360 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:22:47.567171    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-75984d866d-k5g7h" event={"ID":"97691d17-0ce5-4ca4-885a-c092a0317ebb","Type":"ContainerStarted","Data":"1c1ebc0ddb738367ffc6a81d0165034cd95d828a472ccf86772b06fb4b39b53b"}
Oct 31 01:22:47.567360 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:22:47.567271    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-75984d866d-k5g7h"
Oct 31 01:22:47.588811 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:22:47.588786    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-baacee/distinct-w1-baacee-kserve-6c85cfb7d7-stkqq"]
Oct 31 01:22:47.590805 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:22:47.590777    4964 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poddf6d0bdd_382d_49b7_879c_15cb5371246e.slice/crio-d7452c10094d65664e60d3e34fdff39bbf5c56880954582560bd8ab989e6d418 WatchSource:0}: Error finding container d7452c10094d65664e60d3e34fdff39bbf5c56880954582560bd8ab989e6d418: Status 404 returned error can't find the container with id d7452c10094d65664e60d3e34fdff39bbf5c56880954582560bd8ab989e6d418
Oct 31 01:22:47.684167 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:22:47.684104    4964 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-75984d866d-k5g7h" podStartSLOduration=2.684089592 podStartE2EDuration="2.684089592s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:22:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:22:47.68068111 +0000 UTC m=+1136.157247483" watchObservedRunningTime="2026-10-31 01:22:47.684089592 +0000 UTC m=+1136.160656017"
Oct 31 01:22:47.807739 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:22:47.807706    4964 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-a8e157/distinct-2-w3-a8e157-kserve-9ddd45d6-ng7j7"
Oct 31 01:22:47.821502 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:22:47.821420    4964 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-a8e157/distinct-2-w3-a8e157-kserve-9ddd45d6-ng7j7"
Oct 31 01:22:48.575194 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:22:48.575161    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-baacee/distinct-w1-baacee-kserve-6c85cfb7d7-stkqq" event={"ID":"df6d0bdd-382d-49b7-879c-15cb5371246e","Type":"ContainerStarted","Data":"aee05ce5bac0b4342a425b1a06eabf8e95c84855bec79021dec94dccfc04eafb"}
Oct 31 01:22:48.575194 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:22:48.575205    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-baacee/distinct-w1-baacee-kserve-6c85cfb7d7-stkqq" event={"ID":"df6d0bdd-382d-49b7-879c-15cb5371246e","Type":"ContainerStarted","Data":"d7452c10094d65664e60d3e34fdff39bbf5c56880954582560bd8ab989e6d418"}
Oct 31 01:22:48.587069 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:22:48.587045    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-a8e157/distinct-2-w3-a8e157-kserve-9ddd45d6-ng7j7"
Oct 31 01:22:48.691433 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:22:48.691381    4964 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-baacee/distinct-w1-baacee-kserve-6c85cfb7d7-stkqq" podStartSLOduration=2.691366279 podStartE2EDuration="2.691366279s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:22:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:22:48.690154912 +0000 UTC m=+1137.166721293" watchObservedRunningTime="2026-10-31 01:22:48.691366279 +0000 UTC m=+1137.167932659"
Oct 31 01:22:48.931736 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:22:48.931663    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:22:49.207826 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:22:49.207753    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-baacee/distinct-w1-baacee-kserve-6c85cfb7d7-stkqq"
Oct 31 01:22:54.250382 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:22:54.250304    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:22:57.207654 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:22:57.207620    4964 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-baacee/distinct-w1-baacee-kserve-6c85cfb7d7-stkqq"
Oct 31 01:22:57.221603 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:22:57.221579    4964 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-baacee/distinct-w1-baacee-kserve-6c85cfb7d7-stkqq"
Oct 31 01:22:57.740293 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:22:57.740260    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-baacee/distinct-w1-baacee-kserve-6c85cfb7d7-stkqq"
Oct 31 01:22:58.578789 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:22:58.578761    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-75984d866d-k5g7h"
Oct 31 01:22:59.116033 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:22:59.115987    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:23:00.996934 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:23:00.996888    4964 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/periodic-gathering-njx9c-pt7d5"]
Oct 31 01:23:01.002380 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:23:01.002354    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-njx9c-pt7d5"
Oct 31 01:23:01.012556 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:23:01.012526    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/d4ef3da1-2732-4051-be2a-5aba6f5fbcfc-archives-path\") pod \"periodic-gathering-njx9c-pt7d5\" (UID: \"d4ef3da1-2732-4051-be2a-5aba6f5fbcfc\") " pod="openshift-insights/periodic-gathering-njx9c-pt7d5"
Oct 31 01:23:01.012700 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:23:01.012676    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d4ef3da1-2732-4051-be2a-5aba6f5fbcfc-trusted-ca-bundle\") pod \"periodic-gathering-njx9c-pt7d5\" (UID: \"d4ef3da1-2732-4051-be2a-5aba6f5fbcfc\") " pod="openshift-insights/periodic-gathering-njx9c-pt7d5"
Oct 31 01:23:01.012825 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:23:01.012809    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n6r8h\" (UniqueName: \"kubernetes.io/projected/d4ef3da1-2732-4051-be2a-5aba6f5fbcfc-kube-api-access-n6r8h\") pod \"periodic-gathering-njx9c-pt7d5\" (UID: \"d4ef3da1-2732-4051-be2a-5aba6f5fbcfc\") " pod="openshift-insights/periodic-gathering-njx9c-pt7d5"
Oct 31 01:23:01.012886 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:23:01.012872    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d4ef3da1-2732-4051-be2a-5aba6f5fbcfc-service-ca-bundle\") pod \"periodic-gathering-njx9c-pt7d5\" (UID: \"d4ef3da1-2732-4051-be2a-5aba6f5fbcfc\") " pod="openshift-insights/periodic-gathering-njx9c-pt7d5"
Oct 31 01:23:01.103883 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:23:01.103853    4964 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-fbmr7\""
Oct 31 01:23:01.106937 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:23:01.106915    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-njx9c-pt7d5"]
Oct 31 01:23:01.113952 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:23:01.113923    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/d4ef3da1-2732-4051-be2a-5aba6f5fbcfc-archives-path\") pod \"periodic-gathering-njx9c-pt7d5\" (UID: \"d4ef3da1-2732-4051-be2a-5aba6f5fbcfc\") " pod="openshift-insights/periodic-gathering-njx9c-pt7d5"
Oct 31 01:23:01.114077 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:23:01.114021    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d4ef3da1-2732-4051-be2a-5aba6f5fbcfc-trusted-ca-bundle\") pod \"periodic-gathering-njx9c-pt7d5\" (UID: \"d4ef3da1-2732-4051-be2a-5aba6f5fbcfc\") " pod="openshift-insights/periodic-gathering-njx9c-pt7d5"
Oct 31 01:23:01.114145 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:23:01.114117    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n6r8h\" (UniqueName: \"kubernetes.io/projected/d4ef3da1-2732-4051-be2a-5aba6f5fbcfc-kube-api-access-n6r8h\") pod \"periodic-gathering-njx9c-pt7d5\" (UID: \"d4ef3da1-2732-4051-be2a-5aba6f5fbcfc\") " pod="openshift-insights/periodic-gathering-njx9c-pt7d5"
Oct 31 01:23:01.114206 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:23:01.114183    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d4ef3da1-2732-4051-be2a-5aba6f5fbcfc-service-ca-bundle\") pod \"periodic-gathering-njx9c-pt7d5\" (UID: \"d4ef3da1-2732-4051-be2a-5aba6f5fbcfc\") " pod="openshift-insights/periodic-gathering-njx9c-pt7d5"
Oct 31 01:23:01.114392 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:23:01.114373    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/d4ef3da1-2732-4051-be2a-5aba6f5fbcfc-archives-path\") pod \"periodic-gathering-njx9c-pt7d5\" (UID: \"d4ef3da1-2732-4051-be2a-5aba6f5fbcfc\") " pod="openshift-insights/periodic-gathering-njx9c-pt7d5"
Oct 31 01:23:01.114974 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:23:01.114951    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d4ef3da1-2732-4051-be2a-5aba6f5fbcfc-trusted-ca-bundle\") pod \"periodic-gathering-njx9c-pt7d5\" (UID: \"d4ef3da1-2732-4051-be2a-5aba6f5fbcfc\") " pod="openshift-insights/periodic-gathering-njx9c-pt7d5"
Oct 31 01:23:01.115364 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:23:01.115339    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d4ef3da1-2732-4051-be2a-5aba6f5fbcfc-service-ca-bundle\") pod \"periodic-gathering-njx9c-pt7d5\" (UID: \"d4ef3da1-2732-4051-be2a-5aba6f5fbcfc\") " pod="openshift-insights/periodic-gathering-njx9c-pt7d5"
Oct 31 01:23:01.170679 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:23:01.170652    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n6r8h\" (UniqueName: \"kubernetes.io/projected/d4ef3da1-2732-4051-be2a-5aba6f5fbcfc-kube-api-access-n6r8h\") pod \"periodic-gathering-njx9c-pt7d5\" (UID: \"d4ef3da1-2732-4051-be2a-5aba6f5fbcfc\") " pod="openshift-insights/periodic-gathering-njx9c-pt7d5"
Oct 31 01:23:01.314948 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:23:01.314862    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-njx9c-pt7d5"
Oct 31 01:23:01.692571 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:23:01.692545    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-njx9c-pt7d5"]
Oct 31 01:23:01.694579 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:23:01.694546    4964 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd4ef3da1_2732_4051_be2a_5aba6f5fbcfc.slice/crio-dd831a46aff74e979cc8ec90cbda0d740c48f05eac75c57b00ed5cfef2daed9e WatchSource:0}: Error finding container dd831a46aff74e979cc8ec90cbda0d740c48f05eac75c57b00ed5cfef2daed9e: Status 404 returned error can't find the container with id dd831a46aff74e979cc8ec90cbda0d740c48f05eac75c57b00ed5cfef2daed9e
Oct 31 01:23:01.760191 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:23:01.760166    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-njx9c-pt7d5" event={"ID":"d4ef3da1-2732-4051-be2a-5aba6f5fbcfc","Type":"ContainerStarted","Data":"dd831a46aff74e979cc8ec90cbda0d740c48f05eac75c57b00ed5cfef2daed9e"}
Oct 31 01:23:02.767790 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:23:02.767752    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-njx9c-pt7d5" event={"ID":"d4ef3da1-2732-4051-be2a-5aba6f5fbcfc","Type":"ContainerStarted","Data":"abf4433a52ea15fccc9256feb0387c00be247fe08023a7a5928d074f2b5131ac"}
Oct 31 01:23:02.882963 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:23:02.882913    4964 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/periodic-gathering-njx9c-pt7d5" podStartSLOduration=2.88289885 podStartE2EDuration="2.88289885s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:23:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:23:02.880863951 +0000 UTC m=+1151.357430333" watchObservedRunningTime="2026-10-31 01:23:02.88289885 +0000 UTC m=+1151.359465209"
Oct 31 01:23:03.895635 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:23:03.895571    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:23:06.513475 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:23:06.513408    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:23:08.742740 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:23:08.742708    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:23:09.656948 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:23:09.656913    4964 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-7rdlf"]
Oct 31 01:23:09.661999 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:23:09.661974    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-7rdlf"
Oct 31 01:23:09.714832 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:23:09.714794    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/11f27521-38d0-47d3-8884-accc44d652cb-istio-envoy\") pod \"e2e-worker-w3-a8e157-openshift-default-7fb96689c7-7rdlf\" (UID: \"11f27521-38d0-47d3-8884-accc44d652cb\") " pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-7rdlf"
Oct 31 01:23:09.715077 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:23:09.715056    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8fh6j\" (UniqueName: \"kubernetes.io/projected/11f27521-38d0-47d3-8884-accc44d652cb-kube-api-access-8fh6j\") pod \"e2e-worker-w3-a8e157-openshift-default-7fb96689c7-7rdlf\" (UID: \"11f27521-38d0-47d3-8884-accc44d652cb\") " pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-7rdlf"
Oct 31 01:23:09.715225 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:23:09.715209    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/11f27521-38d0-47d3-8884-accc44d652cb-istio-token\") pod \"e2e-worker-w3-a8e157-openshift-default-7fb96689c7-7rdlf\" (UID: \"11f27521-38d0-47d3-8884-accc44d652cb\") " pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-7rdlf"
Oct 31 01:23:09.715382 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:23:09.715365    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/11f27521-38d0-47d3-8884-accc44d652cb-workload-socket\") pod \"e2e-worker-w3-a8e157-openshift-default-7fb96689c7-7rdlf\" (UID: \"11f27521-38d0-47d3-8884-accc44d652cb\") " pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-7rdlf"
Oct 31 01:23:09.715537 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:23:09.715512    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/11f27521-38d0-47d3-8884-accc44d652cb-credential-socket\") pod \"e2e-worker-w3-a8e157-openshift-default-7fb96689c7-7rdlf\" (UID: \"11f27521-38d0-47d3-8884-accc44d652cb\") " pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-7rdlf"
Oct 31 01:23:09.715664 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:23:09.715581    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/11f27521-38d0-47d3-8884-accc44d652cb-istiod-ca-cert\") pod \"e2e-worker-w3-a8e157-openshift-default-7fb96689c7-7rdlf\" (UID: \"11f27521-38d0-47d3-8884-accc44d652cb\") " pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-7rdlf"
Oct 31 01:23:09.715664 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:23:09.715637    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/11f27521-38d0-47d3-8884-accc44d652cb-workload-certs\") pod \"e2e-worker-w3-a8e157-openshift-default-7fb96689c7-7rdlf\" (UID: \"11f27521-38d0-47d3-8884-accc44d652cb\") " pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-7rdlf"
Oct 31 01:23:09.715847 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:23:09.715741    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/11f27521-38d0-47d3-8884-accc44d652cb-istio-data\") pod \"e2e-worker-w3-a8e157-openshift-default-7fb96689c7-7rdlf\" (UID: \"11f27521-38d0-47d3-8884-accc44d652cb\") " pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-7rdlf"
Oct 31 01:23:09.715847 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:23:09.715771    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/11f27521-38d0-47d3-8884-accc44d652cb-istio-podinfo\") pod \"e2e-worker-w3-a8e157-openshift-default-7fb96689c7-7rdlf\" (UID: \"11f27521-38d0-47d3-8884-accc44d652cb\") " pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-7rdlf"
Oct 31 01:23:09.817246 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:23:09.817210    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/11f27521-38d0-47d3-8884-accc44d652cb-workload-certs\") pod \"e2e-worker-w3-a8e157-openshift-default-7fb96689c7-7rdlf\" (UID: \"11f27521-38d0-47d3-8884-accc44d652cb\") " pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-7rdlf"
Oct 31 01:23:09.817846 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:23:09.817814    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/11f27521-38d0-47d3-8884-accc44d652cb-workload-certs\") pod \"e2e-worker-w3-a8e157-openshift-default-7fb96689c7-7rdlf\" (UID: \"11f27521-38d0-47d3-8884-accc44d652cb\") " pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-7rdlf"
Oct 31 01:23:09.818059 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:23:09.818035    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/11f27521-38d0-47d3-8884-accc44d652cb-istio-data\") pod \"e2e-worker-w3-a8e157-openshift-default-7fb96689c7-7rdlf\" (UID: \"11f27521-38d0-47d3-8884-accc44d652cb\") " pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-7rdlf"
Oct 31 01:23:09.818227 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:23:09.818208    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/11f27521-38d0-47d3-8884-accc44d652cb-istio-podinfo\") pod \"e2e-worker-w3-a8e157-openshift-default-7fb96689c7-7rdlf\" (UID: \"11f27521-38d0-47d3-8884-accc44d652cb\") " pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-7rdlf"
Oct 31 01:23:09.818962 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:23:09.818940    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/11f27521-38d0-47d3-8884-accc44d652cb-istio-envoy\") pod \"e2e-worker-w3-a8e157-openshift-default-7fb96689c7-7rdlf\" (UID: \"11f27521-38d0-47d3-8884-accc44d652cb\") " pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-7rdlf"
Oct 31 01:23:09.819127 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:23:09.819110    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8fh6j\" (UniqueName: \"kubernetes.io/projected/11f27521-38d0-47d3-8884-accc44d652cb-kube-api-access-8fh6j\") pod \"e2e-worker-w3-a8e157-openshift-default-7fb96689c7-7rdlf\" (UID: \"11f27521-38d0-47d3-8884-accc44d652cb\") " pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-7rdlf"
Oct 31 01:23:09.819247 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:23:09.819233    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/11f27521-38d0-47d3-8884-accc44d652cb-istio-token\") pod \"e2e-worker-w3-a8e157-openshift-default-7fb96689c7-7rdlf\" (UID: \"11f27521-38d0-47d3-8884-accc44d652cb\") " pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-7rdlf"
Oct 31 01:23:09.819416 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:23:09.819400    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/11f27521-38d0-47d3-8884-accc44d652cb-workload-socket\") pod \"e2e-worker-w3-a8e157-openshift-default-7fb96689c7-7rdlf\" (UID: \"11f27521-38d0-47d3-8884-accc44d652cb\") " pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-7rdlf"
Oct 31 01:23:09.819557 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:23:09.819541    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/11f27521-38d0-47d3-8884-accc44d652cb-credential-socket\") pod \"e2e-worker-w3-a8e157-openshift-default-7fb96689c7-7rdlf\" (UID: \"11f27521-38d0-47d3-8884-accc44d652cb\") " pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-7rdlf"
Oct 31 01:23:09.819712 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:23:09.819696    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/11f27521-38d0-47d3-8884-accc44d652cb-istiod-ca-cert\") pod \"e2e-worker-w3-a8e157-openshift-default-7fb96689c7-7rdlf\" (UID: \"11f27521-38d0-47d3-8884-accc44d652cb\") " pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-7rdlf"
Oct 31 01:23:09.820428 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:23:09.818497    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/11f27521-38d0-47d3-8884-accc44d652cb-istio-data\") pod \"e2e-worker-w3-a8e157-openshift-default-7fb96689c7-7rdlf\" (UID: \"11f27521-38d0-47d3-8884-accc44d652cb\") " pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-7rdlf"
Oct 31 01:23:09.820602 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:23:09.820577    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/11f27521-38d0-47d3-8884-accc44d652cb-istiod-ca-cert\") pod \"e2e-worker-w3-a8e157-openshift-default-7fb96689c7-7rdlf\" (UID: \"11f27521-38d0-47d3-8884-accc44d652cb\") " pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-7rdlf"
Oct 31 01:23:09.821146 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:23:09.821120    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/11f27521-38d0-47d3-8884-accc44d652cb-workload-socket\") pod \"e2e-worker-w3-a8e157-openshift-default-7fb96689c7-7rdlf\" (UID: \"11f27521-38d0-47d3-8884-accc44d652cb\") " pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-7rdlf"
Oct 31 01:23:09.821461 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:23:09.821439    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/11f27521-38d0-47d3-8884-accc44d652cb-credential-socket\") pod \"e2e-worker-w3-a8e157-openshift-default-7fb96689c7-7rdlf\" (UID: \"11f27521-38d0-47d3-8884-accc44d652cb\") " pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-7rdlf"
Oct 31 01:23:09.822731 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:23:09.822707    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/11f27521-38d0-47d3-8884-accc44d652cb-istio-podinfo\") pod \"e2e-worker-w3-a8e157-openshift-default-7fb96689c7-7rdlf\" (UID: \"11f27521-38d0-47d3-8884-accc44d652cb\") " pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-7rdlf"
Oct 31 01:23:09.832050 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:23:09.831985    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/11f27521-38d0-47d3-8884-accc44d652cb-istio-envoy\") pod \"e2e-worker-w3-a8e157-openshift-default-7fb96689c7-7rdlf\" (UID: \"11f27521-38d0-47d3-8884-accc44d652cb\") " pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-7rdlf"
Oct 31 01:23:09.863473 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:23:09.863441    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-7rdlf"]
Oct 31 01:23:09.893377 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:23:09.893303    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8fh6j\" (UniqueName: \"kubernetes.io/projected/11f27521-38d0-47d3-8884-accc44d652cb-kube-api-access-8fh6j\") pod \"e2e-worker-w3-a8e157-openshift-default-7fb96689c7-7rdlf\" (UID: \"11f27521-38d0-47d3-8884-accc44d652cb\") " pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-7rdlf"
Oct 31 01:23:09.896512 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:23:09.896461    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/11f27521-38d0-47d3-8884-accc44d652cb-istio-token\") pod \"e2e-worker-w3-a8e157-openshift-default-7fb96689c7-7rdlf\" (UID: \"11f27521-38d0-47d3-8884-accc44d652cb\") " pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-7rdlf"
Oct 31 01:23:09.981748 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:23:09.981398    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-7rdlf"
Oct 31 01:23:10.467022 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:23:10.466990    4964 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod11f27521_38d0_47d3_8884_accc44d652cb.slice/crio-7e569dfa1a72d8facf5ef11d7513542ee8a36f6b0013a2bddeac16d79fd32d7d WatchSource:0}: Error finding container 7e569dfa1a72d8facf5ef11d7513542ee8a36f6b0013a2bddeac16d79fd32d7d: Status 404 returned error can't find the container with id 7e569dfa1a72d8facf5ef11d7513542ee8a36f6b0013a2bddeac16d79fd32d7d
Oct 31 01:23:10.483341 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:23:10.469511    4964 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 01:23:10.483341 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:23:10.469600    4964 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 01:23:10.483341 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:23:10.469641    4964 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 01:23:10.486527 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:23:10.486111    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-7rdlf"]
Oct 31 01:23:10.967104 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:23:10.966849    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-7rdlf" event={"ID":"11f27521-38d0-47d3-8884-accc44d652cb","Type":"ContainerStarted","Data":"7e569dfa1a72d8facf5ef11d7513542ee8a36f6b0013a2bddeac16d79fd32d7d"}
Oct 31 01:23:11.356029 ip-10-0-133-197 kubenswrapper[4964]: E1031 01:23:11.355853    4964 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q56x5" containerName="extractor"
Oct 31 01:23:12.031340 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:23:12.028504    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-7rdlf" event={"ID":"11f27521-38d0-47d3-8884-accc44d652cb","Type":"ContainerStarted","Data":"8db46917c1436ffe81adc65b48b30690d2948bb71f66b67021c430a74b6368e9"}
Oct 31 01:23:12.349211 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:23:12.349094    4964 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-7rdlf" podStartSLOduration=3.349074576 podStartE2EDuration="3.349074576s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:23:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:23:12.210802215 +0000 UTC m=+1160.687368610" watchObservedRunningTime="2026-10-31 01:23:12.349074576 +0000 UTC m=+1160.825640956"
Oct 31 01:23:12.982486 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:23:12.982456    4964 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-7rdlf"
Oct 31 01:23:12.984292 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:23:12.984265    4964 patch_prober.go:29] interesting pod/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-7rdlf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body=
Oct 31 01:23:12.984439 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:23:12.984346    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-7rdlf" podUID="11f27521-38d0-47d3-8884-accc44d652cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused"
Oct 31 01:23:13.982880 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:23:13.982843    4964 patch_prober.go:29] interesting pod/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-7rdlf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body=
Oct 31 01:23:13.983305 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:23:13.982920    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-7rdlf" podUID="11f27521-38d0-47d3-8884-accc44d652cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused"
Oct 31 01:23:14.221612 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:23:14.221568    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:23:14.982777 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:23:14.982741    4964 patch_prober.go:29] interesting pod/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-7rdlf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body=
Oct 31 01:23:14.982963 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:23:14.982802    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-7rdlf" podUID="11f27521-38d0-47d3-8884-accc44d652cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused"
Oct 31 01:23:15.982946 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:23:15.982903    4964 patch_prober.go:29] interesting pod/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-7rdlf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body=
Oct 31 01:23:15.983471 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:23:15.982957    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-7rdlf" podUID="11f27521-38d0-47d3-8884-accc44d652cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused"
Oct 31 01:23:16.982973 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:23:16.982938    4964 patch_prober.go:29] interesting pod/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-7rdlf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body=
Oct 31 01:23:16.983409 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:23:16.983013    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-7rdlf" podUID="11f27521-38d0-47d3-8884-accc44d652cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused"
Oct 31 01:23:17.982362 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:23:17.982307    4964 patch_prober.go:29] interesting pod/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-7rdlf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body=
Oct 31 01:23:17.982544 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:23:17.982389    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-7rdlf" podUID="11f27521-38d0-47d3-8884-accc44d652cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused"
Oct 31 01:23:18.986799 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:23:18.986771    4964 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-7rdlf"
Oct 31 01:23:18.987277 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:23:18.987256    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-7rdlf"
Oct 31 01:23:18.988014 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:23:18.987994    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-7rdlf"
Oct 31 01:23:19.125854 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:23:19.125815    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:23:23.888694 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:23:23.888656    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:23:25.216177 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:23:25.216141    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:23:28.901158 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:23:28.901118    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:23:33.507798 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:23:33.507768    4964 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-qvt9t"]
Oct 31 01:23:33.513277 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:23:33.513259    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-qvt9t"
Oct 31 01:23:33.564873 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:23:33.564847    4964 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-baacee-openshift-default-dockercfg-pr4tt\""
Oct 31 01:23:33.619680 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:23:33.619647    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-qvt9t"]
Oct 31 01:23:33.699782 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:23:33.699749    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/247208c2-428e-4f58-8dff-55939802b345-istiod-ca-cert\") pod \"e2e-worker-w1-baacee-openshift-default-57f845d5bf-qvt9t\" (UID: \"247208c2-428e-4f58-8dff-55939802b345\") " pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-qvt9t"
Oct 31 01:23:33.699969 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:23:33.699866    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/247208c2-428e-4f58-8dff-55939802b345-credential-socket\") pod \"e2e-worker-w1-baacee-openshift-default-57f845d5bf-qvt9t\" (UID: \"247208c2-428e-4f58-8dff-55939802b345\") " pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-qvt9t"
Oct 31 01:23:33.699969 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:23:33.699901    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pbpmh\" (UniqueName: \"kubernetes.io/projected/247208c2-428e-4f58-8dff-55939802b345-kube-api-access-pbpmh\") pod \"e2e-worker-w1-baacee-openshift-default-57f845d5bf-qvt9t\" (UID: \"247208c2-428e-4f58-8dff-55939802b345\") " pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-qvt9t"
Oct 31 01:23:33.700085 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:23:33.700055    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/247208c2-428e-4f58-8dff-55939802b345-istio-token\") pod \"e2e-worker-w1-baacee-openshift-default-57f845d5bf-qvt9t\" (UID: \"247208c2-428e-4f58-8dff-55939802b345\") " pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-qvt9t"
Oct 31 01:23:33.700137 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:23:33.700097    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/247208c2-428e-4f58-8dff-55939802b345-istio-podinfo\") pod \"e2e-worker-w1-baacee-openshift-default-57f845d5bf-qvt9t\" (UID: \"247208c2-428e-4f58-8dff-55939802b345\") " pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-qvt9t"
Oct 31 01:23:33.700190 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:23:33.700148    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/247208c2-428e-4f58-8dff-55939802b345-istio-envoy\") pod \"e2e-worker-w1-baacee-openshift-default-57f845d5bf-qvt9t\" (UID: \"247208c2-428e-4f58-8dff-55939802b345\") " pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-qvt9t"
Oct 31 01:23:33.700190 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:23:33.700180    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/247208c2-428e-4f58-8dff-55939802b345-workload-certs\") pod \"e2e-worker-w1-baacee-openshift-default-57f845d5bf-qvt9t\" (UID: \"247208c2-428e-4f58-8dff-55939802b345\") " pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-qvt9t"
Oct 31 01:23:33.700291 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:23:33.700238    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/247208c2-428e-4f58-8dff-55939802b345-istio-data\") pod \"e2e-worker-w1-baacee-openshift-default-57f845d5bf-qvt9t\" (UID: \"247208c2-428e-4f58-8dff-55939802b345\") " pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-qvt9t"
Oct 31 01:23:33.700291 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:23:33.700285    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/247208c2-428e-4f58-8dff-55939802b345-workload-socket\") pod \"e2e-worker-w1-baacee-openshift-default-57f845d5bf-qvt9t\" (UID: \"247208c2-428e-4f58-8dff-55939802b345\") " pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-qvt9t"
Oct 31 01:23:33.798838 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:23:33.798738    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:23:33.801850 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:23:33.801823    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/247208c2-428e-4f58-8dff-55939802b345-credential-socket\") pod \"e2e-worker-w1-baacee-openshift-default-57f845d5bf-qvt9t\" (UID: \"247208c2-428e-4f58-8dff-55939802b345\") " pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-qvt9t"
Oct 31 01:23:33.802034 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:23:33.801867    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pbpmh\" (UniqueName: \"kubernetes.io/projected/247208c2-428e-4f58-8dff-55939802b345-kube-api-access-pbpmh\") pod \"e2e-worker-w1-baacee-openshift-default-57f845d5bf-qvt9t\" (UID: \"247208c2-428e-4f58-8dff-55939802b345\") " pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-qvt9t"
Oct 31 01:23:33.802034 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:23:33.801950    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/247208c2-428e-4f58-8dff-55939802b345-istio-token\") pod \"e2e-worker-w1-baacee-openshift-default-57f845d5bf-qvt9t\" (UID: \"247208c2-428e-4f58-8dff-55939802b345\") " pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-qvt9t"
Oct 31 01:23:33.802034 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:23:33.801981    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/247208c2-428e-4f58-8dff-55939802b345-istio-podinfo\") pod \"e2e-worker-w1-baacee-openshift-default-57f845d5bf-qvt9t\" (UID: \"247208c2-428e-4f58-8dff-55939802b345\") " pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-qvt9t"
Oct 31 01:23:33.802034 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:23:33.801999    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/247208c2-428e-4f58-8dff-55939802b345-istio-envoy\") pod \"e2e-worker-w1-baacee-openshift-default-57f845d5bf-qvt9t\" (UID: \"247208c2-428e-4f58-8dff-55939802b345\") " pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-qvt9t"
Oct 31 01:23:33.802034 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:23:33.802022    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/247208c2-428e-4f58-8dff-55939802b345-workload-certs\") pod \"e2e-worker-w1-baacee-openshift-default-57f845d5bf-qvt9t\" (UID: \"247208c2-428e-4f58-8dff-55939802b345\") " pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-qvt9t"
Oct 31 01:23:33.802221 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:23:33.802064    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/247208c2-428e-4f58-8dff-55939802b345-istio-data\") pod \"e2e-worker-w1-baacee-openshift-default-57f845d5bf-qvt9t\" (UID: \"247208c2-428e-4f58-8dff-55939802b345\") " pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-qvt9t"
Oct 31 01:23:33.802221 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:23:33.802090    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/247208c2-428e-4f58-8dff-55939802b345-workload-socket\") pod \"e2e-worker-w1-baacee-openshift-default-57f845d5bf-qvt9t\" (UID: \"247208c2-428e-4f58-8dff-55939802b345\") " pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-qvt9t"
Oct 31 01:23:33.802221 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:23:33.802132    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/247208c2-428e-4f58-8dff-55939802b345-istiod-ca-cert\") pod \"e2e-worker-w1-baacee-openshift-default-57f845d5bf-qvt9t\" (UID: \"247208c2-428e-4f58-8dff-55939802b345\") " pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-qvt9t"
Oct 31 01:23:33.802351 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:23:33.802266    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/247208c2-428e-4f58-8dff-55939802b345-credential-socket\") pod \"e2e-worker-w1-baacee-openshift-default-57f845d5bf-qvt9t\" (UID: \"247208c2-428e-4f58-8dff-55939802b345\") " pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-qvt9t"
Oct 31 01:23:33.802751 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:23:33.802723    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/247208c2-428e-4f58-8dff-55939802b345-workload-socket\") pod \"e2e-worker-w1-baacee-openshift-default-57f845d5bf-qvt9t\" (UID: \"247208c2-428e-4f58-8dff-55939802b345\") " pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-qvt9t"
Oct 31 01:23:33.802885 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:23:33.802797    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/247208c2-428e-4f58-8dff-55939802b345-workload-certs\") pod \"e2e-worker-w1-baacee-openshift-default-57f845d5bf-qvt9t\" (UID: \"247208c2-428e-4f58-8dff-55939802b345\") " pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-qvt9t"
Oct 31 01:23:33.802885 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:23:33.802820    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/247208c2-428e-4f58-8dff-55939802b345-istio-data\") pod \"e2e-worker-w1-baacee-openshift-default-57f845d5bf-qvt9t\" (UID: \"247208c2-428e-4f58-8dff-55939802b345\") " pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-qvt9t"
Oct 31 01:23:33.803006 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:23:33.802893    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/247208c2-428e-4f58-8dff-55939802b345-istiod-ca-cert\") pod \"e2e-worker-w1-baacee-openshift-default-57f845d5bf-qvt9t\" (UID: \"247208c2-428e-4f58-8dff-55939802b345\") " pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-qvt9t"
Oct 31 01:23:33.805136 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:23:33.805110    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/247208c2-428e-4f58-8dff-55939802b345-istio-envoy\") pod \"e2e-worker-w1-baacee-openshift-default-57f845d5bf-qvt9t\" (UID: \"247208c2-428e-4f58-8dff-55939802b345\") " pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-qvt9t"
Oct 31 01:23:33.805415 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:23:33.805394    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/247208c2-428e-4f58-8dff-55939802b345-istio-podinfo\") pod \"e2e-worker-w1-baacee-openshift-default-57f845d5bf-qvt9t\" (UID: \"247208c2-428e-4f58-8dff-55939802b345\") " pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-qvt9t"
Oct 31 01:23:33.858927 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:23:33.858899    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/247208c2-428e-4f58-8dff-55939802b345-istio-token\") pod \"e2e-worker-w1-baacee-openshift-default-57f845d5bf-qvt9t\" (UID: \"247208c2-428e-4f58-8dff-55939802b345\") " pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-qvt9t"
Oct 31 01:23:33.859356 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:23:33.859311    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pbpmh\" (UniqueName: \"kubernetes.io/projected/247208c2-428e-4f58-8dff-55939802b345-kube-api-access-pbpmh\") pod \"e2e-worker-w1-baacee-openshift-default-57f845d5bf-qvt9t\" (UID: \"247208c2-428e-4f58-8dff-55939802b345\") " pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-qvt9t"
Oct 31 01:23:34.127269 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:23:34.127198    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-qvt9t"
Oct 31 01:23:34.561524 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:23:34.561496    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-qvt9t"]
Oct 31 01:23:34.562439 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:23:34.562398    4964 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod247208c2_428e_4f58_8dff_55939802b345.slice/crio-45722704ca95cf9b8e22389fae1ea4c2b330c7485ff7c24cea51129cbfb2454e WatchSource:0}: Error finding container 45722704ca95cf9b8e22389fae1ea4c2b330c7485ff7c24cea51129cbfb2454e: Status 404 returned error can't find the container with id 45722704ca95cf9b8e22389fae1ea4c2b330c7485ff7c24cea51129cbfb2454e
Oct 31 01:23:34.564791 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:23:34.564756    4964 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 01:23:34.564895 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:23:34.564832    4964 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 01:23:34.564895 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:23:34.564873    4964 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 01:23:35.234052 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:23:35.234011    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-qvt9t" event={"ID":"247208c2-428e-4f58-8dff-55939802b345","Type":"ContainerStarted","Data":"257eafd62630265281c3a7219833e8dc6bb23f8dd14c0b1eaa914618b6196b32"}
Oct 31 01:23:35.234247 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:23:35.234080    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-qvt9t" event={"ID":"247208c2-428e-4f58-8dff-55939802b345","Type":"ContainerStarted","Data":"45722704ca95cf9b8e22389fae1ea4c2b330c7485ff7c24cea51129cbfb2454e"}
Oct 31 01:23:35.412342 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:23:35.408010    4964 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-qvt9t" podStartSLOduration=2.407990072 podStartE2EDuration="2.407990072s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:23:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:23:35.403468469 +0000 UTC m=+1183.880034850" watchObservedRunningTime="2026-10-31 01:23:35.407990072 +0000 UTC m=+1183.884556453"
Oct 31 01:23:36.045632 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:23:36.045594    4964 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-jz4tz"]
Oct 31 01:23:36.049979 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:23:36.049956    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-jz4tz"
Oct 31 01:23:36.127407 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:23:36.127369    4964 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-qvt9t"
Oct 31 01:23:36.129423 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:23:36.129378    4964 patch_prober.go:29] interesting pod/e2e-worker-w1-baacee-openshift-default-57f845d5bf-qvt9t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body=
Oct 31 01:23:36.129556 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:23:36.129460    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-qvt9t" podUID="247208c2-428e-4f58-8dff-55939802b345" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused"
Oct 31 01:23:36.130087 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:23:36.130062    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a2038dbc-765f-488b-9701-c4081448ac4a-istio-podinfo\") pod \"e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-jz4tz\" (UID: \"a2038dbc-765f-488b-9701-c4081448ac4a\") " pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-jz4tz"
Oct 31 01:23:36.130196 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:23:36.130160    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2038dbc-765f-488b-9701-c4081448ac4a-workload-socket\") pod \"e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-jz4tz\" (UID: \"a2038dbc-765f-488b-9701-c4081448ac4a\") " pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-jz4tz"
Oct 31 01:23:36.130261 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:23:36.130233    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a2038dbc-765f-488b-9701-c4081448ac4a-istio-token\") pod \"e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-jz4tz\" (UID: \"a2038dbc-765f-488b-9701-c4081448ac4a\") " pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-jz4tz"
Oct 31 01:23:36.130361 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:23:36.130264    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a2038dbc-765f-488b-9701-c4081448ac4a-istio-envoy\") pod \"e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-jz4tz\" (UID: \"a2038dbc-765f-488b-9701-c4081448ac4a\") " pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-jz4tz"
Oct 31 01:23:36.130361 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:23:36.130290    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a2038dbc-765f-488b-9701-c4081448ac4a-istio-data\") pod \"e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-jz4tz\" (UID: \"a2038dbc-765f-488b-9701-c4081448ac4a\") " pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-jz4tz"
Oct 31 01:23:36.130361 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:23:36.130340    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a2038dbc-765f-488b-9701-c4081448ac4a-workload-certs\") pod \"e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-jz4tz\" (UID: \"a2038dbc-765f-488b-9701-c4081448ac4a\") " pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-jz4tz"
Oct 31 01:23:36.130583 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:23:36.130380    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2038dbc-765f-488b-9701-c4081448ac4a-credential-socket\") pod \"e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-jz4tz\" (UID: \"a2038dbc-765f-488b-9701-c4081448ac4a\") " pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-jz4tz"
Oct 31 01:23:36.130583 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:23:36.130419    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c7jcs\" (UniqueName: \"kubernetes.io/projected/a2038dbc-765f-488b-9701-c4081448ac4a-kube-api-access-c7jcs\") pod \"e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-jz4tz\" (UID: \"a2038dbc-765f-488b-9701-c4081448ac4a\") " pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-jz4tz"
Oct 31 01:23:36.130583 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:23:36.130448    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a2038dbc-765f-488b-9701-c4081448ac4a-istiod-ca-cert\") pod \"e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-jz4tz\" (UID: \"a2038dbc-765f-488b-9701-c4081448ac4a\") " pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-jz4tz"
Oct 31 01:23:36.149729 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:23:36.149701    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-jz4tz"]
Oct 31 01:23:36.231597 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:23:36.231555    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a2038dbc-765f-488b-9701-c4081448ac4a-workload-certs\") pod \"e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-jz4tz\" (UID: \"a2038dbc-765f-488b-9701-c4081448ac4a\") " pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-jz4tz"
Oct 31 01:23:36.231781 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:23:36.231625    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2038dbc-765f-488b-9701-c4081448ac4a-credential-socket\") pod \"e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-jz4tz\" (UID: \"a2038dbc-765f-488b-9701-c4081448ac4a\") " pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-jz4tz"
Oct 31 01:23:36.231781 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:23:36.231680    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c7jcs\" (UniqueName: \"kubernetes.io/projected/a2038dbc-765f-488b-9701-c4081448ac4a-kube-api-access-c7jcs\") pod \"e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-jz4tz\" (UID: \"a2038dbc-765f-488b-9701-c4081448ac4a\") " pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-jz4tz"
Oct 31 01:23:36.231781 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:23:36.231722    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a2038dbc-765f-488b-9701-c4081448ac4a-istiod-ca-cert\") pod \"e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-jz4tz\" (UID: \"a2038dbc-765f-488b-9701-c4081448ac4a\") " pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-jz4tz"
Oct 31 01:23:36.231897 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:23:36.231799    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a2038dbc-765f-488b-9701-c4081448ac4a-istio-podinfo\") pod \"e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-jz4tz\" (UID: \"a2038dbc-765f-488b-9701-c4081448ac4a\") " pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-jz4tz"
Oct 31 01:23:36.231897 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:23:36.231846    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2038dbc-765f-488b-9701-c4081448ac4a-workload-socket\") pod \"e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-jz4tz\" (UID: \"a2038dbc-765f-488b-9701-c4081448ac4a\") " pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-jz4tz"
Oct 31 01:23:36.231978 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:23:36.231932    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a2038dbc-765f-488b-9701-c4081448ac4a-istio-token\") pod \"e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-jz4tz\" (UID: \"a2038dbc-765f-488b-9701-c4081448ac4a\") " pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-jz4tz"
Oct 31 01:23:36.232021 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:23:36.231973    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a2038dbc-765f-488b-9701-c4081448ac4a-workload-certs\") pod \"e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-jz4tz\" (UID: \"a2038dbc-765f-488b-9701-c4081448ac4a\") " pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-jz4tz"
Oct 31 01:23:36.232021 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:23:36.232002    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2038dbc-765f-488b-9701-c4081448ac4a-credential-socket\") pod \"e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-jz4tz\" (UID: \"a2038dbc-765f-488b-9701-c4081448ac4a\") " pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-jz4tz"
Oct 31 01:23:36.232139 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:23:36.231979    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a2038dbc-765f-488b-9701-c4081448ac4a-istio-envoy\") pod \"e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-jz4tz\" (UID: \"a2038dbc-765f-488b-9701-c4081448ac4a\") " pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-jz4tz"
Oct 31 01:23:36.232238 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:23:36.232206    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a2038dbc-765f-488b-9701-c4081448ac4a-istio-data\") pod \"e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-jz4tz\" (UID: \"a2038dbc-765f-488b-9701-c4081448ac4a\") " pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-jz4tz"
Oct 31 01:23:36.232552 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:23:36.232527    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2038dbc-765f-488b-9701-c4081448ac4a-workload-socket\") pod \"e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-jz4tz\" (UID: \"a2038dbc-765f-488b-9701-c4081448ac4a\") " pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-jz4tz"
Oct 31 01:23:36.232665 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:23:36.232583    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a2038dbc-765f-488b-9701-c4081448ac4a-istio-data\") pod \"e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-jz4tz\" (UID: \"a2038dbc-765f-488b-9701-c4081448ac4a\") " pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-jz4tz"
Oct 31 01:23:36.232832 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:23:36.232807    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a2038dbc-765f-488b-9701-c4081448ac4a-istiod-ca-cert\") pod \"e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-jz4tz\" (UID: \"a2038dbc-765f-488b-9701-c4081448ac4a\") " pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-jz4tz"
Oct 31 01:23:36.234852 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:23:36.234827    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a2038dbc-765f-488b-9701-c4081448ac4a-istio-podinfo\") pod \"e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-jz4tz\" (UID: \"a2038dbc-765f-488b-9701-c4081448ac4a\") " pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-jz4tz"
Oct 31 01:23:36.235014 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:23:36.234992    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a2038dbc-765f-488b-9701-c4081448ac4a-istio-envoy\") pod \"e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-jz4tz\" (UID: \"a2038dbc-765f-488b-9701-c4081448ac4a\") " pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-jz4tz"
Oct 31 01:23:36.288138 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:23:36.288111    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a2038dbc-765f-488b-9701-c4081448ac4a-istio-token\") pod \"e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-jz4tz\" (UID: \"a2038dbc-765f-488b-9701-c4081448ac4a\") " pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-jz4tz"
Oct 31 01:23:36.288285 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:23:36.288173    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c7jcs\" (UniqueName: \"kubernetes.io/projected/a2038dbc-765f-488b-9701-c4081448ac4a-kube-api-access-c7jcs\") pod \"e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-jz4tz\" (UID: \"a2038dbc-765f-488b-9701-c4081448ac4a\") " pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-jz4tz"
Oct 31 01:23:36.364432 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:23:36.364357    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-jz4tz"
Oct 31 01:23:36.818024 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:23:36.817980    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-jz4tz"]
Oct 31 01:23:36.855884 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:23:36.855858    4964 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2038dbc_765f_488b_9701_c4081448ac4a.slice/crio-aa800ca2997518d7b698bafe7c650562eddc3423f27183cfadae060415313546 WatchSource:0}: Error finding container aa800ca2997518d7b698bafe7c650562eddc3423f27183cfadae060415313546: Status 404 returned error can't find the container with id aa800ca2997518d7b698bafe7c650562eddc3423f27183cfadae060415313546
Oct 31 01:23:36.858265 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:23:36.858236    4964 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 01:23:36.858353 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:23:36.858297    4964 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 01:23:36.858353 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:23:36.858345    4964 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 01:23:37.128390 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:23:37.128280    4964 patch_prober.go:29] interesting pod/e2e-worker-w1-baacee-openshift-default-57f845d5bf-qvt9t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body=
Oct 31 01:23:37.128390 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:23:37.128348    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-qvt9t" podUID="247208c2-428e-4f58-8dff-55939802b345" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused"
Oct 31 01:23:37.253340 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:23:37.253185    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-jz4tz" event={"ID":"a2038dbc-765f-488b-9701-c4081448ac4a","Type":"ContainerStarted","Data":"43a463f81497888dd942f1d16df68e152238a91b7f29fabdf658061d6481f1bc"}
Oct 31 01:23:37.253340 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:23:37.253260    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-jz4tz" event={"ID":"a2038dbc-765f-488b-9701-c4081448ac4a","Type":"ContainerStarted","Data":"aa800ca2997518d7b698bafe7c650562eddc3423f27183cfadae060415313546"}
Oct 31 01:23:37.364805 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:23:37.364771    4964 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-jz4tz"
Oct 31 01:23:37.366478 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:23:37.366442    4964 patch_prober.go:29] interesting pod/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-jz4tz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body=
Oct 31 01:23:37.366621 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:23:37.366498    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-jz4tz" podUID="a2038dbc-765f-488b-9701-c4081448ac4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused"
Oct 31 01:23:37.373998 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:23:37.373949    4964 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-jz4tz" podStartSLOduration=1.373935957 podStartE2EDuration="1.373935957s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:23:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:23:37.370387705 +0000 UTC m=+1185.846954084" watchObservedRunningTime="2026-10-31 01:23:37.373935957 +0000 UTC m=+1185.850502337"
Oct 31 01:23:38.131899 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:23:38.131863    4964 patch_prober.go:29] interesting pod/e2e-worker-w1-baacee-openshift-default-57f845d5bf-qvt9t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body=
Oct 31 01:23:38.132569 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:23:38.132522    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-qvt9t" podUID="247208c2-428e-4f58-8dff-55939802b345" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused"
Oct 31 01:23:38.365101 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:23:38.365067    4964 patch_prober.go:29] interesting pod/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-jz4tz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body=
Oct 31 01:23:38.365268 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:23:38.365119    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-jz4tz" podUID="a2038dbc-765f-488b-9701-c4081448ac4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused"
Oct 31 01:23:38.674338 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:23:38.673814    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:23:39.129761 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:23:39.129724    4964 patch_prober.go:29] interesting pod/e2e-worker-w1-baacee-openshift-default-57f845d5bf-qvt9t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body=
Oct 31 01:23:39.130092 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:23:39.130060    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-qvt9t" podUID="247208c2-428e-4f58-8dff-55939802b345" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused"
Oct 31 01:23:39.364748 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:23:39.364712    4964 patch_prober.go:29] interesting pod/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-jz4tz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body=
Oct 31 01:23:39.365162 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:23:39.364766    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-jz4tz" podUID="a2038dbc-765f-488b-9701-c4081448ac4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused"
Oct 31 01:23:40.131733 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:23:40.131699    4964 patch_prober.go:29] interesting pod/e2e-worker-w1-baacee-openshift-default-57f845d5bf-qvt9t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body=
Oct 31 01:23:40.132038 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:23:40.132007    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-qvt9t" podUID="247208c2-428e-4f58-8dff-55939802b345" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused"
Oct 31 01:23:40.366145 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:23:40.366116    4964 patch_prober.go:29] interesting pod/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-jz4tz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body=
Oct 31 01:23:40.366609 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:23:40.366164    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-jz4tz" podUID="a2038dbc-765f-488b-9701-c4081448ac4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused"
Oct 31 01:23:41.127806 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:23:41.127766    4964 patch_prober.go:29] interesting pod/e2e-worker-w1-baacee-openshift-default-57f845d5bf-qvt9t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body=
Oct 31 01:23:41.127998 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:23:41.127835    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-qvt9t" podUID="247208c2-428e-4f58-8dff-55939802b345" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused"
Oct 31 01:23:41.254133 ip-10-0-133-197 kubenswrapper[4964]: E1031 01:23:41.254098    4964 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q56x5" containerName="extractor"
Oct 31 01:23:41.364958 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:23:41.364928    4964 patch_prober.go:29] interesting pod/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-jz4tz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body=
Oct 31 01:23:41.365171 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:23:41.364999    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-jz4tz" podUID="a2038dbc-765f-488b-9701-c4081448ac4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused"
Oct 31 01:23:42.133543 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:23:42.133499    4964 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-qvt9t"
Oct 31 01:23:42.133981 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:23:42.133832    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-qvt9t"
Oct 31 01:23:42.134973 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:23:42.134953    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-qvt9t"
Oct 31 01:23:42.365700 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:23:42.365669    4964 patch_prober.go:29] interesting pod/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-jz4tz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body=
Oct 31 01:23:42.365863 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:23:42.365715    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-jz4tz" podUID="a2038dbc-765f-488b-9701-c4081448ac4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused"
Oct 31 01:23:43.365057 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:23:43.365013    4964 patch_prober.go:29] interesting pod/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-jz4tz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body=
Oct 31 01:23:43.365563 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:23:43.365069    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-jz4tz" podUID="a2038dbc-765f-488b-9701-c4081448ac4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused"
Oct 31 01:23:43.423602 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:23:43.423567    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:23:44.371060 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:23:44.371028    4964 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-jz4tz"
Oct 31 01:23:44.372163 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:23:44.372145    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-jz4tz"
Oct 31 01:23:44.372918 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:23:44.372891    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-jz4tz"
Oct 31 01:23:45.958411 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:23:45.958377    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:23:48.494422 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:23:48.494388    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:23:51.914186 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:23:51.914156    4964 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-9257n_f0c0ada7-7688-4dc2-97fc-813e4d0c4ef3/console-operator/2.log"
Oct 31 01:23:51.917059 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:23:51.917033    4964 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-9257n_f0c0ada7-7688-4dc2-97fc-813e4d0c4ef3/console-operator/2.log"
Oct 31 01:23:53.305916 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:23:53.305880    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:23:53.538507 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:23:53.538466    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:23:53.715289 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:23:53.715198    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:23:59.353776 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:23:59.353732    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:24:01.194363 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:24:01.188862    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:24:03.323865 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:24:03.323824    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:24:06.871657 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:24:06.871618    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:24:07.098583 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:24:07.098550    4964 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-75984d866d-k5g7h"]
Oct 31 01:24:07.180369 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:24:07.180269    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:24:07.359496 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:24:07.359460    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:24:07.536971 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:24:07.536935    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:24:08.204695 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:24:08.204645    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:24:11.058500 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:24:11.058468    4964 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-w9jpq"]
Oct 31 01:24:11.062726 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:24:11.062707    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-w9jpq"
Oct 31 01:24:11.187599 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:24:11.187560    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/145637b0-60a1-4990-8dfe-dd073d9b3929-istio-token\") pod \"e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-w9jpq\" (UID: \"145637b0-60a1-4990-8dfe-dd073d9b3929\") " pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-w9jpq"
Oct 31 01:24:11.187893 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:24:11.187875    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/145637b0-60a1-4990-8dfe-dd073d9b3929-workload-certs\") pod \"e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-w9jpq\" (UID: \"145637b0-60a1-4990-8dfe-dd073d9b3929\") " pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-w9jpq"
Oct 31 01:24:11.188033 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:24:11.188016    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/145637b0-60a1-4990-8dfe-dd073d9b3929-workload-socket\") pod \"e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-w9jpq\" (UID: \"145637b0-60a1-4990-8dfe-dd073d9b3929\") " pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-w9jpq"
Oct 31 01:24:11.188173 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:24:11.188161    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/145637b0-60a1-4990-8dfe-dd073d9b3929-istio-envoy\") pod \"e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-w9jpq\" (UID: \"145637b0-60a1-4990-8dfe-dd073d9b3929\") " pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-w9jpq"
Oct 31 01:24:11.188280 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:24:11.188268    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/145637b0-60a1-4990-8dfe-dd073d9b3929-istiod-ca-cert\") pod \"e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-w9jpq\" (UID: \"145637b0-60a1-4990-8dfe-dd073d9b3929\") " pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-w9jpq"
Oct 31 01:24:11.188457 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:24:11.188440    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/145637b0-60a1-4990-8dfe-dd073d9b3929-credential-socket\") pod \"e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-w9jpq\" (UID: \"145637b0-60a1-4990-8dfe-dd073d9b3929\") " pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-w9jpq"
Oct 31 01:24:11.188567 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:24:11.188554    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/145637b0-60a1-4990-8dfe-dd073d9b3929-istio-data\") pod \"e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-w9jpq\" (UID: \"145637b0-60a1-4990-8dfe-dd073d9b3929\") " pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-w9jpq"
Oct 31 01:24:11.188683 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:24:11.188645    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/145637b0-60a1-4990-8dfe-dd073d9b3929-istio-podinfo\") pod \"e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-w9jpq\" (UID: \"145637b0-60a1-4990-8dfe-dd073d9b3929\") " pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-w9jpq"
Oct 31 01:24:11.188818 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:24:11.188804    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hkf2b\" (UniqueName: \"kubernetes.io/projected/145637b0-60a1-4990-8dfe-dd073d9b3929-kube-api-access-hkf2b\") pod \"e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-w9jpq\" (UID: \"145637b0-60a1-4990-8dfe-dd073d9b3929\") " pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-w9jpq"
Oct 31 01:24:11.193861 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:24:11.193828    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-w9jpq"]
Oct 31 01:24:11.250235 ip-10-0-133-197 kubenswrapper[4964]: E1031 01:24:11.250186    4964 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q56x5" containerName="extractor"
Oct 31 01:24:11.290250 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:24:11.290218    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/145637b0-60a1-4990-8dfe-dd073d9b3929-istio-token\") pod \"e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-w9jpq\" (UID: \"145637b0-60a1-4990-8dfe-dd073d9b3929\") " pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-w9jpq"
Oct 31 01:24:11.290488 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:24:11.290269    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/145637b0-60a1-4990-8dfe-dd073d9b3929-workload-certs\") pod \"e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-w9jpq\" (UID: \"145637b0-60a1-4990-8dfe-dd073d9b3929\") " pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-w9jpq"
Oct 31 01:24:11.290488 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:24:11.290298    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/145637b0-60a1-4990-8dfe-dd073d9b3929-workload-socket\") pod \"e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-w9jpq\" (UID: \"145637b0-60a1-4990-8dfe-dd073d9b3929\") " pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-w9jpq"
Oct 31 01:24:11.290488 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:24:11.290469    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/145637b0-60a1-4990-8dfe-dd073d9b3929-istio-envoy\") pod \"e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-w9jpq\" (UID: \"145637b0-60a1-4990-8dfe-dd073d9b3929\") " pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-w9jpq"
Oct 31 01:24:11.290657 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:24:11.290530    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/145637b0-60a1-4990-8dfe-dd073d9b3929-istiod-ca-cert\") pod \"e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-w9jpq\" (UID: \"145637b0-60a1-4990-8dfe-dd073d9b3929\") " pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-w9jpq"
Oct 31 01:24:11.290657 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:24:11.290584    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/145637b0-60a1-4990-8dfe-dd073d9b3929-credential-socket\") pod \"e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-w9jpq\" (UID: \"145637b0-60a1-4990-8dfe-dd073d9b3929\") " pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-w9jpq"
Oct 31 01:24:11.290657 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:24:11.290613    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/145637b0-60a1-4990-8dfe-dd073d9b3929-istio-data\") pod \"e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-w9jpq\" (UID: \"145637b0-60a1-4990-8dfe-dd073d9b3929\") " pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-w9jpq"
Oct 31 01:24:11.290657 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:24:11.290638    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/145637b0-60a1-4990-8dfe-dd073d9b3929-istio-podinfo\") pod \"e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-w9jpq\" (UID: \"145637b0-60a1-4990-8dfe-dd073d9b3929\") " pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-w9jpq"
Oct 31 01:24:11.290851 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:24:11.290681    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hkf2b\" (UniqueName: \"kubernetes.io/projected/145637b0-60a1-4990-8dfe-dd073d9b3929-kube-api-access-hkf2b\") pod \"e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-w9jpq\" (UID: \"145637b0-60a1-4990-8dfe-dd073d9b3929\") " pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-w9jpq"
Oct 31 01:24:11.290851 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:24:11.290728    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/145637b0-60a1-4990-8dfe-dd073d9b3929-workload-socket\") pod \"e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-w9jpq\" (UID: \"145637b0-60a1-4990-8dfe-dd073d9b3929\") " pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-w9jpq"
Oct 31 01:24:11.291082 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:24:11.291057    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/145637b0-60a1-4990-8dfe-dd073d9b3929-workload-certs\") pod \"e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-w9jpq\" (UID: \"145637b0-60a1-4990-8dfe-dd073d9b3929\") " pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-w9jpq"
Oct 31 01:24:11.291448 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:24:11.291424    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/145637b0-60a1-4990-8dfe-dd073d9b3929-istio-data\") pod \"e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-w9jpq\" (UID: \"145637b0-60a1-4990-8dfe-dd073d9b3929\") " pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-w9jpq"
Oct 31 01:24:11.291617 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:24:11.291472    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/145637b0-60a1-4990-8dfe-dd073d9b3929-credential-socket\") pod \"e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-w9jpq\" (UID: \"145637b0-60a1-4990-8dfe-dd073d9b3929\") " pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-w9jpq"
Oct 31 01:24:11.291756 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:24:11.291675    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/145637b0-60a1-4990-8dfe-dd073d9b3929-istiod-ca-cert\") pod \"e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-w9jpq\" (UID: \"145637b0-60a1-4990-8dfe-dd073d9b3929\") " pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-w9jpq"
Oct 31 01:24:11.293011 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:24:11.292980    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/145637b0-60a1-4990-8dfe-dd073d9b3929-istio-envoy\") pod \"e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-w9jpq\" (UID: \"145637b0-60a1-4990-8dfe-dd073d9b3929\") " pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-w9jpq"
Oct 31 01:24:11.293533 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:24:11.293513    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/145637b0-60a1-4990-8dfe-dd073d9b3929-istio-podinfo\") pod \"e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-w9jpq\" (UID: \"145637b0-60a1-4990-8dfe-dd073d9b3929\") " pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-w9jpq"
Oct 31 01:24:11.346211 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:24:11.346132    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/145637b0-60a1-4990-8dfe-dd073d9b3929-istio-token\") pod \"e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-w9jpq\" (UID: \"145637b0-60a1-4990-8dfe-dd073d9b3929\") " pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-w9jpq"
Oct 31 01:24:11.346389 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:24:11.346361    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hkf2b\" (UniqueName: \"kubernetes.io/projected/145637b0-60a1-4990-8dfe-dd073d9b3929-kube-api-access-hkf2b\") pod \"e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-w9jpq\" (UID: \"145637b0-60a1-4990-8dfe-dd073d9b3929\") " pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-w9jpq"
Oct 31 01:24:11.376303 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:24:11.376274    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-w9jpq"
Oct 31 01:24:11.810792 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:24:11.810772    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-w9jpq"]
Oct 31 01:24:11.812370 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:24:11.812343    4964 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod145637b0_60a1_4990_8dfe_dd073d9b3929.slice/crio-a17c59f02a0fc62bd8d613147f8ee112f6210d0c2bf8ce4f1e075b11964fedae WatchSource:0}: Error finding container a17c59f02a0fc62bd8d613147f8ee112f6210d0c2bf8ce4f1e075b11964fedae: Status 404 returned error can't find the container with id a17c59f02a0fc62bd8d613147f8ee112f6210d0c2bf8ce4f1e075b11964fedae
Oct 31 01:24:11.814550 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:24:11.814520    4964 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 01:24:11.814643 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:24:11.814580    4964 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 01:24:11.814643 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:24:11.814609    4964 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 01:24:12.099463 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:24:12.099347    4964 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-75984d866d-k5g7h" podUID="97691d17-0ce5-4ca4-885a-c092a0317ebb" containerName="payload-processing" containerID="cri-o://fd694fb4c4ed8385be4f1b2c91ae33c244af4fe12eff06bc99bc64e0889361bc" gracePeriod=25
Oct 31 01:24:12.431727 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:24:12.431697    4964 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-75984d866d-k5g7h"
Oct 31 01:24:12.521892 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:24:12.521848    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/97691d17-0ce5-4ca4-885a-c092a0317ebb-kube-api-access-czj6h\" (UniqueName: \"kubernetes.io/projected/97691d17-0ce5-4ca4-885a-c092a0317ebb-kube-api-access-czj6h\") pod \"97691d17-0ce5-4ca4-885a-c092a0317ebb\" (UID: \"97691d17-0ce5-4ca4-885a-c092a0317ebb\") "
Oct 31 01:24:12.522112 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:24:12.521937    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/97691d17-0ce5-4ca4-885a-c092a0317ebb-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/97691d17-0ce5-4ca4-885a-c092a0317ebb-routing-overlay-volume\") pod \"97691d17-0ce5-4ca4-885a-c092a0317ebb\" (UID: \"97691d17-0ce5-4ca4-885a-c092a0317ebb\") "
Oct 31 01:24:12.522112 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:24:12.521974    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/97691d17-0ce5-4ca4-885a-c092a0317ebb-provider-credentials\" (UniqueName: \"kubernetes.io/projected/97691d17-0ce5-4ca4-885a-c092a0317ebb-provider-credentials\") pod \"97691d17-0ce5-4ca4-885a-c092a0317ebb\" (UID: \"97691d17-0ce5-4ca4-885a-c092a0317ebb\") "
Oct 31 01:24:12.522112 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:24:12.522098    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/97691d17-0ce5-4ca4-885a-c092a0317ebb-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/97691d17-0ce5-4ca4-885a-c092a0317ebb-plugins-config-volume\") pod \"97691d17-0ce5-4ca4-885a-c092a0317ebb\" (UID: \"97691d17-0ce5-4ca4-885a-c092a0317ebb\") "
Oct 31 01:24:12.522757 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:24:12.522722    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/97691d17-0ce5-4ca4-885a-c092a0317ebb-routing-overlay-volume" pod "97691d17-0ce5-4ca4-885a-c092a0317ebb" (UID: "97691d17-0ce5-4ca4-885a-c092a0317ebb"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:24:12.522870 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:24:12.522818    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/97691d17-0ce5-4ca4-885a-c092a0317ebb-plugins-config-volume" pod "97691d17-0ce5-4ca4-885a-c092a0317ebb" (UID: "97691d17-0ce5-4ca4-885a-c092a0317ebb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:24:12.524867 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:24:12.524841    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/97691d17-0ce5-4ca4-885a-c092a0317ebb-kube-api-access-czj6h" pod "97691d17-0ce5-4ca4-885a-c092a0317ebb" (UID: "97691d17-0ce5-4ca4-885a-c092a0317ebb"). InnerVolumeSpecName "kube-api-access-czj6h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:24:12.525134 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:24:12.525113    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/97691d17-0ce5-4ca4-885a-c092a0317ebb-provider-credentials" pod "97691d17-0ce5-4ca4-885a-c092a0317ebb" (UID: "97691d17-0ce5-4ca4-885a-c092a0317ebb"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:24:12.627746 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:24:12.627643    4964 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-czj6h\" (UniqueName: \"kubernetes.io/projected/97691d17-0ce5-4ca4-885a-c092a0317ebb-kube-api-access-czj6h\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:24:12.627746 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:24:12.627686    4964 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/97691d17-0ce5-4ca4-885a-c092a0317ebb-routing-overlay-volume\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:24:12.627746 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:24:12.627702    4964 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/97691d17-0ce5-4ca4-885a-c092a0317ebb-provider-credentials\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:24:12.627746 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:24:12.627717    4964 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/97691d17-0ce5-4ca4-885a-c092a0317ebb-plugins-config-volume\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:24:12.767027 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:24:12.766992    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-w9jpq" event={"ID":"145637b0-60a1-4990-8dfe-dd073d9b3929","Type":"ContainerStarted","Data":"d1f57631c27c364c9c7350fa54c860a7cf74b49618a770c325a6ae1754a9c97d"}
Oct 31 01:24:12.767239 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:24:12.767056    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-w9jpq" event={"ID":"145637b0-60a1-4990-8dfe-dd073d9b3929","Type":"ContainerStarted","Data":"a17c59f02a0fc62bd8d613147f8ee112f6210d0c2bf8ce4f1e075b11964fedae"}
Oct 31 01:24:12.768374 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:24:12.768338    4964 generic.go:356] "Generic (PLEG): container finished" podID="97691d17-0ce5-4ca4-885a-c092a0317ebb" containerID="fd694fb4c4ed8385be4f1b2c91ae33c244af4fe12eff06bc99bc64e0889361bc" exitCode=0
Oct 31 01:24:12.768499 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:24:12.768376    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-75984d866d-k5g7h" event={"ID":"97691d17-0ce5-4ca4-885a-c092a0317ebb","Type":"ContainerDied","Data":"fd694fb4c4ed8385be4f1b2c91ae33c244af4fe12eff06bc99bc64e0889361bc"}
Oct 31 01:24:12.768499 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:24:12.768395    4964 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-75984d866d-k5g7h"
Oct 31 01:24:12.768499 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:24:12.768424    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-75984d866d-k5g7h" event={"ID":"97691d17-0ce5-4ca4-885a-c092a0317ebb","Type":"ContainerDied","Data":"1c1ebc0ddb738367ffc6a81d0165034cd95d828a472ccf86772b06fb4b39b53b"}
Oct 31 01:24:12.768499 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:24:12.768446    4964 scope.go:122] "RemoveContainer" containerID="fd694fb4c4ed8385be4f1b2c91ae33c244af4fe12eff06bc99bc64e0889361bc"
Oct 31 01:24:12.779254 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:24:12.779237    4964 scope.go:122] "RemoveContainer" containerID="fd694fb4c4ed8385be4f1b2c91ae33c244af4fe12eff06bc99bc64e0889361bc"
Oct 31 01:24:12.779556 ip-10-0-133-197 kubenswrapper[4964]: E1031 01:24:12.779538    4964 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fd694fb4c4ed8385be4f1b2c91ae33c244af4fe12eff06bc99bc64e0889361bc\": container with ID starting with fd694fb4c4ed8385be4f1b2c91ae33c244af4fe12eff06bc99bc64e0889361bc not found: ID does not exist" containerID="fd694fb4c4ed8385be4f1b2c91ae33c244af4fe12eff06bc99bc64e0889361bc"
Oct 31 01:24:12.779617 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:24:12.779565    4964 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fd694fb4c4ed8385be4f1b2c91ae33c244af4fe12eff06bc99bc64e0889361bc"} err="failed to get container status \"fd694fb4c4ed8385be4f1b2c91ae33c244af4fe12eff06bc99bc64e0889361bc\": rpc error: code = NotFound desc = could not find container \"fd694fb4c4ed8385be4f1b2c91ae33c244af4fe12eff06bc99bc64e0889361bc\": container with ID starting with fd694fb4c4ed8385be4f1b2c91ae33c244af4fe12eff06bc99bc64e0889361bc not found: ID does not exist"
Oct 31 01:24:12.888548 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:24:12.888449    4964 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-w9jpq" podStartSLOduration=1.888435031 podStartE2EDuration="1.888435031s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:24:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:24:12.886077302 +0000 UTC m=+1221.362643681" watchObservedRunningTime="2026-10-31 01:24:12.888435031 +0000 UTC m=+1221.365001464"
Oct 31 01:24:13.019878 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:24:13.019708    4964 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-75984d866d-k5g7h"]
Oct 31 01:24:13.023621 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:24:13.023577    4964 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-75984d866d-k5g7h"]
Oct 31 01:24:13.256158 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:24:13.255156    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:24:13.377295 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:24:13.377261    4964 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-w9jpq"
Oct 31 01:24:13.379253 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:24:13.379227    4964 patch_prober.go:29] interesting pod/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-w9jpq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body=
Oct 31 01:24:13.379413 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:24:13.379274    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-w9jpq" podUID="145637b0-60a1-4990-8dfe-dd073d9b3929" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused"
Oct 31 01:24:13.777460 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:24:13.777433    4964 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="97691d17-0ce5-4ca4-885a-c092a0317ebb" path="/var/lib/kubelet/pods/97691d17-0ce5-4ca4-885a-c092a0317ebb/volumes"
Oct 31 01:24:14.377008 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:24:14.376971    4964 patch_prober.go:29] interesting pod/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-w9jpq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body=
Oct 31 01:24:14.377540 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:24:14.377027    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-w9jpq" podUID="145637b0-60a1-4990-8dfe-dd073d9b3929" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused"
Oct 31 01:24:15.377244 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:24:15.377207    4964 patch_prober.go:29] interesting pod/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-w9jpq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body=
Oct 31 01:24:15.377677 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:24:15.377262    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-w9jpq" podUID="145637b0-60a1-4990-8dfe-dd073d9b3929" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused"
Oct 31 01:24:16.377247 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:24:16.377213    4964 patch_prober.go:29] interesting pod/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-w9jpq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body=
Oct 31 01:24:16.377710 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:24:16.377267    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-w9jpq" podUID="145637b0-60a1-4990-8dfe-dd073d9b3929" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused"
Oct 31 01:24:17.377680 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:24:17.377639    4964 patch_prober.go:29] interesting pod/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-w9jpq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body=
Oct 31 01:24:17.378175 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:24:17.377699    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-w9jpq" podUID="145637b0-60a1-4990-8dfe-dd073d9b3929" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused"
Oct 31 01:24:18.104536 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:24:18.104498    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:24:18.377261 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:24:18.377177    4964 patch_prober.go:29] interesting pod/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-w9jpq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body=
Oct 31 01:24:18.377261 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:24:18.377245    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-w9jpq" podUID="145637b0-60a1-4990-8dfe-dd073d9b3929" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused"
Oct 31 01:24:19.381168 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:24:19.381139    4964 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-w9jpq"
Oct 31 01:24:19.381627 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:24:19.381610    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-w9jpq"
Oct 31 01:24:19.382427 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:24:19.382406    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-w9jpq"
Oct 31 01:24:24.356151 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:24:24.356100    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:24:26.171099 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:24:26.171072    4964 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-wjg8p"]
Oct 31 01:24:26.171765 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:24:26.171746    4964 cpu_manager.go:414] "containerMap: removing container" podUID="97691d17-0ce5-4ca4-885a-c092a0317ebb" containerName="payload-processing"
Oct 31 01:24:26.171765 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:24:26.171766    4964 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="97691d17-0ce5-4ca4-885a-c092a0317ebb" containerName="payload-processing"
Oct 31 01:24:26.171898 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:24:26.171889    4964 memory_manager.go:365] "RemoveStaleState removing state" podUID="97691d17-0ce5-4ca4-885a-c092a0317ebb" containerName="payload-processing"
Oct 31 01:24:26.176775 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:24:26.176755    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-wjg8p"
Oct 31 01:24:26.204524 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:24:26.204500    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8bdf16a9-47d4-429a-bd0c-d031817ff07b-istio-envoy\") pod \"e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-wjg8p\" (UID: \"8bdf16a9-47d4-429a-bd0c-d031817ff07b\") " pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-wjg8p"
Oct 31 01:24:26.204637 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:24:26.204536    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8bdf16a9-47d4-429a-bd0c-d031817ff07b-workload-certs\") pod \"e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-wjg8p\" (UID: \"8bdf16a9-47d4-429a-bd0c-d031817ff07b\") " pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-wjg8p"
Oct 31 01:24:26.204686 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:24:26.204641    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8bdf16a9-47d4-429a-bd0c-d031817ff07b-istiod-ca-cert\") pod \"e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-wjg8p\" (UID: \"8bdf16a9-47d4-429a-bd0c-d031817ff07b\") " pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-wjg8p"
Oct 31 01:24:26.204736 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:24:26.204716    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8bdf16a9-47d4-429a-bd0c-d031817ff07b-credential-socket\") pod \"e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-wjg8p\" (UID: \"8bdf16a9-47d4-429a-bd0c-d031817ff07b\") " pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-wjg8p"
Oct 31 01:24:26.204779 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:24:26.204765    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8bdf16a9-47d4-429a-bd0c-d031817ff07b-workload-socket\") pod \"e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-wjg8p\" (UID: \"8bdf16a9-47d4-429a-bd0c-d031817ff07b\") " pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-wjg8p"
Oct 31 01:24:26.204818 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:24:26.204798    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8bdf16a9-47d4-429a-bd0c-d031817ff07b-istio-data\") pod \"e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-wjg8p\" (UID: \"8bdf16a9-47d4-429a-bd0c-d031817ff07b\") " pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-wjg8p"
Oct 31 01:24:26.204818 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:24:26.204815    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8n2l2\" (UniqueName: \"kubernetes.io/projected/8bdf16a9-47d4-429a-bd0c-d031817ff07b-kube-api-access-8n2l2\") pod \"e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-wjg8p\" (UID: \"8bdf16a9-47d4-429a-bd0c-d031817ff07b\") " pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-wjg8p"
Oct 31 01:24:26.204898 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:24:26.204840    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8bdf16a9-47d4-429a-bd0c-d031817ff07b-istio-token\") pod \"e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-wjg8p\" (UID: \"8bdf16a9-47d4-429a-bd0c-d031817ff07b\") " pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-wjg8p"
Oct 31 01:24:26.204898 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:24:26.204887    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8bdf16a9-47d4-429a-bd0c-d031817ff07b-istio-podinfo\") pod \"e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-wjg8p\" (UID: \"8bdf16a9-47d4-429a-bd0c-d031817ff07b\") " pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-wjg8p"
Oct 31 01:24:26.276958 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:24:26.276928    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-wjg8p"]
Oct 31 01:24:26.306161 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:24:26.306128    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8bdf16a9-47d4-429a-bd0c-d031817ff07b-credential-socket\") pod \"e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-wjg8p\" (UID: \"8bdf16a9-47d4-429a-bd0c-d031817ff07b\") " pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-wjg8p"
Oct 31 01:24:26.306362 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:24:26.306188    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8bdf16a9-47d4-429a-bd0c-d031817ff07b-workload-socket\") pod \"e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-wjg8p\" (UID: \"8bdf16a9-47d4-429a-bd0c-d031817ff07b\") " pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-wjg8p"
Oct 31 01:24:26.306362 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:24:26.306215    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8bdf16a9-47d4-429a-bd0c-d031817ff07b-istio-data\") pod \"e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-wjg8p\" (UID: \"8bdf16a9-47d4-429a-bd0c-d031817ff07b\") " pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-wjg8p"
Oct 31 01:24:26.306362 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:24:26.306231    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8n2l2\" (UniqueName: \"kubernetes.io/projected/8bdf16a9-47d4-429a-bd0c-d031817ff07b-kube-api-access-8n2l2\") pod \"e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-wjg8p\" (UID: \"8bdf16a9-47d4-429a-bd0c-d031817ff07b\") " pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-wjg8p"
Oct 31 01:24:26.306362 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:24:26.306286    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8bdf16a9-47d4-429a-bd0c-d031817ff07b-istio-token\") pod \"e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-wjg8p\" (UID: \"8bdf16a9-47d4-429a-bd0c-d031817ff07b\") " pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-wjg8p"
Oct 31 01:24:26.306362 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:24:26.306349    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8bdf16a9-47d4-429a-bd0c-d031817ff07b-istio-podinfo\") pod \"e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-wjg8p\" (UID: \"8bdf16a9-47d4-429a-bd0c-d031817ff07b\") " pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-wjg8p"
Oct 31 01:24:26.306649 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:24:26.306434    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8bdf16a9-47d4-429a-bd0c-d031817ff07b-istio-envoy\") pod \"e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-wjg8p\" (UID: \"8bdf16a9-47d4-429a-bd0c-d031817ff07b\") " pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-wjg8p"
Oct 31 01:24:26.306649 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:24:26.306481    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8bdf16a9-47d4-429a-bd0c-d031817ff07b-workload-certs\") pod \"e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-wjg8p\" (UID: \"8bdf16a9-47d4-429a-bd0c-d031817ff07b\") " pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-wjg8p"
Oct 31 01:24:26.306649 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:24:26.306567    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8bdf16a9-47d4-429a-bd0c-d031817ff07b-istiod-ca-cert\") pod \"e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-wjg8p\" (UID: \"8bdf16a9-47d4-429a-bd0c-d031817ff07b\") " pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-wjg8p"
Oct 31 01:24:26.308875 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:24:26.308720    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8bdf16a9-47d4-429a-bd0c-d031817ff07b-credential-socket\") pod \"e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-wjg8p\" (UID: \"8bdf16a9-47d4-429a-bd0c-d031817ff07b\") " pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-wjg8p"
Oct 31 01:24:26.309155 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:24:26.309131    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8bdf16a9-47d4-429a-bd0c-d031817ff07b-istio-data\") pod \"e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-wjg8p\" (UID: \"8bdf16a9-47d4-429a-bd0c-d031817ff07b\") " pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-wjg8p"
Oct 31 01:24:26.309364 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:24:26.309191    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8bdf16a9-47d4-429a-bd0c-d031817ff07b-istiod-ca-cert\") pod \"e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-wjg8p\" (UID: \"8bdf16a9-47d4-429a-bd0c-d031817ff07b\") " pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-wjg8p"
Oct 31 01:24:26.309645 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:24:26.309220    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8bdf16a9-47d4-429a-bd0c-d031817ff07b-workload-certs\") pod \"e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-wjg8p\" (UID: \"8bdf16a9-47d4-429a-bd0c-d031817ff07b\") " pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-wjg8p"
Oct 31 01:24:26.309645 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:24:26.309255    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8bdf16a9-47d4-429a-bd0c-d031817ff07b-workload-socket\") pod \"e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-wjg8p\" (UID: \"8bdf16a9-47d4-429a-bd0c-d031817ff07b\") " pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-wjg8p"
Oct 31 01:24:26.311264 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:24:26.311237    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8bdf16a9-47d4-429a-bd0c-d031817ff07b-istio-envoy\") pod \"e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-wjg8p\" (UID: \"8bdf16a9-47d4-429a-bd0c-d031817ff07b\") " pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-wjg8p"
Oct 31 01:24:26.311736 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:24:26.311713    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8bdf16a9-47d4-429a-bd0c-d031817ff07b-istio-podinfo\") pod \"e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-wjg8p\" (UID: \"8bdf16a9-47d4-429a-bd0c-d031817ff07b\") " pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-wjg8p"
Oct 31 01:24:26.363247 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:24:26.363223    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8n2l2\" (UniqueName: \"kubernetes.io/projected/8bdf16a9-47d4-429a-bd0c-d031817ff07b-kube-api-access-8n2l2\") pod \"e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-wjg8p\" (UID: \"8bdf16a9-47d4-429a-bd0c-d031817ff07b\") " pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-wjg8p"
Oct 31 01:24:26.365638 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:24:26.365609    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8bdf16a9-47d4-429a-bd0c-d031817ff07b-istio-token\") pod \"e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-wjg8p\" (UID: \"8bdf16a9-47d4-429a-bd0c-d031817ff07b\") " pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-wjg8p"
Oct 31 01:24:26.500626 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:24:26.500595    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-wjg8p"
Oct 31 01:24:26.932231 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:24:26.932204    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-wjg8p"]
Oct 31 01:24:26.933550 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:24:26.933527    4964 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8bdf16a9_47d4_429a_bd0c_d031817ff07b.slice/crio-80d6a9f4325aa9a4993a59595a0d6a55afd4d7d47bcc95d04a207fbd40f2e920 WatchSource:0}: Error finding container 80d6a9f4325aa9a4993a59595a0d6a55afd4d7d47bcc95d04a207fbd40f2e920: Status 404 returned error can't find the container with id 80d6a9f4325aa9a4993a59595a0d6a55afd4d7d47bcc95d04a207fbd40f2e920
Oct 31 01:24:26.935803 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:24:26.935774    4964 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 01:24:26.935896 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:24:26.935834    4964 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 01:24:26.935896 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:24:26.935867    4964 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 01:24:26.958991 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:24:26.958957    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-wjg8p" event={"ID":"8bdf16a9-47d4-429a-bd0c-d031817ff07b","Type":"ContainerStarted","Data":"80d6a9f4325aa9a4993a59595a0d6a55afd4d7d47bcc95d04a207fbd40f2e920"}
Oct 31 01:24:27.967358 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:24:27.967307    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-wjg8p" event={"ID":"8bdf16a9-47d4-429a-bd0c-d031817ff07b","Type":"ContainerStarted","Data":"c9bb7002cb60b09b72f1b4715f5375e8ff9de7e4da36c295f89be4a6edc5857a"}
Oct 31 01:24:28.134949 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:24:28.134898    4964 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-wjg8p" podStartSLOduration=2.134884533 podStartE2EDuration="2.134884533s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:24:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:24:28.132377343 +0000 UTC m=+1236.608943720" watchObservedRunningTime="2026-10-31 01:24:28.134884533 +0000 UTC m=+1236.611450913"
Oct 31 01:24:28.507343 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:24:28.506687    4964 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-wjg8p"
Oct 31 01:24:28.512342 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:24:28.510431    4964 patch_prober.go:29] interesting pod/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-wjg8p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" start-of-body=
Oct 31 01:24:28.512342 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:24:28.510501    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-wjg8p" podUID="8bdf16a9-47d4-429a-bd0c-d031817ff07b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused"
Oct 31 01:24:29.425646 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:24:29.425608    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:24:29.507764 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:24:29.507725    4964 patch_prober.go:29] interesting pod/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-wjg8p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" start-of-body=
Oct 31 01:24:29.508471 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:24:29.508029    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-wjg8p" podUID="8bdf16a9-47d4-429a-bd0c-d031817ff07b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused"
Oct 31 01:24:30.505792 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:24:30.505755    4964 patch_prober.go:29] interesting pod/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-wjg8p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" start-of-body=
Oct 31 01:24:30.506400 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:24:30.506366    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-wjg8p" podUID="8bdf16a9-47d4-429a-bd0c-d031817ff07b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused"
Oct 31 01:24:31.507342 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:24:31.506687    4964 patch_prober.go:29] interesting pod/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-wjg8p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" start-of-body=
Oct 31 01:24:31.507342 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:24:31.506750    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-wjg8p" podUID="8bdf16a9-47d4-429a-bd0c-d031817ff07b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused"
Oct 31 01:24:32.515342 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:24:32.514469    4964 patch_prober.go:29] interesting pod/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-wjg8p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" start-of-body=
Oct 31 01:24:32.515342 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:24:32.514559    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-wjg8p" podUID="8bdf16a9-47d4-429a-bd0c-d031817ff07b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused"
Oct 31 01:24:32.844671 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:24:32.844561    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:24:33.500994 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:24:33.500957    4964 patch_prober.go:29] interesting pod/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-wjg8p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" start-of-body=
Oct 31 01:24:33.501193 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:24:33.501018    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-wjg8p" podUID="8bdf16a9-47d4-429a-bd0c-d031817ff07b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused"
Oct 31 01:24:33.987932 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:24:33.987897    4964 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-46rnw"]
Oct 31 01:24:33.992787 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:24:33.992760    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-46rnw"
Oct 31 01:24:34.092735 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:24:34.092704    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-46rnw"]
Oct 31 01:24:34.121717 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:24:34.119649    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/30c81ef3-5c85-411a-ac4e-524dbd5e3d3d-istiod-ca-cert\") pod \"e2e-worker-w1-baacee-openshift-default-57f845d5bf-46rnw\" (UID: \"30c81ef3-5c85-411a-ac4e-524dbd5e3d3d\") " pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-46rnw"
Oct 31 01:24:34.121717 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:24:34.119719    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/30c81ef3-5c85-411a-ac4e-524dbd5e3d3d-workload-socket\") pod \"e2e-worker-w1-baacee-openshift-default-57f845d5bf-46rnw\" (UID: \"30c81ef3-5c85-411a-ac4e-524dbd5e3d3d\") " pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-46rnw"
Oct 31 01:24:34.121717 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:24:34.119746    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/30c81ef3-5c85-411a-ac4e-524dbd5e3d3d-istio-podinfo\") pod \"e2e-worker-w1-baacee-openshift-default-57f845d5bf-46rnw\" (UID: \"30c81ef3-5c85-411a-ac4e-524dbd5e3d3d\") " pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-46rnw"
Oct 31 01:24:34.121717 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:24:34.119843    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/30c81ef3-5c85-411a-ac4e-524dbd5e3d3d-istio-data\") pod \"e2e-worker-w1-baacee-openshift-default-57f845d5bf-46rnw\" (UID: \"30c81ef3-5c85-411a-ac4e-524dbd5e3d3d\") " pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-46rnw"
Oct 31 01:24:34.121717 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:24:34.119922    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/30c81ef3-5c85-411a-ac4e-524dbd5e3d3d-istio-envoy\") pod \"e2e-worker-w1-baacee-openshift-default-57f845d5bf-46rnw\" (UID: \"30c81ef3-5c85-411a-ac4e-524dbd5e3d3d\") " pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-46rnw"
Oct 31 01:24:34.121717 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:24:34.119980    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/30c81ef3-5c85-411a-ac4e-524dbd5e3d3d-credential-socket\") pod \"e2e-worker-w1-baacee-openshift-default-57f845d5bf-46rnw\" (UID: \"30c81ef3-5c85-411a-ac4e-524dbd5e3d3d\") " pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-46rnw"
Oct 31 01:24:34.121717 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:24:34.120013    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/30c81ef3-5c85-411a-ac4e-524dbd5e3d3d-workload-certs\") pod \"e2e-worker-w1-baacee-openshift-default-57f845d5bf-46rnw\" (UID: \"30c81ef3-5c85-411a-ac4e-524dbd5e3d3d\") " pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-46rnw"
Oct 31 01:24:34.121717 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:24:34.120066    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/30c81ef3-5c85-411a-ac4e-524dbd5e3d3d-istio-token\") pod \"e2e-worker-w1-baacee-openshift-default-57f845d5bf-46rnw\" (UID: \"30c81ef3-5c85-411a-ac4e-524dbd5e3d3d\") " pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-46rnw"
Oct 31 01:24:34.121717 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:24:34.120109    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s7xvt\" (UniqueName: \"kubernetes.io/projected/30c81ef3-5c85-411a-ac4e-524dbd5e3d3d-kube-api-access-s7xvt\") pod \"e2e-worker-w1-baacee-openshift-default-57f845d5bf-46rnw\" (UID: \"30c81ef3-5c85-411a-ac4e-524dbd5e3d3d\") " pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-46rnw"
Oct 31 01:24:34.221802 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:24:34.221774    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/30c81ef3-5c85-411a-ac4e-524dbd5e3d3d-istio-envoy\") pod \"e2e-worker-w1-baacee-openshift-default-57f845d5bf-46rnw\" (UID: \"30c81ef3-5c85-411a-ac4e-524dbd5e3d3d\") " pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-46rnw"
Oct 31 01:24:34.221992 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:24:34.221827    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/30c81ef3-5c85-411a-ac4e-524dbd5e3d3d-credential-socket\") pod \"e2e-worker-w1-baacee-openshift-default-57f845d5bf-46rnw\" (UID: \"30c81ef3-5c85-411a-ac4e-524dbd5e3d3d\") " pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-46rnw"
Oct 31 01:24:34.221992 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:24:34.221851    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/30c81ef3-5c85-411a-ac4e-524dbd5e3d3d-workload-certs\") pod \"e2e-worker-w1-baacee-openshift-default-57f845d5bf-46rnw\" (UID: \"30c81ef3-5c85-411a-ac4e-524dbd5e3d3d\") " pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-46rnw"
Oct 31 01:24:34.221992 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:24:34.221880    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/30c81ef3-5c85-411a-ac4e-524dbd5e3d3d-istio-token\") pod \"e2e-worker-w1-baacee-openshift-default-57f845d5bf-46rnw\" (UID: \"30c81ef3-5c85-411a-ac4e-524dbd5e3d3d\") " pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-46rnw"
Oct 31 01:24:34.221992 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:24:34.221907    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s7xvt\" (UniqueName: \"kubernetes.io/projected/30c81ef3-5c85-411a-ac4e-524dbd5e3d3d-kube-api-access-s7xvt\") pod \"e2e-worker-w1-baacee-openshift-default-57f845d5bf-46rnw\" (UID: \"30c81ef3-5c85-411a-ac4e-524dbd5e3d3d\") " pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-46rnw"
Oct 31 01:24:34.221992 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:24:34.221988    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/30c81ef3-5c85-411a-ac4e-524dbd5e3d3d-istiod-ca-cert\") pod \"e2e-worker-w1-baacee-openshift-default-57f845d5bf-46rnw\" (UID: \"30c81ef3-5c85-411a-ac4e-524dbd5e3d3d\") " pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-46rnw"
Oct 31 01:24:34.222400 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:24:34.222372    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/30c81ef3-5c85-411a-ac4e-524dbd5e3d3d-workload-certs\") pod \"e2e-worker-w1-baacee-openshift-default-57f845d5bf-46rnw\" (UID: \"30c81ef3-5c85-411a-ac4e-524dbd5e3d3d\") " pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-46rnw"
Oct 31 01:24:34.222548 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:24:34.222416    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/30c81ef3-5c85-411a-ac4e-524dbd5e3d3d-credential-socket\") pod \"e2e-worker-w1-baacee-openshift-default-57f845d5bf-46rnw\" (UID: \"30c81ef3-5c85-411a-ac4e-524dbd5e3d3d\") " pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-46rnw"
Oct 31 01:24:34.222628 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:24:34.222607    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/30c81ef3-5c85-411a-ac4e-524dbd5e3d3d-workload-socket\") pod \"e2e-worker-w1-baacee-openshift-default-57f845d5bf-46rnw\" (UID: \"30c81ef3-5c85-411a-ac4e-524dbd5e3d3d\") " pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-46rnw"
Oct 31 01:24:34.222666 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:24:34.222643    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/30c81ef3-5c85-411a-ac4e-524dbd5e3d3d-istio-podinfo\") pod \"e2e-worker-w1-baacee-openshift-default-57f845d5bf-46rnw\" (UID: \"30c81ef3-5c85-411a-ac4e-524dbd5e3d3d\") " pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-46rnw"
Oct 31 01:24:34.222734 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:24:34.222721    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/30c81ef3-5c85-411a-ac4e-524dbd5e3d3d-istio-data\") pod \"e2e-worker-w1-baacee-openshift-default-57f845d5bf-46rnw\" (UID: \"30c81ef3-5c85-411a-ac4e-524dbd5e3d3d\") " pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-46rnw"
Oct 31 01:24:34.223043 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:24:34.222965    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/30c81ef3-5c85-411a-ac4e-524dbd5e3d3d-workload-socket\") pod \"e2e-worker-w1-baacee-openshift-default-57f845d5bf-46rnw\" (UID: \"30c81ef3-5c85-411a-ac4e-524dbd5e3d3d\") " pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-46rnw"
Oct 31 01:24:34.223114 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:24:34.223072    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/30c81ef3-5c85-411a-ac4e-524dbd5e3d3d-istio-data\") pod \"e2e-worker-w1-baacee-openshift-default-57f845d5bf-46rnw\" (UID: \"30c81ef3-5c85-411a-ac4e-524dbd5e3d3d\") " pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-46rnw"
Oct 31 01:24:34.223485 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:24:34.223469    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/30c81ef3-5c85-411a-ac4e-524dbd5e3d3d-istiod-ca-cert\") pod \"e2e-worker-w1-baacee-openshift-default-57f845d5bf-46rnw\" (UID: \"30c81ef3-5c85-411a-ac4e-524dbd5e3d3d\") " pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-46rnw"
Oct 31 01:24:34.224586 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:24:34.224569    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/30c81ef3-5c85-411a-ac4e-524dbd5e3d3d-istio-envoy\") pod \"e2e-worker-w1-baacee-openshift-default-57f845d5bf-46rnw\" (UID: \"30c81ef3-5c85-411a-ac4e-524dbd5e3d3d\") " pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-46rnw"
Oct 31 01:24:34.225029 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:24:34.225010    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/30c81ef3-5c85-411a-ac4e-524dbd5e3d3d-istio-podinfo\") pod \"e2e-worker-w1-baacee-openshift-default-57f845d5bf-46rnw\" (UID: \"30c81ef3-5c85-411a-ac4e-524dbd5e3d3d\") " pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-46rnw"
Oct 31 01:24:34.279213 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:24:34.279161    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/30c81ef3-5c85-411a-ac4e-524dbd5e3d3d-istio-token\") pod \"e2e-worker-w1-baacee-openshift-default-57f845d5bf-46rnw\" (UID: \"30c81ef3-5c85-411a-ac4e-524dbd5e3d3d\") " pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-46rnw"
Oct 31 01:24:34.280401 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:24:34.280383    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s7xvt\" (UniqueName: \"kubernetes.io/projected/30c81ef3-5c85-411a-ac4e-524dbd5e3d3d-kube-api-access-s7xvt\") pod \"e2e-worker-w1-baacee-openshift-default-57f845d5bf-46rnw\" (UID: \"30c81ef3-5c85-411a-ac4e-524dbd5e3d3d\") " pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-46rnw"
Oct 31 01:24:34.311233 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:24:34.311210    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-46rnw"
Oct 31 01:24:34.343418 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:24:34.343380    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:24:34.504908 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:24:34.504881    4964 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-wjg8p"
Oct 31 01:24:34.505219 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:24:34.505187    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-wjg8p"
Oct 31 01:24:34.506156 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:24:34.506134    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-wjg8p"
Oct 31 01:24:34.886333 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:24:34.886284    4964 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30c81ef3_5c85_411a_ac4e_524dbd5e3d3d.slice/crio-3a2522da97df75337b3c7a8587d855bcc128e1d9acb53ac725a04c141531694b WatchSource:0}: Error finding container 3a2522da97df75337b3c7a8587d855bcc128e1d9acb53ac725a04c141531694b: Status 404 returned error can't find the container with id 3a2522da97df75337b3c7a8587d855bcc128e1d9acb53ac725a04c141531694b
Oct 31 01:24:34.895713 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:24:34.894557    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-46rnw"]
Oct 31 01:24:34.904365 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:24:34.902189    4964 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 01:24:34.904365 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:24:34.902256    4964 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 01:24:34.904365 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:24:34.902286    4964 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 01:24:35.082349 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:24:35.072168    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-46rnw" event={"ID":"30c81ef3-5c85-411a-ac4e-524dbd5e3d3d","Type":"ContainerStarted","Data":"187f6077ebc1061103fc3aedc2a74a0807d6f6780d5f7a473b50836b977d3c61"}
Oct 31 01:24:35.082914 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:24:35.082885    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-46rnw" event={"ID":"30c81ef3-5c85-411a-ac4e-524dbd5e3d3d","Type":"ContainerStarted","Data":"3a2522da97df75337b3c7a8587d855bcc128e1d9acb53ac725a04c141531694b"}
Oct 31 01:24:35.201400 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:24:35.198780    4964 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-46rnw" podStartSLOduration=2.198762408 podStartE2EDuration="2.198762408s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:24:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:24:35.193806648 +0000 UTC m=+1243.670373029" watchObservedRunningTime="2026-10-31 01:24:35.198762408 +0000 UTC m=+1243.675328790"
Oct 31 01:24:35.311944 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:24:35.311763    4964 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-46rnw"
Oct 31 01:24:35.313159 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:24:35.313137    4964 patch_prober.go:29] interesting pod/e2e-worker-w1-baacee-openshift-default-57f845d5bf-46rnw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" start-of-body=
Oct 31 01:24:35.313491 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:24:35.313452    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-46rnw" podUID="30c81ef3-5c85-411a-ac4e-524dbd5e3d3d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused"
Oct 31 01:24:36.321343 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:24:36.321007    4964 patch_prober.go:29] interesting pod/e2e-worker-w1-baacee-openshift-default-57f845d5bf-46rnw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" start-of-body=
Oct 31 01:24:36.321343 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:24:36.321064    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-46rnw" podUID="30c81ef3-5c85-411a-ac4e-524dbd5e3d3d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused"
Oct 31 01:24:36.715509 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:24:36.715420    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:24:37.318344 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:24:37.316594    4964 patch_prober.go:29] interesting pod/e2e-worker-w1-baacee-openshift-default-57f845d5bf-46rnw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" start-of-body=
Oct 31 01:24:37.318344 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:24:37.316650    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-46rnw" podUID="30c81ef3-5c85-411a-ac4e-524dbd5e3d3d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused"
Oct 31 01:24:38.172650 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:24:38.172614    4964 generic.go:356] "Generic (PLEG): container finished" podID="d4ef3da1-2732-4051-be2a-5aba6f5fbcfc" containerID="abf4433a52ea15fccc9256feb0387c00be247fe08023a7a5928d074f2b5131ac" exitCode=0
Oct 31 01:24:38.173048 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:24:38.172650    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-njx9c-pt7d5" event={"ID":"d4ef3da1-2732-4051-be2a-5aba6f5fbcfc","Type":"ContainerDied","Data":"abf4433a52ea15fccc9256feb0387c00be247fe08023a7a5928d074f2b5131ac"}
Oct 31 01:24:38.312366 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:24:38.311728    4964 patch_prober.go:29] interesting pod/e2e-worker-w1-baacee-openshift-default-57f845d5bf-46rnw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" start-of-body=
Oct 31 01:24:38.312366 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:24:38.311784    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-46rnw" podUID="30c81ef3-5c85-411a-ac4e-524dbd5e3d3d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused"
Oct 31 01:24:39.323340 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:24:39.322733    4964 patch_prober.go:29] interesting pod/e2e-worker-w1-baacee-openshift-default-57f845d5bf-46rnw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" start-of-body=
Oct 31 01:24:39.323340 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:24:39.322809    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-46rnw" podUID="30c81ef3-5c85-411a-ac4e-524dbd5e3d3d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused"
Oct 31 01:24:39.430606 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:24:39.430569    4964 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-njx9c-pt7d5"
Oct 31 01:24:39.520731 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:24:39.520684    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:24:39.567416 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:24:39.567377    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d4ef3da1-2732-4051-be2a-5aba6f5fbcfc-kube-api-access-n6r8h\" (UniqueName: \"kubernetes.io/projected/d4ef3da1-2732-4051-be2a-5aba6f5fbcfc-kube-api-access-n6r8h\") pod \"d4ef3da1-2732-4051-be2a-5aba6f5fbcfc\" (UID: \"d4ef3da1-2732-4051-be2a-5aba6f5fbcfc\") "
Oct 31 01:24:39.567622 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:24:39.567545    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d4ef3da1-2732-4051-be2a-5aba6f5fbcfc-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d4ef3da1-2732-4051-be2a-5aba6f5fbcfc-trusted-ca-bundle\") pod \"d4ef3da1-2732-4051-be2a-5aba6f5fbcfc\" (UID: \"d4ef3da1-2732-4051-be2a-5aba6f5fbcfc\") "
Oct 31 01:24:39.567697 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:24:39.567653    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d4ef3da1-2732-4051-be2a-5aba6f5fbcfc-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d4ef3da1-2732-4051-be2a-5aba6f5fbcfc-service-ca-bundle\") pod \"d4ef3da1-2732-4051-be2a-5aba6f5fbcfc\" (UID: \"d4ef3da1-2732-4051-be2a-5aba6f5fbcfc\") "
Oct 31 01:24:39.567891 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:24:39.567795    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d4ef3da1-2732-4051-be2a-5aba6f5fbcfc-archives-path\" (UniqueName: \"kubernetes.io/empty-dir/d4ef3da1-2732-4051-be2a-5aba6f5fbcfc-archives-path\") pod \"d4ef3da1-2732-4051-be2a-5aba6f5fbcfc\" (UID: \"d4ef3da1-2732-4051-be2a-5aba6f5fbcfc\") "
Oct 31 01:24:39.568820 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:24:39.568750    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4ef3da1-2732-4051-be2a-5aba6f5fbcfc-archives-path" pod "d4ef3da1-2732-4051-be2a-5aba6f5fbcfc" (UID: "d4ef3da1-2732-4051-be2a-5aba6f5fbcfc"). InnerVolumeSpecName "archives-path". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:24:39.568820 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:24:39.568766    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d4ef3da1-2732-4051-be2a-5aba6f5fbcfc-trusted-ca-bundle" pod "d4ef3da1-2732-4051-be2a-5aba6f5fbcfc" (UID: "d4ef3da1-2732-4051-be2a-5aba6f5fbcfc"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:24:39.569193 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:24:39.569151    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d4ef3da1-2732-4051-be2a-5aba6f5fbcfc-service-ca-bundle" pod "d4ef3da1-2732-4051-be2a-5aba6f5fbcfc" (UID: "d4ef3da1-2732-4051-be2a-5aba6f5fbcfc"). InnerVolumeSpecName "service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:24:39.572456 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:24:39.572434    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d4ef3da1-2732-4051-be2a-5aba6f5fbcfc-kube-api-access-n6r8h" pod "d4ef3da1-2732-4051-be2a-5aba6f5fbcfc" (UID: "d4ef3da1-2732-4051-be2a-5aba6f5fbcfc"). InnerVolumeSpecName "kube-api-access-n6r8h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:24:39.670084 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:24:39.670051    4964 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d4ef3da1-2732-4051-be2a-5aba6f5fbcfc-trusted-ca-bundle\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:24:39.670084 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:24:39.670083    4964 reconciler_common.go:299] "Volume detached for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d4ef3da1-2732-4051-be2a-5aba6f5fbcfc-service-ca-bundle\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:24:39.670293 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:24:39.670093    4964 reconciler_common.go:299] "Volume detached for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/d4ef3da1-2732-4051-be2a-5aba6f5fbcfc-archives-path\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:24:39.670293 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:24:39.670101    4964 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n6r8h\" (UniqueName: \"kubernetes.io/projected/d4ef3da1-2732-4051-be2a-5aba6f5fbcfc-kube-api-access-n6r8h\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:24:40.218020 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:24:40.217915    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-njx9c-pt7d5" event={"ID":"d4ef3da1-2732-4051-be2a-5aba6f5fbcfc","Type":"ContainerDied","Data":"dd831a46aff74e979cc8ec90cbda0d740c48f05eac75c57b00ed5cfef2daed9e"}
Oct 31 01:24:40.218286 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:24:40.218258    4964 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dd831a46aff74e979cc8ec90cbda0d740c48f05eac75c57b00ed5cfef2daed9e"
Oct 31 01:24:40.218576 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:24:40.217971    4964 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-njx9c-pt7d5"
Oct 31 01:24:40.318419 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:24:40.314806    4964 patch_prober.go:29] interesting pod/e2e-worker-w1-baacee-openshift-default-57f845d5bf-46rnw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" start-of-body=
Oct 31 01:24:40.318419 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:24:40.314875    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-46rnw" podUID="30c81ef3-5c85-411a-ac4e-524dbd5e3d3d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused"
Oct 31 01:24:41.212338 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:24:41.209246    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:24:41.250163 ip-10-0-133-197 kubenswrapper[4964]: E1031 01:24:41.250133    4964 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q56x5" containerName="extractor"
Oct 31 01:24:41.313341 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:24:41.312944    4964 patch_prober.go:29] interesting pod/e2e-worker-w1-baacee-openshift-default-57f845d5bf-46rnw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" start-of-body=
Oct 31 01:24:41.313341 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:24:41.313002    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-46rnw" podUID="30c81ef3-5c85-411a-ac4e-524dbd5e3d3d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused"
Oct 31 01:24:41.383651 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:24:41.383615    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:24:42.688232 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:24:42.688190    4964 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-46rnw"
Oct 31 01:24:42.688736 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:24:42.688606    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-46rnw"
Oct 31 01:24:42.689708 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:24:42.689683    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-46rnw"
Oct 31 01:24:44.322673 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:24:44.322636    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:24:46.296943 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:24:46.296905    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:24:46.533209 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:24:46.533158    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:24:47.749364 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:24:47.748485    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:24:48.541130 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:24:48.541097    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:24:49.643342 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:24:49.643011    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:24:49.981076 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:24:49.981025    4964 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-00e840/premium-w2-00e840-kserve-6f6d876787-cv7w8" podUID="b8c12d5f-8ada-4c53-880c-446f5f0b5998" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:24:50.115342 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:24:50.114445    4964 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a8e157/distinct-w3-a8e157-kserve-549559dd5c-t29tw" podUID="4b0fd818-20c2-4dd6-9846-f361c236dbd9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": context deadline exceeded"
Oct 31 01:24:50.179053 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:24:50.178957    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:24:50.847621 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:24:50.847578    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:24:51.642200 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:24:51.639637    4964 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-hppbk"]
Oct 31 01:24:51.642200 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:24:51.641095    4964 cpu_manager.go:414] "containerMap: removing container" podUID="d4ef3da1-2732-4051-be2a-5aba6f5fbcfc" containerName="insights-gathering"
Oct 31 01:24:51.642200 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:24:51.641115    4964 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d4ef3da1-2732-4051-be2a-5aba6f5fbcfc" containerName="insights-gathering"
Oct 31 01:24:51.642200 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:24:51.641504    4964 memory_manager.go:365] "RemoveStaleState removing state" podUID="d4ef3da1-2732-4051-be2a-5aba6f5fbcfc" containerName="insights-gathering"
Oct 31 01:24:51.654035 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:24:51.654004    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-hppbk"
Oct 31 01:24:51.688468 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:24:51.688445    4964 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-ls7ft"]
Oct 31 01:24:51.692422 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:24:51.692406    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-ls7ft"
Oct 31 01:24:51.742738 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:24:51.742712    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-hppbk"]
Oct 31 01:24:51.802020 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:24:51.797030    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-ls7ft"]
Oct 31 01:24:51.814467 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:24:51.814439    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/38e3a193-4f84-478a-910e-bed4fac148e2-istio-token\") pod \"e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-ls7ft\" (UID: \"38e3a193-4f84-478a-910e-bed4fac148e2\") " pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-ls7ft"
Oct 31 01:24:51.814615 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:24:51.814479    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/479904d8-8280-451e-9182-e7d15c28c13d-istiod-ca-cert\") pod \"e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-hppbk\" (UID: \"479904d8-8280-451e-9182-e7d15c28c13d\") " pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-hppbk"
Oct 31 01:24:51.814615 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:24:51.814518    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/38e3a193-4f84-478a-910e-bed4fac148e2-workload-certs\") pod \"e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-ls7ft\" (UID: \"38e3a193-4f84-478a-910e-bed4fac148e2\") " pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-ls7ft"
Oct 31 01:24:51.814726 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:24:51.814614    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b8cxw\" (UniqueName: \"kubernetes.io/projected/479904d8-8280-451e-9182-e7d15c28c13d-kube-api-access-b8cxw\") pod \"e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-hppbk\" (UID: \"479904d8-8280-451e-9182-e7d15c28c13d\") " pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-hppbk"
Oct 31 01:24:51.814798 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:24:51.814781    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ln4sn\" (UniqueName: \"kubernetes.io/projected/38e3a193-4f84-478a-910e-bed4fac148e2-kube-api-access-ln4sn\") pod \"e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-ls7ft\" (UID: \"38e3a193-4f84-478a-910e-bed4fac148e2\") " pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-ls7ft"
Oct 31 01:24:51.814855 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:24:51.814822    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/479904d8-8280-451e-9182-e7d15c28c13d-istio-token\") pod \"e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-hppbk\" (UID: \"479904d8-8280-451e-9182-e7d15c28c13d\") " pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-hppbk"
Oct 31 01:24:51.814933 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:24:51.814920    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/479904d8-8280-451e-9182-e7d15c28c13d-istio-data\") pod \"e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-hppbk\" (UID: \"479904d8-8280-451e-9182-e7d15c28c13d\") " pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-hppbk"
Oct 31 01:24:51.815027 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:24:51.815015    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/479904d8-8280-451e-9182-e7d15c28c13d-workload-certs\") pod \"e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-hppbk\" (UID: \"479904d8-8280-451e-9182-e7d15c28c13d\") " pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-hppbk"
Oct 31 01:24:51.815083 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:24:51.815052    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/38e3a193-4f84-478a-910e-bed4fac148e2-istio-envoy\") pod \"e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-ls7ft\" (UID: \"38e3a193-4f84-478a-910e-bed4fac148e2\") " pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-ls7ft"
Oct 31 01:24:51.815127 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:24:51.815082    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/479904d8-8280-451e-9182-e7d15c28c13d-istio-envoy\") pod \"e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-hppbk\" (UID: \"479904d8-8280-451e-9182-e7d15c28c13d\") " pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-hppbk"
Oct 31 01:24:51.815171 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:24:51.815151    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/479904d8-8280-451e-9182-e7d15c28c13d-credential-socket\") pod \"e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-hppbk\" (UID: \"479904d8-8280-451e-9182-e7d15c28c13d\") " pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-hppbk"
Oct 31 01:24:51.815218 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:24:51.815203    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/38e3a193-4f84-478a-910e-bed4fac148e2-istio-podinfo\") pod \"e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-ls7ft\" (UID: \"38e3a193-4f84-478a-910e-bed4fac148e2\") " pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-ls7ft"
Oct 31 01:24:51.815291 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:24:51.815277    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/38e3a193-4f84-478a-910e-bed4fac148e2-istiod-ca-cert\") pod \"e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-ls7ft\" (UID: \"38e3a193-4f84-478a-910e-bed4fac148e2\") " pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-ls7ft"
Oct 31 01:24:51.815371 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:24:51.815357    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/38e3a193-4f84-478a-910e-bed4fac148e2-workload-socket\") pod \"e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-ls7ft\" (UID: \"38e3a193-4f84-478a-910e-bed4fac148e2\") " pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-ls7ft"
Oct 31 01:24:51.815411 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:24:51.815382    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/479904d8-8280-451e-9182-e7d15c28c13d-istio-podinfo\") pod \"e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-hppbk\" (UID: \"479904d8-8280-451e-9182-e7d15c28c13d\") " pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-hppbk"
Oct 31 01:24:51.815411 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:24:51.815400    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/38e3a193-4f84-478a-910e-bed4fac148e2-credential-socket\") pod \"e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-ls7ft\" (UID: \"38e3a193-4f84-478a-910e-bed4fac148e2\") " pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-ls7ft"
Oct 31 01:24:51.815507 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:24:51.815422    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/479904d8-8280-451e-9182-e7d15c28c13d-workload-socket\") pod \"e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-hppbk\" (UID: \"479904d8-8280-451e-9182-e7d15c28c13d\") " pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-hppbk"
Oct 31 01:24:51.815507 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:24:51.815472    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/38e3a193-4f84-478a-910e-bed4fac148e2-istio-data\") pod \"e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-ls7ft\" (UID: \"38e3a193-4f84-478a-910e-bed4fac148e2\") " pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-ls7ft"
Oct 31 01:24:51.916508 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:24:51.916439    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/479904d8-8280-451e-9182-e7d15c28c13d-istiod-ca-cert\") pod \"e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-hppbk\" (UID: \"479904d8-8280-451e-9182-e7d15c28c13d\") " pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-hppbk"
Oct 31 01:24:51.916508 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:24:51.916486    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/38e3a193-4f84-478a-910e-bed4fac148e2-workload-certs\") pod \"e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-ls7ft\" (UID: \"38e3a193-4f84-478a-910e-bed4fac148e2\") " pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-ls7ft"
Oct 31 01:24:51.917037 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:24:51.916515    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b8cxw\" (UniqueName: \"kubernetes.io/projected/479904d8-8280-451e-9182-e7d15c28c13d-kube-api-access-b8cxw\") pod \"e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-hppbk\" (UID: \"479904d8-8280-451e-9182-e7d15c28c13d\") " pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-hppbk"
Oct 31 01:24:51.917037 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:24:51.916580    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ln4sn\" (UniqueName: \"kubernetes.io/projected/38e3a193-4f84-478a-910e-bed4fac148e2-kube-api-access-ln4sn\") pod \"e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-ls7ft\" (UID: \"38e3a193-4f84-478a-910e-bed4fac148e2\") " pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-ls7ft"
Oct 31 01:24:51.917037 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:24:51.916611    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/479904d8-8280-451e-9182-e7d15c28c13d-istio-token\") pod \"e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-hppbk\" (UID: \"479904d8-8280-451e-9182-e7d15c28c13d\") " pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-hppbk"
Oct 31 01:24:51.917037 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:24:51.916665    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/479904d8-8280-451e-9182-e7d15c28c13d-istio-data\") pod \"e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-hppbk\" (UID: \"479904d8-8280-451e-9182-e7d15c28c13d\") " pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-hppbk"
Oct 31 01:24:51.917037 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:24:51.916760    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/479904d8-8280-451e-9182-e7d15c28c13d-workload-certs\") pod \"e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-hppbk\" (UID: \"479904d8-8280-451e-9182-e7d15c28c13d\") " pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-hppbk"
Oct 31 01:24:51.917037 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:24:51.916797    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/38e3a193-4f84-478a-910e-bed4fac148e2-istio-envoy\") pod \"e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-ls7ft\" (UID: \"38e3a193-4f84-478a-910e-bed4fac148e2\") " pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-ls7ft"
Oct 31 01:24:51.917037 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:24:51.916825    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/479904d8-8280-451e-9182-e7d15c28c13d-istio-envoy\") pod \"e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-hppbk\" (UID: \"479904d8-8280-451e-9182-e7d15c28c13d\") " pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-hppbk"
Oct 31 01:24:51.917037 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:24:51.916836    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/38e3a193-4f84-478a-910e-bed4fac148e2-workload-certs\") pod \"e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-ls7ft\" (UID: \"38e3a193-4f84-478a-910e-bed4fac148e2\") " pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-ls7ft"
Oct 31 01:24:51.917037 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:24:51.916873    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/479904d8-8280-451e-9182-e7d15c28c13d-credential-socket\") pod \"e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-hppbk\" (UID: \"479904d8-8280-451e-9182-e7d15c28c13d\") " pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-hppbk"
Oct 31 01:24:51.917037 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:24:51.916917    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/38e3a193-4f84-478a-910e-bed4fac148e2-istio-podinfo\") pod \"e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-ls7ft\" (UID: \"38e3a193-4f84-478a-910e-bed4fac148e2\") " pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-ls7ft"
Oct 31 01:24:51.917037 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:24:51.916974    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/38e3a193-4f84-478a-910e-bed4fac148e2-istiod-ca-cert\") pod \"e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-ls7ft\" (UID: \"38e3a193-4f84-478a-910e-bed4fac148e2\") " pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-ls7ft"
Oct 31 01:24:51.917037 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:24:51.917015    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/38e3a193-4f84-478a-910e-bed4fac148e2-workload-socket\") pod \"e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-ls7ft\" (UID: \"38e3a193-4f84-478a-910e-bed4fac148e2\") " pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-ls7ft"
Oct 31 01:24:51.918258 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:24:51.917047    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/479904d8-8280-451e-9182-e7d15c28c13d-istio-podinfo\") pod \"e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-hppbk\" (UID: \"479904d8-8280-451e-9182-e7d15c28c13d\") " pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-hppbk"
Oct 31 01:24:51.918258 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:24:51.917081    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/38e3a193-4f84-478a-910e-bed4fac148e2-credential-socket\") pod \"e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-ls7ft\" (UID: \"38e3a193-4f84-478a-910e-bed4fac148e2\") " pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-ls7ft"
Oct 31 01:24:51.918258 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:24:51.917111    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/479904d8-8280-451e-9182-e7d15c28c13d-workload-socket\") pod \"e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-hppbk\" (UID: \"479904d8-8280-451e-9182-e7d15c28c13d\") " pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-hppbk"
Oct 31 01:24:51.918258 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:24:51.917141    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/38e3a193-4f84-478a-910e-bed4fac148e2-istio-data\") pod \"e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-ls7ft\" (UID: \"38e3a193-4f84-478a-910e-bed4fac148e2\") " pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-ls7ft"
Oct 31 01:24:51.918258 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:24:51.917178    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/38e3a193-4f84-478a-910e-bed4fac148e2-istio-token\") pod \"e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-ls7ft\" (UID: \"38e3a193-4f84-478a-910e-bed4fac148e2\") " pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-ls7ft"
Oct 31 01:24:51.918258 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:24:51.917202    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/479904d8-8280-451e-9182-e7d15c28c13d-istiod-ca-cert\") pod \"e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-hppbk\" (UID: \"479904d8-8280-451e-9182-e7d15c28c13d\") " pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-hppbk"
Oct 31 01:24:51.918258 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:24:51.917045    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/479904d8-8280-451e-9182-e7d15c28c13d-workload-certs\") pod \"e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-hppbk\" (UID: \"479904d8-8280-451e-9182-e7d15c28c13d\") " pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-hppbk"
Oct 31 01:24:51.918258 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:24:51.917249    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/479904d8-8280-451e-9182-e7d15c28c13d-credential-socket\") pod \"e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-hppbk\" (UID: \"479904d8-8280-451e-9182-e7d15c28c13d\") " pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-hppbk"
Oct 31 01:24:51.918258 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:24:51.917563    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/38e3a193-4f84-478a-910e-bed4fac148e2-istio-data\") pod \"e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-ls7ft\" (UID: \"38e3a193-4f84-478a-910e-bed4fac148e2\") " pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-ls7ft"
Oct 31 01:24:51.918258 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:24:51.917656    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/38e3a193-4f84-478a-910e-bed4fac148e2-workload-socket\") pod \"e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-ls7ft\" (UID: \"38e3a193-4f84-478a-910e-bed4fac148e2\") " pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-ls7ft"
Oct 31 01:24:51.918258 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:24:51.917893    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/479904d8-8280-451e-9182-e7d15c28c13d-istio-data\") pod \"e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-hppbk\" (UID: \"479904d8-8280-451e-9182-e7d15c28c13d\") " pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-hppbk"
Oct 31 01:24:51.918258 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:24:51.918131    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/479904d8-8280-451e-9182-e7d15c28c13d-workload-socket\") pod \"e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-hppbk\" (UID: \"479904d8-8280-451e-9182-e7d15c28c13d\") " pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-hppbk"
Oct 31 01:24:51.918929 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:24:51.918413    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/38e3a193-4f84-478a-910e-bed4fac148e2-istiod-ca-cert\") pod \"e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-ls7ft\" (UID: \"38e3a193-4f84-478a-910e-bed4fac148e2\") " pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-ls7ft"
Oct 31 01:24:51.918929 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:24:51.918425    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/38e3a193-4f84-478a-910e-bed4fac148e2-credential-socket\") pod \"e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-ls7ft\" (UID: \"38e3a193-4f84-478a-910e-bed4fac148e2\") " pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-ls7ft"
Oct 31 01:24:51.920051 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:24:51.920027    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/479904d8-8280-451e-9182-e7d15c28c13d-istio-envoy\") pod \"e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-hppbk\" (UID: \"479904d8-8280-451e-9182-e7d15c28c13d\") " pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-hppbk"
Oct 31 01:24:51.920142 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:24:51.920073    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/479904d8-8280-451e-9182-e7d15c28c13d-istio-podinfo\") pod \"e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-hppbk\" (UID: \"479904d8-8280-451e-9182-e7d15c28c13d\") " pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-hppbk"
Oct 31 01:24:51.920142 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:24:51.920105    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/38e3a193-4f84-478a-910e-bed4fac148e2-istio-envoy\") pod \"e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-ls7ft\" (UID: \"38e3a193-4f84-478a-910e-bed4fac148e2\") " pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-ls7ft"
Oct 31 01:24:51.920271 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:24:51.920248    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/38e3a193-4f84-478a-910e-bed4fac148e2-istio-podinfo\") pod \"e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-ls7ft\" (UID: \"38e3a193-4f84-478a-910e-bed4fac148e2\") " pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-ls7ft"
Oct 31 01:24:51.973477 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:24:51.973383    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ln4sn\" (UniqueName: \"kubernetes.io/projected/38e3a193-4f84-478a-910e-bed4fac148e2-kube-api-access-ln4sn\") pod \"e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-ls7ft\" (UID: \"38e3a193-4f84-478a-910e-bed4fac148e2\") " pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-ls7ft"
Oct 31 01:24:51.973477 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:24:51.973439    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/479904d8-8280-451e-9182-e7d15c28c13d-istio-token\") pod \"e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-hppbk\" (UID: \"479904d8-8280-451e-9182-e7d15c28c13d\") " pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-hppbk"
Oct 31 01:24:51.973647 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:24:51.973555    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/38e3a193-4f84-478a-910e-bed4fac148e2-istio-token\") pod \"e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-ls7ft\" (UID: \"38e3a193-4f84-478a-910e-bed4fac148e2\") " pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-ls7ft"
Oct 31 01:24:51.973647 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:24:51.973599    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b8cxw\" (UniqueName: \"kubernetes.io/projected/479904d8-8280-451e-9182-e7d15c28c13d-kube-api-access-b8cxw\") pod \"e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-hppbk\" (UID: \"479904d8-8280-451e-9182-e7d15c28c13d\") " pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-hppbk"
Oct 31 01:24:52.006927 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:24:52.006907    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-ls7ft"
Oct 31 01:24:52.269176 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:24:52.269146    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-hppbk"
Oct 31 01:24:52.449088 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:24:52.449061    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-ls7ft"]
Oct 31 01:24:52.450159 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:24:52.450136    4964 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod38e3a193_4f84_478a_910e_bed4fac148e2.slice/crio-a84d06b5dda177894ba2429e7f9162c8b1d58c4bdc5e603bfa06284673f176eb WatchSource:0}: Error finding container a84d06b5dda177894ba2429e7f9162c8b1d58c4bdc5e603bfa06284673f176eb: Status 404 returned error can't find the container with id a84d06b5dda177894ba2429e7f9162c8b1d58c4bdc5e603bfa06284673f176eb
Oct 31 01:24:52.452514 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:24:52.452481    4964 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 01:24:52.452621 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:24:52.452561    4964 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 01:24:52.452621 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:24:52.452603    4964 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 01:24:52.653687 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:24:52.653662    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-hppbk"]
Oct 31 01:24:52.654333 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:24:52.654294    4964 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod479904d8_8280_451e_9182_e7d15c28c13d.slice/crio-8ee1886b623fa4f2cd4cb3a002f01aab919ba08ee32d49a545ef9bc4bdd9018a WatchSource:0}: Error finding container 8ee1886b623fa4f2cd4cb3a002f01aab919ba08ee32d49a545ef9bc4bdd9018a: Status 404 returned error can't find the container with id 8ee1886b623fa4f2cd4cb3a002f01aab919ba08ee32d49a545ef9bc4bdd9018a
Oct 31 01:24:52.656903 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:24:52.656867    4964 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 01:24:52.656980 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:24:52.656953    4964 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 01:24:52.657015 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:24:52.656993    4964 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 01:24:53.377483 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:24:53.377433    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-hppbk" event={"ID":"479904d8-8280-451e-9182-e7d15c28c13d","Type":"ContainerStarted","Data":"d3a14bedd81cb4cee73bdcd48999d61050eb6b4d9883e179b34b6c3cd0377ac8"}
Oct 31 01:24:53.378027 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:24:53.378007    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-hppbk" event={"ID":"479904d8-8280-451e-9182-e7d15c28c13d","Type":"ContainerStarted","Data":"8ee1886b623fa4f2cd4cb3a002f01aab919ba08ee32d49a545ef9bc4bdd9018a"}
Oct 31 01:24:53.379196 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:24:53.379160    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-ls7ft" event={"ID":"38e3a193-4f84-478a-910e-bed4fac148e2","Type":"ContainerStarted","Data":"b74fe5fc9f28cac3591d90f5d7113bf442f66534f83451f94e39505a09b5db56"}
Oct 31 01:24:53.379303 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:24:53.379216    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-ls7ft" event={"ID":"38e3a193-4f84-478a-910e-bed4fac148e2","Type":"ContainerStarted","Data":"a84d06b5dda177894ba2429e7f9162c8b1d58c4bdc5e603bfa06284673f176eb"}
Oct 31 01:24:53.721448 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:24:53.721379    4964 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-ls7ft" podStartSLOduration=2.7213609930000002 podStartE2EDuration="2.721360993s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:24:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:24:53.720853257 +0000 UTC m=+1262.197419637" watchObservedRunningTime="2026-10-31 01:24:53.721360993 +0000 UTC m=+1262.197927373"
Oct 31 01:24:53.723761 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:24:53.723716    4964 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-hppbk" podStartSLOduration=2.723699925 podStartE2EDuration="2.723699925s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:24:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:24:53.585589343 +0000 UTC m=+1262.062155725" watchObservedRunningTime="2026-10-31 01:24:53.723699925 +0000 UTC m=+1262.200266306"
Oct 31 01:24:54.008399 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:24:54.007682    4964 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-ls7ft"
Oct 31 01:24:54.025339 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:24:54.024465    4964 patch_prober.go:29] interesting pod/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-ls7ft container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" start-of-body=
Oct 31 01:24:54.025339 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:24:54.024543    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-ls7ft" podUID="38e3a193-4f84-478a-910e-bed4fac148e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused"
Oct 31 01:24:54.269752 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:24:54.269667    4964 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-hppbk"
Oct 31 01:24:54.271449 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:24:54.271367    4964 patch_prober.go:29] interesting pod/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-hppbk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" start-of-body=
Oct 31 01:24:54.271449 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:24:54.271434    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-hppbk" podUID="479904d8-8280-451e-9182-e7d15c28c13d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused"
Oct 31 01:24:55.015341 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:24:55.014427    4964 patch_prober.go:29] interesting pod/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-ls7ft container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" start-of-body=
Oct 31 01:24:55.015341 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:24:55.014519    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-ls7ft" podUID="38e3a193-4f84-478a-910e-bed4fac148e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused"
Oct 31 01:24:55.242341 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:24:55.241951    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:24:55.271251 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:24:55.269880    4964 patch_prober.go:29] interesting pod/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-hppbk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" start-of-body=
Oct 31 01:24:55.271251 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:24:55.269975    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-hppbk" podUID="479904d8-8280-451e-9182-e7d15c28c13d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused"
Oct 31 01:24:56.009341 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:24:56.008742    4964 patch_prober.go:29] interesting pod/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-ls7ft container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" start-of-body=
Oct 31 01:24:56.009341 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:24:56.008815    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-ls7ft" podUID="38e3a193-4f84-478a-910e-bed4fac148e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused"
Oct 31 01:24:56.269735 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:24:56.269651    4964 patch_prober.go:29] interesting pod/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-hppbk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" start-of-body=
Oct 31 01:24:56.269735 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:24:56.269706    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-hppbk" podUID="479904d8-8280-451e-9182-e7d15c28c13d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused"
Oct 31 01:24:57.021832 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:24:57.016846    4964 patch_prober.go:29] interesting pod/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-ls7ft container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" start-of-body=
Oct 31 01:24:57.021832 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:24:57.016905    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-ls7ft" podUID="38e3a193-4f84-478a-910e-bed4fac148e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused"
Oct 31 01:24:57.270466 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:24:57.270436    4964 patch_prober.go:29] interesting pod/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-hppbk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" start-of-body=
Oct 31 01:24:57.270870 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:24:57.270487    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-hppbk" podUID="479904d8-8280-451e-9182-e7d15c28c13d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused"
Oct 31 01:24:58.007720 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:24:58.007683    4964 patch_prober.go:29] interesting pod/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-ls7ft container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" start-of-body=
Oct 31 01:24:58.008006 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:24:58.007975    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-ls7ft" podUID="38e3a193-4f84-478a-910e-bed4fac148e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused"
Oct 31 01:24:58.269865 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:24:58.269786    4964 patch_prober.go:29] interesting pod/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-hppbk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" start-of-body=
Oct 31 01:24:58.269865 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:24:58.269833    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-hppbk" podUID="479904d8-8280-451e-9182-e7d15c28c13d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused"
Oct 31 01:24:59.008064 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:24:59.008019    4964 patch_prober.go:29] interesting pod/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-ls7ft container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" start-of-body=
Oct 31 01:24:59.008577 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:24:59.008095    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-ls7ft" podUID="38e3a193-4f84-478a-910e-bed4fac148e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused"
Oct 31 01:24:59.269756 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:24:59.269665    4964 patch_prober.go:29] interesting pod/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-hppbk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" start-of-body=
Oct 31 01:24:59.269931 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:24:59.269745    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-hppbk" podUID="479904d8-8280-451e-9182-e7d15c28c13d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused"
Oct 31 01:25:00.012005 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:25:00.011976    4964 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-ls7ft"
Oct 31 01:25:00.012480 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:25:00.012350    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-ls7ft"
Oct 31 01:25:00.013119 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:25:00.013102    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-ls7ft"
Oct 31 01:25:00.238076 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:25:00.238010    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:25:00.274095 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:25:00.274015    4964 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-hppbk"
Oct 31 01:25:00.274340 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:25:00.274304    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-hppbk"
Oct 31 01:25:00.275038 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:25:00.275019    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-hppbk"
Oct 31 01:25:03.218502 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:25:03.218457    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:25:03.406172 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:25:03.406124    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:25:04.751207 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:25:04.751166    4964 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-baacee/distinct-w1-baacee-kserve-6c85cfb7d7-stkqq" podUID="df6d0bdd-382d-49b7-879c-15cb5371246e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": context deadline exceeded"
Oct 31 01:25:04.805841 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:25:04.804426    4964 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-baacee/premium-w1-baacee-kserve-69cc8c6456-7ngq9" podUID="15028ba0-c888-4780-b770-6acb2d03d70a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded"
Oct 31 01:25:04.968423 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:25:04.968384    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:25:05.334899 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:25:05.334858    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:25:05.521377 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:25:05.521240    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:25:06.089143 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:25:06.089093    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:25:07.202888 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:25:07.202848    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:25:07.872888 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:25:07.872857    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:25:08.488393 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:25:08.488357    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:25:08.662963 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:25:08.662930    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:25:08.869600 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:25:08.869526    4964 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-18bf762c-c84588867-fcv87"]
Oct 31 01:25:08.873931 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:25:08.873913    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-18bf762c-c84588867-fcv87"
Oct 31 01:25:08.930329 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:25:08.930287    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-18bf762c-c84588867-fcv87"]
Oct 31 01:25:08.977703 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:25:08.977674    4964 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-18bf762c\""
Oct 31 01:25:09.004866 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:25:09.004841    4964 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-18bf762c-dockercfg-fbpk8\""
Oct 31 01:25:09.077357 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:25:09.074981    4964 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-18bf762c-6cc59897d9-w4n2s"]
Oct 31 01:25:09.077971 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:25:09.077934    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p6vqk\" (UniqueName: \"kubernetes.io/projected/74f9ca59-18e2-4fb6-b596-08f86168bb39-kube-api-access-p6vqk\") pod \"payload-pre-processing-e2e-mt-18bf762c-c84588867-fcv87\" (UID: \"74f9ca59-18e2-4fb6-b596-08f86168bb39\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-18bf762c-c84588867-fcv87"
Oct 31 01:25:09.078662 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:25:09.078640    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/74f9ca59-18e2-4fb6-b596-08f86168bb39-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-18bf762c-c84588867-fcv87\" (UID: \"74f9ca59-18e2-4fb6-b596-08f86168bb39\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-18bf762c-c84588867-fcv87"
Oct 31 01:25:09.096522 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:25:09.096492    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-18bf762c-6cc59897d9-w4n2s"]
Oct 31 01:25:09.096700 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:25:09.096580    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-18bf762c-6cc59897d9-w4n2s"
Oct 31 01:25:09.180339 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:25:09.180244    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-df7jz\" (UniqueName: \"kubernetes.io/projected/4a70412f-a2ba-4f8c-896d-945e58e057db-kube-api-access-df7jz\") pod \"payload-processing-e2e-mt-18bf762c-6cc59897d9-w4n2s\" (UID: \"4a70412f-a2ba-4f8c-896d-945e58e057db\") " pod="openshift-ingress/payload-processing-e2e-mt-18bf762c-6cc59897d9-w4n2s"
Oct 31 01:25:09.180339 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:25:09.180301    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/74f9ca59-18e2-4fb6-b596-08f86168bb39-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-18bf762c-c84588867-fcv87\" (UID: \"74f9ca59-18e2-4fb6-b596-08f86168bb39\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-18bf762c-c84588867-fcv87"
Oct 31 01:25:09.180575 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:25:09.180448    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p6vqk\" (UniqueName: \"kubernetes.io/projected/74f9ca59-18e2-4fb6-b596-08f86168bb39-kube-api-access-p6vqk\") pod \"payload-pre-processing-e2e-mt-18bf762c-c84588867-fcv87\" (UID: \"74f9ca59-18e2-4fb6-b596-08f86168bb39\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-18bf762c-c84588867-fcv87"
Oct 31 01:25:09.180575 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:25:09.180489    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4a70412f-a2ba-4f8c-896d-945e58e057db-plugins-config-volume\") pod \"payload-processing-e2e-mt-18bf762c-6cc59897d9-w4n2s\" (UID: \"4a70412f-a2ba-4f8c-896d-945e58e057db\") " pod="openshift-ingress/payload-processing-e2e-mt-18bf762c-6cc59897d9-w4n2s"
Oct 31 01:25:09.181066 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:25:09.181042    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/74f9ca59-18e2-4fb6-b596-08f86168bb39-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-18bf762c-c84588867-fcv87\" (UID: \"74f9ca59-18e2-4fb6-b596-08f86168bb39\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-18bf762c-c84588867-fcv87"
Oct 31 01:25:09.237361 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:25:09.237295    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p6vqk\" (UniqueName: \"kubernetes.io/projected/74f9ca59-18e2-4fb6-b596-08f86168bb39-kube-api-access-p6vqk\") pod \"payload-pre-processing-e2e-mt-18bf762c-c84588867-fcv87\" (UID: \"74f9ca59-18e2-4fb6-b596-08f86168bb39\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-18bf762c-c84588867-fcv87"
Oct 31 01:25:09.282010 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:25:09.281978    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4a70412f-a2ba-4f8c-896d-945e58e057db-plugins-config-volume\") pod \"payload-processing-e2e-mt-18bf762c-6cc59897d9-w4n2s\" (UID: \"4a70412f-a2ba-4f8c-896d-945e58e057db\") " pod="openshift-ingress/payload-processing-e2e-mt-18bf762c-6cc59897d9-w4n2s"
Oct 31 01:25:09.282180 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:25:09.282166    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-df7jz\" (UniqueName: \"kubernetes.io/projected/4a70412f-a2ba-4f8c-896d-945e58e057db-kube-api-access-df7jz\") pod \"payload-processing-e2e-mt-18bf762c-6cc59897d9-w4n2s\" (UID: \"4a70412f-a2ba-4f8c-896d-945e58e057db\") " pod="openshift-ingress/payload-processing-e2e-mt-18bf762c-6cc59897d9-w4n2s"
Oct 31 01:25:09.282743 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:25:09.282719    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4a70412f-a2ba-4f8c-896d-945e58e057db-plugins-config-volume\") pod \"payload-processing-e2e-mt-18bf762c-6cc59897d9-w4n2s\" (UID: \"4a70412f-a2ba-4f8c-896d-945e58e057db\") " pod="openshift-ingress/payload-processing-e2e-mt-18bf762c-6cc59897d9-w4n2s"
Oct 31 01:25:09.338498 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:25:09.338459    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-df7jz\" (UniqueName: \"kubernetes.io/projected/4a70412f-a2ba-4f8c-896d-945e58e057db-kube-api-access-df7jz\") pod \"payload-processing-e2e-mt-18bf762c-6cc59897d9-w4n2s\" (UID: \"4a70412f-a2ba-4f8c-896d-945e58e057db\") " pod="openshift-ingress/payload-processing-e2e-mt-18bf762c-6cc59897d9-w4n2s"
Oct 31 01:25:09.414202 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:25:09.414161    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-18bf762c-6cc59897d9-w4n2s"
Oct 31 01:25:09.489887 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:25:09.489654    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-18bf762c-c84588867-fcv87"
Oct 31 01:25:09.790344 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:25:09.790299    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-18bf762c-6cc59897d9-w4n2s"]
Oct 31 01:25:09.792055 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:25:09.792028    4964 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a70412f_a2ba_4f8c_896d_945e58e057db.slice/crio-7c280845b05c1d0540cdb32387897eeb970a35b2fec3cfc55d17f690a22aac12 WatchSource:0}: Error finding container 7c280845b05c1d0540cdb32387897eeb970a35b2fec3cfc55d17f690a22aac12: Status 404 returned error can't find the container with id 7c280845b05c1d0540cdb32387897eeb970a35b2fec3cfc55d17f690a22aac12
Oct 31 01:25:09.902357 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:25:09.902329    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-18bf762c-c84588867-fcv87"]
Oct 31 01:25:09.903301 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:25:09.903279    4964 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod74f9ca59_18e2_4fb6_b596_08f86168bb39.slice/crio-c8b4b6f4a48f2ad6de69676f4d7a902fb8f5ec1acfa83905249b6b81afd526cb WatchSource:0}: Error finding container c8b4b6f4a48f2ad6de69676f4d7a902fb8f5ec1acfa83905249b6b81afd526cb: Status 404 returned error can't find the container with id c8b4b6f4a48f2ad6de69676f4d7a902fb8f5ec1acfa83905249b6b81afd526cb
Oct 31 01:25:09.940737 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:25:09.940707    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-18bf762c-6cc59897d9-w4n2s" event={"ID":"4a70412f-a2ba-4f8c-896d-945e58e057db","Type":"ContainerStarted","Data":"0d826f90a037e14dcd2b85c2a53a3d1ea5ce5e08644cbdb45fa9fa8577e665da"}
Oct 31 01:25:09.940844 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:25:09.940753    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-18bf762c-6cc59897d9-w4n2s" event={"ID":"4a70412f-a2ba-4f8c-896d-945e58e057db","Type":"ContainerStarted","Data":"7c280845b05c1d0540cdb32387897eeb970a35b2fec3cfc55d17f690a22aac12"}
Oct 31 01:25:09.940844 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:25:09.940803    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-18bf762c-6cc59897d9-w4n2s"
Oct 31 01:25:09.942366 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:25:09.942340    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-18bf762c-c84588867-fcv87" event={"ID":"74f9ca59-18e2-4fb6-b596-08f86168bb39","Type":"ContainerStarted","Data":"c8b4b6f4a48f2ad6de69676f4d7a902fb8f5ec1acfa83905249b6b81afd526cb"}
Oct 31 01:25:10.055232 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:25:10.055133    4964 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-18bf762c-6cc59897d9-w4n2s" podStartSLOduration=2.055120058 podStartE2EDuration="2.055120058s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:25:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:25:10.052423595 +0000 UTC m=+1278.528989972" watchObservedRunningTime="2026-10-31 01:25:10.055120058 +0000 UTC m=+1278.531686481"
Oct 31 01:25:10.950713 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:25:10.950676    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-18bf762c-c84588867-fcv87" event={"ID":"74f9ca59-18e2-4fb6-b596-08f86168bb39","Type":"ContainerStarted","Data":"d686c8d947f75ed277c24a533e1078c025f9571cd00b15757a1c30871ae644b1"}
Oct 31 01:25:10.951164 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:25:10.950841    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-18bf762c-c84588867-fcv87"
Oct 31 01:25:11.052474 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:25:11.052442    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:25:11.073451 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:25:11.073293    4964 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-18bf762c-c84588867-fcv87" podStartSLOduration=3.073272523 podStartE2EDuration="3.073272523s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:25:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:25:11.071705491 +0000 UTC m=+1279.548271870" watchObservedRunningTime="2026-10-31 01:25:11.073272523 +0000 UTC m=+1279.549838904"
Oct 31 01:25:11.226726 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:25:11.226690    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:25:11.238124 ip-10-0-133-197 kubenswrapper[4964]: E1031 01:25:11.238093    4964 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q56x5" containerName="extractor"
Oct 31 01:25:14.149937 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:25:14.149907    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:25:14.738931 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:25:14.738899    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:25:15.910490 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:25:15.910430    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:25:19.378485 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:25:19.378453    4964 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-cdtt2"]
Oct 31 01:25:19.382949 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:25:19.382926    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-cdtt2"
Oct 31 01:25:19.487681 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:25:19.487651    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-cdtt2"]
Oct 31 01:25:19.526396 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:25:19.526358    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea794d28-3a8e-4262-bdaa-734f3510cfbe-credential-socket\") pod \"e2e-worker-w1-baacee-openshift-default-57f845d5bf-cdtt2\" (UID: \"ea794d28-3a8e-4262-bdaa-734f3510cfbe\") " pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-cdtt2"
Oct 31 01:25:19.526687 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:25:19.526664    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ea794d28-3a8e-4262-bdaa-734f3510cfbe-workload-certs\") pod \"e2e-worker-w1-baacee-openshift-default-57f845d5bf-cdtt2\" (UID: \"ea794d28-3a8e-4262-bdaa-734f3510cfbe\") " pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-cdtt2"
Oct 31 01:25:19.526835 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:25:19.526812    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ea794d28-3a8e-4262-bdaa-734f3510cfbe-istio-envoy\") pod \"e2e-worker-w1-baacee-openshift-default-57f845d5bf-cdtt2\" (UID: \"ea794d28-3a8e-4262-bdaa-734f3510cfbe\") " pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-cdtt2"
Oct 31 01:25:19.526950 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:25:19.526860    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ea794d28-3a8e-4262-bdaa-734f3510cfbe-istio-token\") pod \"e2e-worker-w1-baacee-openshift-default-57f845d5bf-cdtt2\" (UID: \"ea794d28-3a8e-4262-bdaa-734f3510cfbe\") " pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-cdtt2"
Oct 31 01:25:19.526950 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:25:19.526945    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ea794d28-3a8e-4262-bdaa-734f3510cfbe-istiod-ca-cert\") pod \"e2e-worker-w1-baacee-openshift-default-57f845d5bf-cdtt2\" (UID: \"ea794d28-3a8e-4262-bdaa-734f3510cfbe\") " pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-cdtt2"
Oct 31 01:25:19.527045 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:25:19.527019    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ea794d28-3a8e-4262-bdaa-734f3510cfbe-istio-data\") pod \"e2e-worker-w1-baacee-openshift-default-57f845d5bf-cdtt2\" (UID: \"ea794d28-3a8e-4262-bdaa-734f3510cfbe\") " pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-cdtt2"
Oct 31 01:25:19.527082 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:25:19.527057    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-68kzr\" (UniqueName: \"kubernetes.io/projected/ea794d28-3a8e-4262-bdaa-734f3510cfbe-kube-api-access-68kzr\") pod \"e2e-worker-w1-baacee-openshift-default-57f845d5bf-cdtt2\" (UID: \"ea794d28-3a8e-4262-bdaa-734f3510cfbe\") " pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-cdtt2"
Oct 31 01:25:19.527299 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:25:19.527281    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea794d28-3a8e-4262-bdaa-734f3510cfbe-workload-socket\") pod \"e2e-worker-w1-baacee-openshift-default-57f845d5bf-cdtt2\" (UID: \"ea794d28-3a8e-4262-bdaa-734f3510cfbe\") " pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-cdtt2"
Oct 31 01:25:19.527423 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:25:19.527402    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ea794d28-3a8e-4262-bdaa-734f3510cfbe-istio-podinfo\") pod \"e2e-worker-w1-baacee-openshift-default-57f845d5bf-cdtt2\" (UID: \"ea794d28-3a8e-4262-bdaa-734f3510cfbe\") " pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-cdtt2"
Oct 31 01:25:19.629083 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:25:19.628979    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ea794d28-3a8e-4262-bdaa-734f3510cfbe-istio-podinfo\") pod \"e2e-worker-w1-baacee-openshift-default-57f845d5bf-cdtt2\" (UID: \"ea794d28-3a8e-4262-bdaa-734f3510cfbe\") " pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-cdtt2"
Oct 31 01:25:19.629384 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:25:19.629117    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea794d28-3a8e-4262-bdaa-734f3510cfbe-credential-socket\") pod \"e2e-worker-w1-baacee-openshift-default-57f845d5bf-cdtt2\" (UID: \"ea794d28-3a8e-4262-bdaa-734f3510cfbe\") " pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-cdtt2"
Oct 31 01:25:19.629384 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:25:19.629163    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ea794d28-3a8e-4262-bdaa-734f3510cfbe-workload-certs\") pod \"e2e-worker-w1-baacee-openshift-default-57f845d5bf-cdtt2\" (UID: \"ea794d28-3a8e-4262-bdaa-734f3510cfbe\") " pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-cdtt2"
Oct 31 01:25:19.629384 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:25:19.629183    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ea794d28-3a8e-4262-bdaa-734f3510cfbe-istio-envoy\") pod \"e2e-worker-w1-baacee-openshift-default-57f845d5bf-cdtt2\" (UID: \"ea794d28-3a8e-4262-bdaa-734f3510cfbe\") " pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-cdtt2"
Oct 31 01:25:19.629384 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:25:19.629220    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ea794d28-3a8e-4262-bdaa-734f3510cfbe-istio-token\") pod \"e2e-worker-w1-baacee-openshift-default-57f845d5bf-cdtt2\" (UID: \"ea794d28-3a8e-4262-bdaa-734f3510cfbe\") " pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-cdtt2"
Oct 31 01:25:19.629384 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:25:19.629254    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ea794d28-3a8e-4262-bdaa-734f3510cfbe-istiod-ca-cert\") pod \"e2e-worker-w1-baacee-openshift-default-57f845d5bf-cdtt2\" (UID: \"ea794d28-3a8e-4262-bdaa-734f3510cfbe\") " pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-cdtt2"
Oct 31 01:25:19.629807 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:25:19.629405    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ea794d28-3a8e-4262-bdaa-734f3510cfbe-istio-data\") pod \"e2e-worker-w1-baacee-openshift-default-57f845d5bf-cdtt2\" (UID: \"ea794d28-3a8e-4262-bdaa-734f3510cfbe\") " pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-cdtt2"
Oct 31 01:25:19.629807 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:25:19.629432    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-68kzr\" (UniqueName: \"kubernetes.io/projected/ea794d28-3a8e-4262-bdaa-734f3510cfbe-kube-api-access-68kzr\") pod \"e2e-worker-w1-baacee-openshift-default-57f845d5bf-cdtt2\" (UID: \"ea794d28-3a8e-4262-bdaa-734f3510cfbe\") " pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-cdtt2"
Oct 31 01:25:19.629807 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:25:19.629592    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea794d28-3a8e-4262-bdaa-734f3510cfbe-workload-socket\") pod \"e2e-worker-w1-baacee-openshift-default-57f845d5bf-cdtt2\" (UID: \"ea794d28-3a8e-4262-bdaa-734f3510cfbe\") " pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-cdtt2"
Oct 31 01:25:19.630074 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:25:19.630049    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea794d28-3a8e-4262-bdaa-734f3510cfbe-workload-socket\") pod \"e2e-worker-w1-baacee-openshift-default-57f845d5bf-cdtt2\" (UID: \"ea794d28-3a8e-4262-bdaa-734f3510cfbe\") " pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-cdtt2"
Oct 31 01:25:19.631123 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:25:19.631100    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea794d28-3a8e-4262-bdaa-734f3510cfbe-credential-socket\") pod \"e2e-worker-w1-baacee-openshift-default-57f845d5bf-cdtt2\" (UID: \"ea794d28-3a8e-4262-bdaa-734f3510cfbe\") " pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-cdtt2"
Oct 31 01:25:19.631378 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:25:19.631355    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ea794d28-3a8e-4262-bdaa-734f3510cfbe-workload-certs\") pod \"e2e-worker-w1-baacee-openshift-default-57f845d5bf-cdtt2\" (UID: \"ea794d28-3a8e-4262-bdaa-734f3510cfbe\") " pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-cdtt2"
Oct 31 01:25:19.632242 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:25:19.632218    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ea794d28-3a8e-4262-bdaa-734f3510cfbe-istio-data\") pod \"e2e-worker-w1-baacee-openshift-default-57f845d5bf-cdtt2\" (UID: \"ea794d28-3a8e-4262-bdaa-734f3510cfbe\") " pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-cdtt2"
Oct 31 01:25:19.633475 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:25:19.633453    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ea794d28-3a8e-4262-bdaa-734f3510cfbe-istiod-ca-cert\") pod \"e2e-worker-w1-baacee-openshift-default-57f845d5bf-cdtt2\" (UID: \"ea794d28-3a8e-4262-bdaa-734f3510cfbe\") " pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-cdtt2"
Oct 31 01:25:19.638830 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:25:19.638808    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ea794d28-3a8e-4262-bdaa-734f3510cfbe-istio-podinfo\") pod \"e2e-worker-w1-baacee-openshift-default-57f845d5bf-cdtt2\" (UID: \"ea794d28-3a8e-4262-bdaa-734f3510cfbe\") " pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-cdtt2"
Oct 31 01:25:19.639763 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:25:19.639742    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ea794d28-3a8e-4262-bdaa-734f3510cfbe-istio-envoy\") pod \"e2e-worker-w1-baacee-openshift-default-57f845d5bf-cdtt2\" (UID: \"ea794d28-3a8e-4262-bdaa-734f3510cfbe\") " pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-cdtt2"
Oct 31 01:25:19.686923 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:25:19.686899    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ea794d28-3a8e-4262-bdaa-734f3510cfbe-istio-token\") pod \"e2e-worker-w1-baacee-openshift-default-57f845d5bf-cdtt2\" (UID: \"ea794d28-3a8e-4262-bdaa-734f3510cfbe\") " pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-cdtt2"
Oct 31 01:25:19.688303 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:25:19.688287    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-68kzr\" (UniqueName: \"kubernetes.io/projected/ea794d28-3a8e-4262-bdaa-734f3510cfbe-kube-api-access-68kzr\") pod \"e2e-worker-w1-baacee-openshift-default-57f845d5bf-cdtt2\" (UID: \"ea794d28-3a8e-4262-bdaa-734f3510cfbe\") " pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-cdtt2"
Oct 31 01:25:19.697251 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:25:19.697231    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-cdtt2"
Oct 31 01:25:20.141230 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:25:20.141153    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-cdtt2"]
Oct 31 01:25:20.145124 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:25:20.145088    4964 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea794d28_3a8e_4262_bdaa_734f3510cfbe.slice/crio-2af817e99adf4134d43937626b2a054cff12c21db7d84b7f887a07d570ef6d05 WatchSource:0}: Error finding container 2af817e99adf4134d43937626b2a054cff12c21db7d84b7f887a07d570ef6d05: Status 404 returned error can't find the container with id 2af817e99adf4134d43937626b2a054cff12c21db7d84b7f887a07d570ef6d05
Oct 31 01:25:20.148369 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:25:20.148283    4964 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 01:25:20.148498 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:25:20.148387    4964 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 01:25:20.148498 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:25:20.148438    4964 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 01:25:20.758788 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:25:20.758751    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:25:20.954871 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:25:20.954840    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-18bf762c-6cc59897d9-w4n2s"
Oct 31 01:25:21.033181 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:25:21.033094    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-cdtt2" event={"ID":"ea794d28-3a8e-4262-bdaa-734f3510cfbe","Type":"ContainerStarted","Data":"4900b06e3bcba7ad06cfc24b420b0e3221df977167ed690370636bcae546bee2"}
Oct 31 01:25:21.033181 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:25:21.033146    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-cdtt2" event={"ID":"ea794d28-3a8e-4262-bdaa-734f3510cfbe","Type":"ContainerStarted","Data":"2af817e99adf4134d43937626b2a054cff12c21db7d84b7f887a07d570ef6d05"}
Oct 31 01:25:21.715654 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:25:21.715626    4964 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-cdtt2"
Oct 31 01:25:21.722339 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:25:21.717552    4964 patch_prober.go:29] interesting pod/e2e-worker-w1-baacee-openshift-default-57f845d5bf-cdtt2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body=
Oct 31 01:25:21.722339 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:25:21.717604    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-cdtt2" podUID="ea794d28-3a8e-4262-bdaa-734f3510cfbe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused"
Oct 31 01:25:21.961625 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:25:21.961594    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-18bf762c-c84588867-fcv87"
Oct 31 01:25:22.079185 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:25:22.079128    4964 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-cdtt2" podStartSLOduration=3.07911304 podStartE2EDuration="3.07911304s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:25:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:25:21.226737399 +0000 UTC m=+1289.703303779" watchObservedRunningTime="2026-10-31 01:25:22.07911304 +0000 UTC m=+1290.555679418"
Oct 31 01:25:22.698772 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:25:22.698729    4964 patch_prober.go:29] interesting pod/e2e-worker-w1-baacee-openshift-default-57f845d5bf-cdtt2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body=
Oct 31 01:25:22.698982 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:25:22.698809    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-cdtt2" podUID="ea794d28-3a8e-4262-bdaa-734f3510cfbe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused"
Oct 31 01:25:23.710960 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:25:23.710923    4964 patch_prober.go:29] interesting pod/e2e-worker-w1-baacee-openshift-default-57f845d5bf-cdtt2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body=
Oct 31 01:25:23.711862 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:25:23.711828    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-cdtt2" podUID="ea794d28-3a8e-4262-bdaa-734f3510cfbe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused"
Oct 31 01:25:24.583338 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:25:24.583217    4964 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a8e157/distinct-2-w3-a8e157-kserve-9ddd45d6-ng7j7" podUID="bc7a71df-c523-4590-91df-cf8fdf3ed19e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:25:24.703344 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:25:24.698745    4964 patch_prober.go:29] interesting pod/e2e-worker-w1-baacee-openshift-default-57f845d5bf-cdtt2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body=
Oct 31 01:25:24.703344 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:25:24.698808    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-cdtt2" podUID="ea794d28-3a8e-4262-bdaa-734f3510cfbe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused"
Oct 31 01:25:24.703645 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:25:24.703392    4964 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-baacee/premium-w1-baacee-kserve-69cc8c6456-7ngq9" podUID="15028ba0-c888-4780-b770-6acb2d03d70a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:25:24.735344 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:25:24.734785    4964 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-baacee/distinct-w1-baacee-kserve-6c85cfb7d7-stkqq" podUID="df6d0bdd-382d-49b7-879c-15cb5371246e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:25:25.147345 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:25:25.146932    4964 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a8e157/distinct-w3-a8e157-kserve-549559dd5c-t29tw" podUID="4b0fd818-20c2-4dd6-9846-f361c236dbd9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:25:25.710344 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:25:25.708417    4964 patch_prober.go:29] interesting pod/e2e-worker-w1-baacee-openshift-default-57f845d5bf-cdtt2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body=
Oct 31 01:25:25.710344 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:25:25.708485    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-cdtt2" podUID="ea794d28-3a8e-4262-bdaa-734f3510cfbe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused"
Oct 31 01:25:26.388004 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:25:26.387968    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:25:26.715439 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:25:26.707199    4964 patch_prober.go:29] interesting pod/e2e-worker-w1-baacee-openshift-default-57f845d5bf-cdtt2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body=
Oct 31 01:25:26.715439 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:25:26.707254    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-cdtt2" podUID="ea794d28-3a8e-4262-bdaa-734f3510cfbe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused"
Oct 31 01:25:27.697996 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:25:27.697958    4964 patch_prober.go:29] interesting pod/e2e-worker-w1-baacee-openshift-default-57f845d5bf-cdtt2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body=
Oct 31 01:25:27.698526 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:25:27.698021    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-cdtt2" podUID="ea794d28-3a8e-4262-bdaa-734f3510cfbe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused"
Oct 31 01:25:28.703414 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:25:28.703384    4964 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-cdtt2"
Oct 31 01:25:28.703912 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:25:28.703891    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-cdtt2"
Oct 31 01:25:28.704657 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:25:28.704626    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-cdtt2"
Oct 31 01:25:29.565078 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:25:29.565047    4964 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-8f6b5b/e2e-conflict-a-8bd02cd2-kserve-7c55c85bd6-rwlfr"]
Oct 31 01:25:29.646383 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:25:29.646348    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:25:30.227852 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:25:30.227766    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:25:30.438023 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:25:30.437985    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:25:31.336924 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:25:31.336870    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:25:31.613713 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:25:31.613629    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:25:31.826475 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:25:31.826442    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:25:36.262840 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:25:36.262800    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:25:36.487456 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:25:36.487420    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:25:41.238376 ip-10-0-133-197 kubenswrapper[4964]: E1031 01:25:41.238309    4964 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q56x5" containerName="extractor"
Oct 31 01:25:41.329737 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:25:41.329698    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:25:44.587060 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:25:44.587000    4964 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-8f6b5b/e2e-conflict-a-8bd02cd2-kserve-7c55c85bd6-rwlfr" podUID="766d96dd-2ba9-4216-9388-7a29c6aaf42c" containerName="main" containerID="cri-o://6bf86c13591c3c1f4557f5340eb804a56d5aa4e2055fe659cbfb4a52450faeb9" gracePeriod=45
Oct 31 01:25:44.881913 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:25:44.881886    4964 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-8f6b5b/e2e-conflict-a-8bd02cd2-kserve-7c55c85bd6-rwlfr"
Oct 31 01:25:44.973183 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:25:44.973154    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/766d96dd-2ba9-4216-9388-7a29c6aaf42c-tls-certs\" (UniqueName: \"kubernetes.io/secret/766d96dd-2ba9-4216-9388-7a29c6aaf42c-tls-certs\") pod \"766d96dd-2ba9-4216-9388-7a29c6aaf42c\" (UID: \"766d96dd-2ba9-4216-9388-7a29c6aaf42c\") "
Oct 31 01:25:44.973410 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:25:44.973230    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/766d96dd-2ba9-4216-9388-7a29c6aaf42c-kube-api-access-vsbkz\" (UniqueName: \"kubernetes.io/projected/766d96dd-2ba9-4216-9388-7a29c6aaf42c-kube-api-access-vsbkz\") pod \"766d96dd-2ba9-4216-9388-7a29c6aaf42c\" (UID: \"766d96dd-2ba9-4216-9388-7a29c6aaf42c\") "
Oct 31 01:25:44.973410 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:25:44.973382    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/766d96dd-2ba9-4216-9388-7a29c6aaf42c-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/766d96dd-2ba9-4216-9388-7a29c6aaf42c-tmp-dir\") pod \"766d96dd-2ba9-4216-9388-7a29c6aaf42c\" (UID: \"766d96dd-2ba9-4216-9388-7a29c6aaf42c\") "
Oct 31 01:25:44.973511 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:25:44.973453    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/766d96dd-2ba9-4216-9388-7a29c6aaf42c-home\" (UniqueName: \"kubernetes.io/empty-dir/766d96dd-2ba9-4216-9388-7a29c6aaf42c-home\") pod \"766d96dd-2ba9-4216-9388-7a29c6aaf42c\" (UID: \"766d96dd-2ba9-4216-9388-7a29c6aaf42c\") "
Oct 31 01:25:44.973562 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:25:44.973525    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/766d96dd-2ba9-4216-9388-7a29c6aaf42c-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/766d96dd-2ba9-4216-9388-7a29c6aaf42c-model-cache\") pod \"766d96dd-2ba9-4216-9388-7a29c6aaf42c\" (UID: \"766d96dd-2ba9-4216-9388-7a29c6aaf42c\") "
Oct 31 01:25:44.973614 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:25:44.973582    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/766d96dd-2ba9-4216-9388-7a29c6aaf42c-dshm\" (UniqueName: \"kubernetes.io/empty-dir/766d96dd-2ba9-4216-9388-7a29c6aaf42c-dshm\") pod \"766d96dd-2ba9-4216-9388-7a29c6aaf42c\" (UID: \"766d96dd-2ba9-4216-9388-7a29c6aaf42c\") "
Oct 31 01:25:44.973912 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:25:44.973850    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/766d96dd-2ba9-4216-9388-7a29c6aaf42c-tmp-dir" pod "766d96dd-2ba9-4216-9388-7a29c6aaf42c" (UID: "766d96dd-2ba9-4216-9388-7a29c6aaf42c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:25:44.973912 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:25:44.973857    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/766d96dd-2ba9-4216-9388-7a29c6aaf42c-home" pod "766d96dd-2ba9-4216-9388-7a29c6aaf42c" (UID: "766d96dd-2ba9-4216-9388-7a29c6aaf42c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:25:44.973912 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:25:44.973895    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/766d96dd-2ba9-4216-9388-7a29c6aaf42c-model-cache" pod "766d96dd-2ba9-4216-9388-7a29c6aaf42c" (UID: "766d96dd-2ba9-4216-9388-7a29c6aaf42c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:25:44.974251 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:25:44.974234    4964 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/766d96dd-2ba9-4216-9388-7a29c6aaf42c-tmp-dir\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:25:44.974251 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:25:44.974252    4964 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/766d96dd-2ba9-4216-9388-7a29c6aaf42c-home\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:25:44.974373 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:25:44.974261    4964 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/766d96dd-2ba9-4216-9388-7a29c6aaf42c-model-cache\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:25:44.976115 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:25:44.976092    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/766d96dd-2ba9-4216-9388-7a29c6aaf42c-dshm" pod "766d96dd-2ba9-4216-9388-7a29c6aaf42c" (UID: "766d96dd-2ba9-4216-9388-7a29c6aaf42c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:25:44.976240 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:25:44.976174    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/766d96dd-2ba9-4216-9388-7a29c6aaf42c-kube-api-access-vsbkz" pod "766d96dd-2ba9-4216-9388-7a29c6aaf42c" (UID: "766d96dd-2ba9-4216-9388-7a29c6aaf42c"). InnerVolumeSpecName "kube-api-access-vsbkz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:25:44.977113 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:25:44.977088    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/766d96dd-2ba9-4216-9388-7a29c6aaf42c-tls-certs" pod "766d96dd-2ba9-4216-9388-7a29c6aaf42c" (UID: "766d96dd-2ba9-4216-9388-7a29c6aaf42c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:25:45.075962 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:25:45.075929    4964 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/766d96dd-2ba9-4216-9388-7a29c6aaf42c-tls-certs\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:25:45.075962 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:25:45.075956    4964 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vsbkz\" (UniqueName: \"kubernetes.io/projected/766d96dd-2ba9-4216-9388-7a29c6aaf42c-kube-api-access-vsbkz\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:25:45.075962 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:25:45.075966    4964 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/766d96dd-2ba9-4216-9388-7a29c6aaf42c-dshm\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:25:45.390037 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:25:45.390005    4964 generic.go:356] "Generic (PLEG): container finished" podID="766d96dd-2ba9-4216-9388-7a29c6aaf42c" containerID="6bf86c13591c3c1f4557f5340eb804a56d5aa4e2055fe659cbfb4a52450faeb9" exitCode=0
Oct 31 01:25:45.390241 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:25:45.390053    4964 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-8f6b5b/e2e-conflict-a-8bd02cd2-kserve-7c55c85bd6-rwlfr"
Oct 31 01:25:45.390241 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:25:45.390091    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-8f6b5b/e2e-conflict-a-8bd02cd2-kserve-7c55c85bd6-rwlfr" event={"ID":"766d96dd-2ba9-4216-9388-7a29c6aaf42c","Type":"ContainerDied","Data":"6bf86c13591c3c1f4557f5340eb804a56d5aa4e2055fe659cbfb4a52450faeb9"}
Oct 31 01:25:45.390241 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:25:45.390134    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-8f6b5b/e2e-conflict-a-8bd02cd2-kserve-7c55c85bd6-rwlfr" event={"ID":"766d96dd-2ba9-4216-9388-7a29c6aaf42c","Type":"ContainerDied","Data":"9aae13c0c963abccb67f26a39bf482885a8470ac1d3d37eb8fe2da1e4aa504cb"}
Oct 31 01:25:45.390241 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:25:45.390156    4964 scope.go:122] "RemoveContainer" containerID="6bf86c13591c3c1f4557f5340eb804a56d5aa4e2055fe659cbfb4a52450faeb9"
Oct 31 01:25:45.405928 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:25:45.405907    4964 scope.go:122] "RemoveContainer" containerID="6bf86c13591c3c1f4557f5340eb804a56d5aa4e2055fe659cbfb4a52450faeb9"
Oct 31 01:25:45.406226 ip-10-0-133-197 kubenswrapper[4964]: E1031 01:25:45.406199    4964 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6bf86c13591c3c1f4557f5340eb804a56d5aa4e2055fe659cbfb4a52450faeb9\": container with ID starting with 6bf86c13591c3c1f4557f5340eb804a56d5aa4e2055fe659cbfb4a52450faeb9 not found: ID does not exist" containerID="6bf86c13591c3c1f4557f5340eb804a56d5aa4e2055fe659cbfb4a52450faeb9"
Oct 31 01:25:45.406353 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:25:45.406235    4964 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6bf86c13591c3c1f4557f5340eb804a56d5aa4e2055fe659cbfb4a52450faeb9"} err="failed to get container status \"6bf86c13591c3c1f4557f5340eb804a56d5aa4e2055fe659cbfb4a52450faeb9\": rpc error: code = NotFound desc = could not find container \"6bf86c13591c3c1f4557f5340eb804a56d5aa4e2055fe659cbfb4a52450faeb9\": container with ID starting with 6bf86c13591c3c1f4557f5340eb804a56d5aa4e2055fe659cbfb4a52450faeb9 not found: ID does not exist"
Oct 31 01:25:45.562964 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:25:45.562928    4964 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-8f6b5b/e2e-conflict-a-8bd02cd2-kserve-7c55c85bd6-rwlfr"]
Oct 31 01:25:45.606213 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:25:45.606182    4964 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-8f6b5b/e2e-conflict-a-8bd02cd2-kserve-7c55c85bd6-rwlfr"]
Oct 31 01:25:45.777003 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:25:45.776973    4964 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="766d96dd-2ba9-4216-9388-7a29c6aaf42c" path="/var/lib/kubelet/pods/766d96dd-2ba9-4216-9388-7a29c6aaf42c/volumes"
Oct 31 01:25:46.043810 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:25:46.043734    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:25:46.242547 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:25:46.242498    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:25:46.647775 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:25:46.647738    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:25:49.519642 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:25:49.519609    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:25:51.126953 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:25:51.126914    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:25:51.796199 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:25:51.796161    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:25:55.519529 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:25:55.519440    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:25:55.596346 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:25:55.588259    4964 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a8e157/distinct-2-w3-a8e157-kserve-9ddd45d6-ng7j7" podUID="bc7a71df-c523-4590-91df-cf8fdf3ed19e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:25:55.706509 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:25:55.706469    4964 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-baacee/premium-w1-baacee-kserve-69cc8c6456-7ngq9" podUID="15028ba0-c888-4780-b770-6acb2d03d70a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:25:55.850551 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:25:55.850450    4964 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-baacee/distinct-w1-baacee-kserve-6c85cfb7d7-stkqq" podUID="df6d0bdd-382d-49b7-879c-15cb5371246e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:25:55.980026 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:25:55.979975    4964 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-00e840/premium-w2-00e840-kserve-6f6d876787-cv7w8" podUID="b8c12d5f-8ada-4c53-880c-446f5f0b5998" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:25:56.500302 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:25:56.500262    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:26:01.426181 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:26:01.426134    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:26:06.343813 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:26:06.343776    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:26:07.578566 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:26:07.578518    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:26:09.516478 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:26:09.516410    4964 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-klw8t container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 01:26:09.517173 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:26:09.517136    4964 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-klw8t" podUID="c15fa68c-09a2-453a-81c1-687a3ce2b87a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:26:09.595439 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:26:09.595399    4964 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-b4hfr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 01:26:09.595738 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:26:09.595708    4964 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-b4hfr" podUID="a8754d1f-9656-460d-879e-c8d3f917492e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:26:10.488009 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:26:10.487974    4964 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-00e840/distinct-2-w2-00e840-kserve-6564ddcb89-cftvc"]
Oct 31 01:26:10.488640 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:26:10.488626    4964 cpu_manager.go:414] "containerMap: removing container" podUID="766d96dd-2ba9-4216-9388-7a29c6aaf42c" containerName="main"
Oct 31 01:26:10.488690 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:26:10.488642    4964 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="766d96dd-2ba9-4216-9388-7a29c6aaf42c" containerName="main"
Oct 31 01:26:10.488766 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:26:10.488756    4964 memory_manager.go:365] "RemoveStaleState removing state" podUID="766d96dd-2ba9-4216-9388-7a29c6aaf42c" containerName="main"
Oct 31 01:26:10.492303 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:26:10.492278    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-00e840/distinct-2-w2-00e840-kserve-6564ddcb89-cftvc"
Oct 31 01:26:10.592466 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:26:10.592433    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/facb04d4-8cd2-48aa-9b75-f165d7ca3f4a-model-cache\") pod \"distinct-2-w2-00e840-kserve-6564ddcb89-cftvc\" (UID: \"facb04d4-8cd2-48aa-9b75-f165d7ca3f4a\") " pod="e2e-models-e2e-worker-w2-00e840/distinct-2-w2-00e840-kserve-6564ddcb89-cftvc"
Oct 31 01:26:10.592953 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:26:10.592571    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/facb04d4-8cd2-48aa-9b75-f165d7ca3f4a-home\") pod \"distinct-2-w2-00e840-kserve-6564ddcb89-cftvc\" (UID: \"facb04d4-8cd2-48aa-9b75-f165d7ca3f4a\") " pod="e2e-models-e2e-worker-w2-00e840/distinct-2-w2-00e840-kserve-6564ddcb89-cftvc"
Oct 31 01:26:10.592953 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:26:10.592611    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/facb04d4-8cd2-48aa-9b75-f165d7ca3f4a-dshm\") pod \"distinct-2-w2-00e840-kserve-6564ddcb89-cftvc\" (UID: \"facb04d4-8cd2-48aa-9b75-f165d7ca3f4a\") " pod="e2e-models-e2e-worker-w2-00e840/distinct-2-w2-00e840-kserve-6564ddcb89-cftvc"
Oct 31 01:26:10.592953 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:26:10.592655    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/facb04d4-8cd2-48aa-9b75-f165d7ca3f4a-tls-certs\") pod \"distinct-2-w2-00e840-kserve-6564ddcb89-cftvc\" (UID: \"facb04d4-8cd2-48aa-9b75-f165d7ca3f4a\") " pod="e2e-models-e2e-worker-w2-00e840/distinct-2-w2-00e840-kserve-6564ddcb89-cftvc"
Oct 31 01:26:10.592953 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:26:10.592783    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-27dw2\" (UniqueName: \"kubernetes.io/projected/facb04d4-8cd2-48aa-9b75-f165d7ca3f4a-kube-api-access-27dw2\") pod \"distinct-2-w2-00e840-kserve-6564ddcb89-cftvc\" (UID: \"facb04d4-8cd2-48aa-9b75-f165d7ca3f4a\") " pod="e2e-models-e2e-worker-w2-00e840/distinct-2-w2-00e840-kserve-6564ddcb89-cftvc"
Oct 31 01:26:10.592953 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:26:10.592820    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/facb04d4-8cd2-48aa-9b75-f165d7ca3f4a-tmp-dir\") pod \"distinct-2-w2-00e840-kserve-6564ddcb89-cftvc\" (UID: \"facb04d4-8cd2-48aa-9b75-f165d7ca3f4a\") " pod="e2e-models-e2e-worker-w2-00e840/distinct-2-w2-00e840-kserve-6564ddcb89-cftvc"
Oct 31 01:26:10.593722 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:26:10.593691    4964 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-00e840\"/\"distinct-2-w2-00e840-kserve-self-signed-certs\""
Oct 31 01:26:10.597273 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:26:10.597250    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-00e840/distinct-2-w2-00e840-kserve-6564ddcb89-cftvc"]
Oct 31 01:26:10.618841 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:26:10.618812    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:26:10.694420 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:26:10.694363    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/facb04d4-8cd2-48aa-9b75-f165d7ca3f4a-dshm\") pod \"distinct-2-w2-00e840-kserve-6564ddcb89-cftvc\" (UID: \"facb04d4-8cd2-48aa-9b75-f165d7ca3f4a\") " pod="e2e-models-e2e-worker-w2-00e840/distinct-2-w2-00e840-kserve-6564ddcb89-cftvc"
Oct 31 01:26:10.694593 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:26:10.694446    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/facb04d4-8cd2-48aa-9b75-f165d7ca3f4a-tls-certs\") pod \"distinct-2-w2-00e840-kserve-6564ddcb89-cftvc\" (UID: \"facb04d4-8cd2-48aa-9b75-f165d7ca3f4a\") " pod="e2e-models-e2e-worker-w2-00e840/distinct-2-w2-00e840-kserve-6564ddcb89-cftvc"
Oct 31 01:26:10.694677 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:26:10.694594    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-27dw2\" (UniqueName: \"kubernetes.io/projected/facb04d4-8cd2-48aa-9b75-f165d7ca3f4a-kube-api-access-27dw2\") pod \"distinct-2-w2-00e840-kserve-6564ddcb89-cftvc\" (UID: \"facb04d4-8cd2-48aa-9b75-f165d7ca3f4a\") " pod="e2e-models-e2e-worker-w2-00e840/distinct-2-w2-00e840-kserve-6564ddcb89-cftvc"
Oct 31 01:26:10.694677 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:26:10.694646    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/facb04d4-8cd2-48aa-9b75-f165d7ca3f4a-tmp-dir\") pod \"distinct-2-w2-00e840-kserve-6564ddcb89-cftvc\" (UID: \"facb04d4-8cd2-48aa-9b75-f165d7ca3f4a\") " pod="e2e-models-e2e-worker-w2-00e840/distinct-2-w2-00e840-kserve-6564ddcb89-cftvc"
Oct 31 01:26:10.694790 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:26:10.694699    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/facb04d4-8cd2-48aa-9b75-f165d7ca3f4a-model-cache\") pod \"distinct-2-w2-00e840-kserve-6564ddcb89-cftvc\" (UID: \"facb04d4-8cd2-48aa-9b75-f165d7ca3f4a\") " pod="e2e-models-e2e-worker-w2-00e840/distinct-2-w2-00e840-kserve-6564ddcb89-cftvc"
Oct 31 01:26:10.694845 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:26:10.694812    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/facb04d4-8cd2-48aa-9b75-f165d7ca3f4a-home\") pod \"distinct-2-w2-00e840-kserve-6564ddcb89-cftvc\" (UID: \"facb04d4-8cd2-48aa-9b75-f165d7ca3f4a\") " pod="e2e-models-e2e-worker-w2-00e840/distinct-2-w2-00e840-kserve-6564ddcb89-cftvc"
Oct 31 01:26:10.695346 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:26:10.695302    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/facb04d4-8cd2-48aa-9b75-f165d7ca3f4a-home\") pod \"distinct-2-w2-00e840-kserve-6564ddcb89-cftvc\" (UID: \"facb04d4-8cd2-48aa-9b75-f165d7ca3f4a\") " pod="e2e-models-e2e-worker-w2-00e840/distinct-2-w2-00e840-kserve-6564ddcb89-cftvc"
Oct 31 01:26:10.695796 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:26:10.695773    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/facb04d4-8cd2-48aa-9b75-f165d7ca3f4a-tmp-dir\") pod \"distinct-2-w2-00e840-kserve-6564ddcb89-cftvc\" (UID: \"facb04d4-8cd2-48aa-9b75-f165d7ca3f4a\") " pod="e2e-models-e2e-worker-w2-00e840/distinct-2-w2-00e840-kserve-6564ddcb89-cftvc"
Oct 31 01:26:10.695963 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:26:10.695936    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/facb04d4-8cd2-48aa-9b75-f165d7ca3f4a-model-cache\") pod \"distinct-2-w2-00e840-kserve-6564ddcb89-cftvc\" (UID: \"facb04d4-8cd2-48aa-9b75-f165d7ca3f4a\") " pod="e2e-models-e2e-worker-w2-00e840/distinct-2-w2-00e840-kserve-6564ddcb89-cftvc"
Oct 31 01:26:10.699888 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:26:10.699864    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/facb04d4-8cd2-48aa-9b75-f165d7ca3f4a-dshm\") pod \"distinct-2-w2-00e840-kserve-6564ddcb89-cftvc\" (UID: \"facb04d4-8cd2-48aa-9b75-f165d7ca3f4a\") " pod="e2e-models-e2e-worker-w2-00e840/distinct-2-w2-00e840-kserve-6564ddcb89-cftvc"
Oct 31 01:26:10.700533 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:26:10.700511    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/facb04d4-8cd2-48aa-9b75-f165d7ca3f4a-tls-certs\") pod \"distinct-2-w2-00e840-kserve-6564ddcb89-cftvc\" (UID: \"facb04d4-8cd2-48aa-9b75-f165d7ca3f4a\") " pod="e2e-models-e2e-worker-w2-00e840/distinct-2-w2-00e840-kserve-6564ddcb89-cftvc"
Oct 31 01:26:10.751841 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:26:10.751760    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-27dw2\" (UniqueName: \"kubernetes.io/projected/facb04d4-8cd2-48aa-9b75-f165d7ca3f4a-kube-api-access-27dw2\") pod \"distinct-2-w2-00e840-kserve-6564ddcb89-cftvc\" (UID: \"facb04d4-8cd2-48aa-9b75-f165d7ca3f4a\") " pod="e2e-models-e2e-worker-w2-00e840/distinct-2-w2-00e840-kserve-6564ddcb89-cftvc"
Oct 31 01:26:10.804433 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:26:10.804404    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-00e840/distinct-2-w2-00e840-kserve-6564ddcb89-cftvc"
Oct 31 01:26:11.182552 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:26:11.182523    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-00e840/distinct-2-w2-00e840-kserve-6564ddcb89-cftvc"]
Oct 31 01:26:11.183467 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:26:11.183442    4964 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfacb04d4_8cd2_48aa_9b75_f165d7ca3f4a.slice/crio-49f8c07fbd634fda9c2e1383f71442efb0d42a58f6a8823ae793c5330bd217ca WatchSource:0}: Error finding container 49f8c07fbd634fda9c2e1383f71442efb0d42a58f6a8823ae793c5330bd217ca: Status 404 returned error can't find the container with id 49f8c07fbd634fda9c2e1383f71442efb0d42a58f6a8823ae793c5330bd217ca
Oct 31 01:26:11.201672 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:26:11.199694    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:26:11.247334 ip-10-0-133-197 kubenswrapper[4964]: E1031 01:26:11.247278    4964 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q56x5" containerName="extractor"
Oct 31 01:26:11.708112 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:26:11.708081    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-00e840/distinct-2-w2-00e840-kserve-6564ddcb89-cftvc" event={"ID":"facb04d4-8cd2-48aa-9b75-f165d7ca3f4a","Type":"ContainerStarted","Data":"f3adde0f6d194273609102065a86275d9f95736af59ae08a21ff2f80280da2ba"}
Oct 31 01:26:11.708612 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:26:11.708126    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-00e840/distinct-2-w2-00e840-kserve-6564ddcb89-cftvc" event={"ID":"facb04d4-8cd2-48aa-9b75-f165d7ca3f4a","Type":"ContainerStarted","Data":"49f8c07fbd634fda9c2e1383f71442efb0d42a58f6a8823ae793c5330bd217ca"}
Oct 31 01:26:11.805274 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:26:11.805246    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-00e840/distinct-2-w2-00e840-kserve-6564ddcb89-cftvc"
Oct 31 01:26:11.824794 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:26:11.824748    4964 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-00e840/distinct-2-w2-00e840-kserve-6564ddcb89-cftvc" podStartSLOduration=1.824734747 podStartE2EDuration="1.824734747s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:26:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:26:11.822703022 +0000 UTC m=+1340.299269403" watchObservedRunningTime="2026-10-31 01:26:11.824734747 +0000 UTC m=+1340.301301126"
Oct 31 01:26:16.036384 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:26:16.036348    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:26:20.805390 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:26:20.805356    4964 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-00e840/distinct-2-w2-00e840-kserve-6564ddcb89-cftvc"
Oct 31 01:26:20.817865 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:26:20.817844    4964 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-00e840/distinct-2-w2-00e840-kserve-6564ddcb89-cftvc"
Oct 31 01:26:21.241582 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:26:21.241550    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:26:21.488713 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:26:21.488678    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:26:21.789008 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:26:21.788982    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-00e840/distinct-2-w2-00e840-kserve-6564ddcb89-cftvc"
Oct 31 01:26:26.023999 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:26:26.023960    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:26:27.780045 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:26:27.780010    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:26:30.792205 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:26:30.792166    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:26:35.517151 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:26:35.517115    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:26:37.943945 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:26:37.943881    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:26:40.491823 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:26:40.491785    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:26:41.238142 ip-10-0-133-197 kubenswrapper[4964]: E1031 01:26:41.238107    4964 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q56x5" containerName="extractor"
Oct 31 01:26:45.292166 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:26:45.292131    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:26:46.974993 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:26:46.974955    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:26:50.121386 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:26:50.121309    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:26:54.896834 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:26:54.896795    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:27:01.624824 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:27:01.624736    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:27:03.495700 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:27:03.495660    4964 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-tt9c2"]
Oct 31 01:27:03.500036 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:27:03.500018    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-tt9c2"
Oct 31 01:27:03.559131 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:27:03.559089    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2a53cf91-69f9-4c47-8271-79736ce35af9-istiod-ca-cert\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-tt9c2\" (UID: \"2a53cf91-69f9-4c47-8271-79736ce35af9\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-tt9c2"
Oct 31 01:27:03.559283 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:27:03.559135    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mqg5w\" (UniqueName: \"kubernetes.io/projected/2a53cf91-69f9-4c47-8271-79736ce35af9-kube-api-access-mqg5w\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-tt9c2\" (UID: \"2a53cf91-69f9-4c47-8271-79736ce35af9\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-tt9c2"
Oct 31 01:27:03.559283 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:27:03.559257    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2a53cf91-69f9-4c47-8271-79736ce35af9-istio-envoy\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-tt9c2\" (UID: \"2a53cf91-69f9-4c47-8271-79736ce35af9\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-tt9c2"
Oct 31 01:27:03.559416 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:27:03.559306    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2a53cf91-69f9-4c47-8271-79736ce35af9-istio-data\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-tt9c2\" (UID: \"2a53cf91-69f9-4c47-8271-79736ce35af9\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-tt9c2"
Oct 31 01:27:03.559463 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:27:03.559433    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2a53cf91-69f9-4c47-8271-79736ce35af9-istio-token\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-tt9c2\" (UID: \"2a53cf91-69f9-4c47-8271-79736ce35af9\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-tt9c2"
Oct 31 01:27:03.559508 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:27:03.559478    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2a53cf91-69f9-4c47-8271-79736ce35af9-istio-podinfo\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-tt9c2\" (UID: \"2a53cf91-69f9-4c47-8271-79736ce35af9\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-tt9c2"
Oct 31 01:27:03.559571 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:27:03.559557    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a53cf91-69f9-4c47-8271-79736ce35af9-credential-socket\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-tt9c2\" (UID: \"2a53cf91-69f9-4c47-8271-79736ce35af9\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-tt9c2"
Oct 31 01:27:03.559612 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:27:03.559600    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a53cf91-69f9-4c47-8271-79736ce35af9-workload-socket\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-tt9c2\" (UID: \"2a53cf91-69f9-4c47-8271-79736ce35af9\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-tt9c2"
Oct 31 01:27:03.559669 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:27:03.559653    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2a53cf91-69f9-4c47-8271-79736ce35af9-workload-certs\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-tt9c2\" (UID: \"2a53cf91-69f9-4c47-8271-79736ce35af9\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-tt9c2"
Oct 31 01:27:03.605902 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:27:03.605875    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-tt9c2"]
Oct 31 01:27:03.661165 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:27:03.661128    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2a53cf91-69f9-4c47-8271-79736ce35af9-istiod-ca-cert\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-tt9c2\" (UID: \"2a53cf91-69f9-4c47-8271-79736ce35af9\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-tt9c2"
Oct 31 01:27:03.661165 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:27:03.661171    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mqg5w\" (UniqueName: \"kubernetes.io/projected/2a53cf91-69f9-4c47-8271-79736ce35af9-kube-api-access-mqg5w\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-tt9c2\" (UID: \"2a53cf91-69f9-4c47-8271-79736ce35af9\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-tt9c2"
Oct 31 01:27:03.661455 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:27:03.661208    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2a53cf91-69f9-4c47-8271-79736ce35af9-istio-envoy\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-tt9c2\" (UID: \"2a53cf91-69f9-4c47-8271-79736ce35af9\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-tt9c2"
Oct 31 01:27:03.661455 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:27:03.661245    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2a53cf91-69f9-4c47-8271-79736ce35af9-istio-data\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-tt9c2\" (UID: \"2a53cf91-69f9-4c47-8271-79736ce35af9\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-tt9c2"
Oct 31 01:27:03.661455 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:27:03.661311    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2a53cf91-69f9-4c47-8271-79736ce35af9-istio-token\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-tt9c2\" (UID: \"2a53cf91-69f9-4c47-8271-79736ce35af9\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-tt9c2"
Oct 31 01:27:03.661455 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:27:03.661391    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2a53cf91-69f9-4c47-8271-79736ce35af9-istio-podinfo\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-tt9c2\" (UID: \"2a53cf91-69f9-4c47-8271-79736ce35af9\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-tt9c2"
Oct 31 01:27:03.661680 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:27:03.661459    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a53cf91-69f9-4c47-8271-79736ce35af9-credential-socket\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-tt9c2\" (UID: \"2a53cf91-69f9-4c47-8271-79736ce35af9\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-tt9c2"
Oct 31 01:27:03.661680 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:27:03.661504    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a53cf91-69f9-4c47-8271-79736ce35af9-workload-socket\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-tt9c2\" (UID: \"2a53cf91-69f9-4c47-8271-79736ce35af9\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-tt9c2"
Oct 31 01:27:03.661680 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:27:03.661584    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2a53cf91-69f9-4c47-8271-79736ce35af9-workload-certs\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-tt9c2\" (UID: \"2a53cf91-69f9-4c47-8271-79736ce35af9\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-tt9c2"
Oct 31 01:27:03.661831 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:27:03.661803    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2a53cf91-69f9-4c47-8271-79736ce35af9-istio-data\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-tt9c2\" (UID: \"2a53cf91-69f9-4c47-8271-79736ce35af9\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-tt9c2"
Oct 31 01:27:03.661958 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:27:03.661932    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a53cf91-69f9-4c47-8271-79736ce35af9-credential-socket\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-tt9c2\" (UID: \"2a53cf91-69f9-4c47-8271-79736ce35af9\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-tt9c2"
Oct 31 01:27:03.662018 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:27:03.661998    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a53cf91-69f9-4c47-8271-79736ce35af9-workload-socket\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-tt9c2\" (UID: \"2a53cf91-69f9-4c47-8271-79736ce35af9\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-tt9c2"
Oct 31 01:27:03.662257 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:27:03.662234    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2a53cf91-69f9-4c47-8271-79736ce35af9-workload-certs\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-tt9c2\" (UID: \"2a53cf91-69f9-4c47-8271-79736ce35af9\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-tt9c2"
Oct 31 01:27:03.662420 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:27:03.662279    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2a53cf91-69f9-4c47-8271-79736ce35af9-istiod-ca-cert\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-tt9c2\" (UID: \"2a53cf91-69f9-4c47-8271-79736ce35af9\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-tt9c2"
Oct 31 01:27:03.664103 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:27:03.664072    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2a53cf91-69f9-4c47-8271-79736ce35af9-istio-podinfo\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-tt9c2\" (UID: \"2a53cf91-69f9-4c47-8271-79736ce35af9\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-tt9c2"
Oct 31 01:27:03.664219 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:27:03.664147    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2a53cf91-69f9-4c47-8271-79736ce35af9-istio-envoy\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-tt9c2\" (UID: \"2a53cf91-69f9-4c47-8271-79736ce35af9\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-tt9c2"
Oct 31 01:27:03.720566 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:27:03.720533    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2a53cf91-69f9-4c47-8271-79736ce35af9-istio-token\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-tt9c2\" (UID: \"2a53cf91-69f9-4c47-8271-79736ce35af9\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-tt9c2"
Oct 31 01:27:03.720896 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:27:03.720873    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mqg5w\" (UniqueName: \"kubernetes.io/projected/2a53cf91-69f9-4c47-8271-79736ce35af9-kube-api-access-mqg5w\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-tt9c2\" (UID: \"2a53cf91-69f9-4c47-8271-79736ce35af9\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-tt9c2"
Oct 31 01:27:03.826205 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:27:03.826124    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-tt9c2"
Oct 31 01:27:04.262004 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:27:04.261981    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-tt9c2"]
Oct 31 01:27:04.263463 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:27:04.263441    4964 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a53cf91_69f9_4c47_8271_79736ce35af9.slice/crio-2ceab8619f6f8e164630736e01b9e2348332566d903aa4d45fb00af7acc1ab90 WatchSource:0}: Error finding container 2ceab8619f6f8e164630736e01b9e2348332566d903aa4d45fb00af7acc1ab90: Status 404 returned error can't find the container with id 2ceab8619f6f8e164630736e01b9e2348332566d903aa4d45fb00af7acc1ab90
Oct 31 01:27:04.265914 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:27:04.265886    4964 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 01:27:04.266014 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:27:04.265944    4964 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 01:27:04.266014 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:27:04.265973    4964 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 01:27:04.412369 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:27:04.412307    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:27:05.137854 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:27:05.137819    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-tt9c2" event={"ID":"2a53cf91-69f9-4c47-8271-79736ce35af9","Type":"ContainerStarted","Data":"d43ea5f023a2f0d664642c01246e01a25f303a5ef1d04b70f6c9e573d24c1320"}
Oct 31 01:27:05.138277 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:27:05.137870    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-tt9c2" event={"ID":"2a53cf91-69f9-4c47-8271-79736ce35af9","Type":"ContainerStarted","Data":"2ceab8619f6f8e164630736e01b9e2348332566d903aa4d45fb00af7acc1ab90"}
Oct 31 01:27:05.827297 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:27:05.827260    4964 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-tt9c2"
Oct 31 01:27:05.829176 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:27:05.829139    4964 patch_prober.go:29] interesting pod/e2e-worker-w2-00e840-openshift-default-59b96fb879-tt9c2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body=
Oct 31 01:27:05.829311 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:27:05.829191    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-tt9c2" podUID="2a53cf91-69f9-4c47-8271-79736ce35af9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused"
Oct 31 01:27:06.827037 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:27:06.827003    4964 patch_prober.go:29] interesting pod/e2e-worker-w2-00e840-openshift-default-59b96fb879-tt9c2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body=
Oct 31 01:27:06.827524 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:27:06.827063    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-tt9c2" podUID="2a53cf91-69f9-4c47-8271-79736ce35af9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused"
Oct 31 01:27:07.826506 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:27:07.826469    4964 patch_prober.go:29] interesting pod/e2e-worker-w2-00e840-openshift-default-59b96fb879-tt9c2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body=
Oct 31 01:27:07.826693 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:27:07.826523    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-tt9c2" podUID="2a53cf91-69f9-4c47-8271-79736ce35af9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused"
Oct 31 01:27:07.922106 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:27:07.922056    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:27:08.583194 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:27:08.583152    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:27:08.827098 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:27:08.827061    4964 patch_prober.go:29] interesting pod/e2e-worker-w2-00e840-openshift-default-59b96fb879-tt9c2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body=
Oct 31 01:27:08.827279 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:27:08.827135    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-tt9c2" podUID="2a53cf91-69f9-4c47-8271-79736ce35af9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused"
Oct 31 01:27:09.227140 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:27:09.227101    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:27:09.827724 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:27:09.827687    4964 patch_prober.go:29] interesting pod/e2e-worker-w2-00e840-openshift-default-59b96fb879-tt9c2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body=
Oct 31 01:27:09.828027 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:27:09.827989    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-tt9c2" podUID="2a53cf91-69f9-4c47-8271-79736ce35af9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused"
Oct 31 01:27:10.826941 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:27:10.826908    4964 patch_prober.go:29] interesting pod/e2e-worker-w2-00e840-openshift-default-59b96fb879-tt9c2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body=
Oct 31 01:27:10.827427 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:27:10.826975    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-tt9c2" podUID="2a53cf91-69f9-4c47-8271-79736ce35af9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused"
Oct 31 01:27:11.241904 ip-10-0-133-197 kubenswrapper[4964]: E1031 01:27:11.241869    4964 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q56x5" containerName="extractor"
Oct 31 01:27:12.833651 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:27:12.833615    4964 patch_prober.go:29] interesting pod/e2e-worker-w2-00e840-openshift-default-59b96fb879-tt9c2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 01:27:12.834141 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:27:12.833775    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-tt9c2" podUID="2a53cf91-69f9-4c47-8271-79736ce35af9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:27:12.851267 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:27:12.851239    4964 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-tt9c2"
Oct 31 01:27:12.851773 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:27:12.851754    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-tt9c2"
Oct 31 01:27:12.852428 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:27:12.852412    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-tt9c2"
Oct 31 01:27:13.011422 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:27:13.011363    4964 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-tt9c2" podStartSLOduration=10.011344476 podStartE2EDuration="10.011344476s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:27:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:27:05.321656012 +0000 UTC m=+1393.798222393" watchObservedRunningTime="2026-10-31 01:27:13.011344476 +0000 UTC m=+1401.487910856"
Oct 31 01:27:14.109041 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:27:14.109010    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:27:16.520494 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:27:16.520456    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:27:17.034995 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:27:17.034963    4964 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-7cec02-openshift-default-6fc456b49-jn9wm"]
Oct 31 01:27:17.039486 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:27:17.039469    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-7cec02-openshift-default-6fc456b49-jn9wm"
Oct 31 01:27:17.113166 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:27:17.113125    4964 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-7cec02-openshift-default-dockercfg-w8vll\""
Oct 31 01:27:17.175623 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:27:17.175573    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-7cec02-openshift-default-6fc456b49-jn9wm"]
Oct 31 01:27:17.194095 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:27:17.194061    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/45450e7a-2b62-4417-acca-d370251f6d54-istio-data\") pod \"e2e-api-a-7cec02-openshift-default-6fc456b49-jn9wm\" (UID: \"45450e7a-2b62-4417-acca-d370251f6d54\") " pod="openshift-ingress/e2e-api-a-7cec02-openshift-default-6fc456b49-jn9wm"
Oct 31 01:27:17.194358 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:27:17.194119    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/45450e7a-2b62-4417-acca-d370251f6d54-istio-token\") pod \"e2e-api-a-7cec02-openshift-default-6fc456b49-jn9wm\" (UID: \"45450e7a-2b62-4417-acca-d370251f6d54\") " pod="openshift-ingress/e2e-api-a-7cec02-openshift-default-6fc456b49-jn9wm"
Oct 31 01:27:17.194358 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:27:17.194170    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/45450e7a-2b62-4417-acca-d370251f6d54-workload-certs\") pod \"e2e-api-a-7cec02-openshift-default-6fc456b49-jn9wm\" (UID: \"45450e7a-2b62-4417-acca-d370251f6d54\") " pod="openshift-ingress/e2e-api-a-7cec02-openshift-default-6fc456b49-jn9wm"
Oct 31 01:27:17.194358 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:27:17.194263    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/45450e7a-2b62-4417-acca-d370251f6d54-credential-socket\") pod \"e2e-api-a-7cec02-openshift-default-6fc456b49-jn9wm\" (UID: \"45450e7a-2b62-4417-acca-d370251f6d54\") " pod="openshift-ingress/e2e-api-a-7cec02-openshift-default-6fc456b49-jn9wm"
Oct 31 01:27:17.194563 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:27:17.194380    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/45450e7a-2b62-4417-acca-d370251f6d54-istiod-ca-cert\") pod \"e2e-api-a-7cec02-openshift-default-6fc456b49-jn9wm\" (UID: \"45450e7a-2b62-4417-acca-d370251f6d54\") " pod="openshift-ingress/e2e-api-a-7cec02-openshift-default-6fc456b49-jn9wm"
Oct 31 01:27:17.194563 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:27:17.194407    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/45450e7a-2b62-4417-acca-d370251f6d54-workload-socket\") pod \"e2e-api-a-7cec02-openshift-default-6fc456b49-jn9wm\" (UID: \"45450e7a-2b62-4417-acca-d370251f6d54\") " pod="openshift-ingress/e2e-api-a-7cec02-openshift-default-6fc456b49-jn9wm"
Oct 31 01:27:17.194563 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:27:17.194430    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lpn8j\" (UniqueName: \"kubernetes.io/projected/45450e7a-2b62-4417-acca-d370251f6d54-kube-api-access-lpn8j\") pod \"e2e-api-a-7cec02-openshift-default-6fc456b49-jn9wm\" (UID: \"45450e7a-2b62-4417-acca-d370251f6d54\") " pod="openshift-ingress/e2e-api-a-7cec02-openshift-default-6fc456b49-jn9wm"
Oct 31 01:27:17.194563 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:27:17.194467    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/45450e7a-2b62-4417-acca-d370251f6d54-istio-podinfo\") pod \"e2e-api-a-7cec02-openshift-default-6fc456b49-jn9wm\" (UID: \"45450e7a-2b62-4417-acca-d370251f6d54\") " pod="openshift-ingress/e2e-api-a-7cec02-openshift-default-6fc456b49-jn9wm"
Oct 31 01:27:17.194563 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:27:17.194504    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/45450e7a-2b62-4417-acca-d370251f6d54-istio-envoy\") pod \"e2e-api-a-7cec02-openshift-default-6fc456b49-jn9wm\" (UID: \"45450e7a-2b62-4417-acca-d370251f6d54\") " pod="openshift-ingress/e2e-api-a-7cec02-openshift-default-6fc456b49-jn9wm"
Oct 31 01:27:17.195578 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:27:17.195550    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:27:17.296745 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:27:17.296648    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/45450e7a-2b62-4417-acca-d370251f6d54-istio-envoy\") pod \"e2e-api-a-7cec02-openshift-default-6fc456b49-jn9wm\" (UID: \"45450e7a-2b62-4417-acca-d370251f6d54\") " pod="openshift-ingress/e2e-api-a-7cec02-openshift-default-6fc456b49-jn9wm"
Oct 31 01:27:17.296931 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:27:17.296807    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/45450e7a-2b62-4417-acca-d370251f6d54-istio-data\") pod \"e2e-api-a-7cec02-openshift-default-6fc456b49-jn9wm\" (UID: \"45450e7a-2b62-4417-acca-d370251f6d54\") " pod="openshift-ingress/e2e-api-a-7cec02-openshift-default-6fc456b49-jn9wm"
Oct 31 01:27:17.296931 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:27:17.296848    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/45450e7a-2b62-4417-acca-d370251f6d54-istio-token\") pod \"e2e-api-a-7cec02-openshift-default-6fc456b49-jn9wm\" (UID: \"45450e7a-2b62-4417-acca-d370251f6d54\") " pod="openshift-ingress/e2e-api-a-7cec02-openshift-default-6fc456b49-jn9wm"
Oct 31 01:27:17.296931 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:27:17.296902    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/45450e7a-2b62-4417-acca-d370251f6d54-workload-certs\") pod \"e2e-api-a-7cec02-openshift-default-6fc456b49-jn9wm\" (UID: \"45450e7a-2b62-4417-acca-d370251f6d54\") " pod="openshift-ingress/e2e-api-a-7cec02-openshift-default-6fc456b49-jn9wm"
Oct 31 01:27:17.297133 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:27:17.296999    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/45450e7a-2b62-4417-acca-d370251f6d54-credential-socket\") pod \"e2e-api-a-7cec02-openshift-default-6fc456b49-jn9wm\" (UID: \"45450e7a-2b62-4417-acca-d370251f6d54\") " pod="openshift-ingress/e2e-api-a-7cec02-openshift-default-6fc456b49-jn9wm"
Oct 31 01:27:17.297192 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:27:17.297128    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/45450e7a-2b62-4417-acca-d370251f6d54-istiod-ca-cert\") pod \"e2e-api-a-7cec02-openshift-default-6fc456b49-jn9wm\" (UID: \"45450e7a-2b62-4417-acca-d370251f6d54\") " pod="openshift-ingress/e2e-api-a-7cec02-openshift-default-6fc456b49-jn9wm"
Oct 31 01:27:17.297192 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:27:17.297168    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/45450e7a-2b62-4417-acca-d370251f6d54-workload-socket\") pod \"e2e-api-a-7cec02-openshift-default-6fc456b49-jn9wm\" (UID: \"45450e7a-2b62-4417-acca-d370251f6d54\") " pod="openshift-ingress/e2e-api-a-7cec02-openshift-default-6fc456b49-jn9wm"
Oct 31 01:27:17.297294 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:27:17.297196    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lpn8j\" (UniqueName: \"kubernetes.io/projected/45450e7a-2b62-4417-acca-d370251f6d54-kube-api-access-lpn8j\") pod \"e2e-api-a-7cec02-openshift-default-6fc456b49-jn9wm\" (UID: \"45450e7a-2b62-4417-acca-d370251f6d54\") " pod="openshift-ingress/e2e-api-a-7cec02-openshift-default-6fc456b49-jn9wm"
Oct 31 01:27:17.297294 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:27:17.297234    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/45450e7a-2b62-4417-acca-d370251f6d54-istio-podinfo\") pod \"e2e-api-a-7cec02-openshift-default-6fc456b49-jn9wm\" (UID: \"45450e7a-2b62-4417-acca-d370251f6d54\") " pod="openshift-ingress/e2e-api-a-7cec02-openshift-default-6fc456b49-jn9wm"
Oct 31 01:27:17.298719 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:27:17.298692    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/45450e7a-2b62-4417-acca-d370251f6d54-istiod-ca-cert\") pod \"e2e-api-a-7cec02-openshift-default-6fc456b49-jn9wm\" (UID: \"45450e7a-2b62-4417-acca-d370251f6d54\") " pod="openshift-ingress/e2e-api-a-7cec02-openshift-default-6fc456b49-jn9wm"
Oct 31 01:27:17.299999 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:27:17.299711    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/45450e7a-2b62-4417-acca-d370251f6d54-workload-certs\") pod \"e2e-api-a-7cec02-openshift-default-6fc456b49-jn9wm\" (UID: \"45450e7a-2b62-4417-acca-d370251f6d54\") " pod="openshift-ingress/e2e-api-a-7cec02-openshift-default-6fc456b49-jn9wm"
Oct 31 01:27:17.299999 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:27:17.299959    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/45450e7a-2b62-4417-acca-d370251f6d54-workload-socket\") pod \"e2e-api-a-7cec02-openshift-default-6fc456b49-jn9wm\" (UID: \"45450e7a-2b62-4417-acca-d370251f6d54\") " pod="openshift-ingress/e2e-api-a-7cec02-openshift-default-6fc456b49-jn9wm"
Oct 31 01:27:17.300446 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:27:17.300218    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/45450e7a-2b62-4417-acca-d370251f6d54-istio-data\") pod \"e2e-api-a-7cec02-openshift-default-6fc456b49-jn9wm\" (UID: \"45450e7a-2b62-4417-acca-d370251f6d54\") " pod="openshift-ingress/e2e-api-a-7cec02-openshift-default-6fc456b49-jn9wm"
Oct 31 01:27:17.300596 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:27:17.300356    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/45450e7a-2b62-4417-acca-d370251f6d54-credential-socket\") pod \"e2e-api-a-7cec02-openshift-default-6fc456b49-jn9wm\" (UID: \"45450e7a-2b62-4417-acca-d370251f6d54\") " pod="openshift-ingress/e2e-api-a-7cec02-openshift-default-6fc456b49-jn9wm"
Oct 31 01:27:17.307417 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:27:17.307388    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/45450e7a-2b62-4417-acca-d370251f6d54-istio-envoy\") pod \"e2e-api-a-7cec02-openshift-default-6fc456b49-jn9wm\" (UID: \"45450e7a-2b62-4417-acca-d370251f6d54\") " pod="openshift-ingress/e2e-api-a-7cec02-openshift-default-6fc456b49-jn9wm"
Oct 31 01:27:17.330385 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:27:17.327303    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/45450e7a-2b62-4417-acca-d370251f6d54-istio-podinfo\") pod \"e2e-api-a-7cec02-openshift-default-6fc456b49-jn9wm\" (UID: \"45450e7a-2b62-4417-acca-d370251f6d54\") " pod="openshift-ingress/e2e-api-a-7cec02-openshift-default-6fc456b49-jn9wm"
Oct 31 01:27:17.355097 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:27:17.355048    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lpn8j\" (UniqueName: \"kubernetes.io/projected/45450e7a-2b62-4417-acca-d370251f6d54-kube-api-access-lpn8j\") pod \"e2e-api-a-7cec02-openshift-default-6fc456b49-jn9wm\" (UID: \"45450e7a-2b62-4417-acca-d370251f6d54\") " pod="openshift-ingress/e2e-api-a-7cec02-openshift-default-6fc456b49-jn9wm"
Oct 31 01:27:17.362948 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:27:17.362916    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/45450e7a-2b62-4417-acca-d370251f6d54-istio-token\") pod \"e2e-api-a-7cec02-openshift-default-6fc456b49-jn9wm\" (UID: \"45450e7a-2b62-4417-acca-d370251f6d54\") " pod="openshift-ingress/e2e-api-a-7cec02-openshift-default-6fc456b49-jn9wm"
Oct 31 01:27:17.425916 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:27:17.425877    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:27:17.654257 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:27:17.654166    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-7cec02-openshift-default-6fc456b49-jn9wm"
Oct 31 01:27:18.104156 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:27:18.104130    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-7cec02-openshift-default-6fc456b49-jn9wm"]
Oct 31 01:27:18.105936 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:27:18.105912    4964 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45450e7a_2b62_4417_acca_d370251f6d54.slice/crio-2675968707a4e50660ef1f8ced15cb1dd0dbfdd6f20dc619fa426b24dd4a1b53 WatchSource:0}: Error finding container 2675968707a4e50660ef1f8ced15cb1dd0dbfdd6f20dc619fa426b24dd4a1b53: Status 404 returned error can't find the container with id 2675968707a4e50660ef1f8ced15cb1dd0dbfdd6f20dc619fa426b24dd4a1b53
Oct 31 01:27:18.109052 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:27:18.109014    4964 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 01:27:18.109137 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:27:18.109102    4964 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 01:27:18.109171 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:27:18.109142    4964 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 01:27:18.261092 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:27:18.261049    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-7cec02-openshift-default-6fc456b49-jn9wm" event={"ID":"45450e7a-2b62-4417-acca-d370251f6d54","Type":"ContainerStarted","Data":"362bac2a8bd4ba033fd5150f6a3973611b86e493d849686055f98cf44a5b996a"}
Oct 31 01:27:18.261282 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:27:18.261132    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-7cec02-openshift-default-6fc456b49-jn9wm" event={"ID":"45450e7a-2b62-4417-acca-d370251f6d54","Type":"ContainerStarted","Data":"2675968707a4e50660ef1f8ced15cb1dd0dbfdd6f20dc619fa426b24dd4a1b53"}
Oct 31 01:27:18.381343 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:27:18.381210    4964 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-7cec02-openshift-default-6fc456b49-jn9wm" podStartSLOduration=2.381195589 podStartE2EDuration="2.381195589s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:27:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:27:18.377166206 +0000 UTC m=+1406.853732587" watchObservedRunningTime="2026-10-31 01:27:18.381195589 +0000 UTC m=+1406.857761969"
Oct 31 01:27:18.654746 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:27:18.654671    4964 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-7cec02-openshift-default-6fc456b49-jn9wm"
Oct 31 01:27:19.043101 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:27:19.043058    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:27:19.660448 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:27:19.660414    4964 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-7cec02-openshift-default-6fc456b49-jn9wm"
Oct 31 01:27:20.276971 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:27:20.276942    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-7cec02-openshift-default-6fc456b49-jn9wm"
Oct 31 01:27:20.277840 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:27:20.277819    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-7cec02-openshift-default-6fc456b49-jn9wm"
Oct 31 01:27:23.890663 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:27:23.890622    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:27:28.703068 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:27:28.702977    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:27:30.914574 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:27:30.914535    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:27:31.434449 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:27:31.434413    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:27:31.632625 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:27:31.632588    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:27:33.815059 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:27:33.815018    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:27:34.704904 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:27:34.704872    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:27:34.882901 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:27:34.882865    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:27:35.082824 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:27:35.082791    4964 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-7cec02-64cb84f795-865w7"]
Oct 31 01:27:35.088136 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:27:35.088114    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-7cec02-64cb84f795-865w7"
Oct 31 01:27:35.171597 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:27:35.171564    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b9a6e0ab-3025-4363-8ad7-4434733148b7-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-7cec02-64cb84f795-865w7\" (UID: \"b9a6e0ab-3025-4363-8ad7-4434733148b7\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-7cec02-64cb84f795-865w7"
Oct 31 01:27:35.171780 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:27:35.171697    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8mjd4\" (UniqueName: \"kubernetes.io/projected/b9a6e0ab-3025-4363-8ad7-4434733148b7-kube-api-access-8mjd4\") pod \"payload-pre-processing-e2e-api-a-7cec02-64cb84f795-865w7\" (UID: \"b9a6e0ab-3025-4363-8ad7-4434733148b7\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-7cec02-64cb84f795-865w7"
Oct 31 01:27:35.189263 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:27:35.189232    4964 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-7cec02\""
Oct 31 01:27:35.189501 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:27:35.189356    4964 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-7cec02-dockercfg-xbhvt\""
Oct 31 01:27:35.193251 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:27:35.193228    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-7cec02-64cb84f795-865w7"]
Oct 31 01:27:35.265514 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:27:35.265482    4964 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-7cec02-6d95f44b95-wgspn"]
Oct 31 01:27:35.270403 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:27:35.270376    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-7cec02-6d95f44b95-wgspn"
Oct 31 01:27:35.300866 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:27:35.299640    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b9a6e0ab-3025-4363-8ad7-4434733148b7-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-7cec02-64cb84f795-865w7\" (UID: \"b9a6e0ab-3025-4363-8ad7-4434733148b7\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-7cec02-64cb84f795-865w7"
Oct 31 01:27:35.300866 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:27:35.299739    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8mjd4\" (UniqueName: \"kubernetes.io/projected/b9a6e0ab-3025-4363-8ad7-4434733148b7-kube-api-access-8mjd4\") pod \"payload-pre-processing-e2e-api-a-7cec02-64cb84f795-865w7\" (UID: \"b9a6e0ab-3025-4363-8ad7-4434733148b7\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-7cec02-64cb84f795-865w7"
Oct 31 01:27:35.302834 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:27:35.301123    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b9a6e0ab-3025-4363-8ad7-4434733148b7-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-7cec02-64cb84f795-865w7\" (UID: \"b9a6e0ab-3025-4363-8ad7-4434733148b7\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-7cec02-64cb84f795-865w7"
Oct 31 01:27:35.302834 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:27:35.302104    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-7cec02-6d95f44b95-wgspn"]
Oct 31 01:27:35.369424 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:27:35.365490    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8mjd4\" (UniqueName: \"kubernetes.io/projected/b9a6e0ab-3025-4363-8ad7-4434733148b7-kube-api-access-8mjd4\") pod \"payload-pre-processing-e2e-api-a-7cec02-64cb84f795-865w7\" (UID: \"b9a6e0ab-3025-4363-8ad7-4434733148b7\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-7cec02-64cb84f795-865w7"
Oct 31 01:27:35.401338 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:27:35.401287    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a26b5198-d0fe-403b-858f-0e6619c15c3b-plugins-config-volume\") pod \"payload-processing-e2e-api-a-7cec02-6d95f44b95-wgspn\" (UID: \"a26b5198-d0fe-403b-858f-0e6619c15c3b\") " pod="openshift-ingress/payload-processing-e2e-api-a-7cec02-6d95f44b95-wgspn"
Oct 31 01:27:35.401524 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:27:35.401463    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vd4rr\" (UniqueName: \"kubernetes.io/projected/a26b5198-d0fe-403b-858f-0e6619c15c3b-kube-api-access-vd4rr\") pod \"payload-processing-e2e-api-a-7cec02-6d95f44b95-wgspn\" (UID: \"a26b5198-d0fe-403b-858f-0e6619c15c3b\") " pod="openshift-ingress/payload-processing-e2e-api-a-7cec02-6d95f44b95-wgspn"
Oct 31 01:27:35.406916 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:27:35.406890    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-7cec02-64cb84f795-865w7"
Oct 31 01:27:35.502906 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:27:35.502860    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a26b5198-d0fe-403b-858f-0e6619c15c3b-plugins-config-volume\") pod \"payload-processing-e2e-api-a-7cec02-6d95f44b95-wgspn\" (UID: \"a26b5198-d0fe-403b-858f-0e6619c15c3b\") " pod="openshift-ingress/payload-processing-e2e-api-a-7cec02-6d95f44b95-wgspn"
Oct 31 01:27:35.503936 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:27:35.503807    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a26b5198-d0fe-403b-858f-0e6619c15c3b-plugins-config-volume\") pod \"payload-processing-e2e-api-a-7cec02-6d95f44b95-wgspn\" (UID: \"a26b5198-d0fe-403b-858f-0e6619c15c3b\") " pod="openshift-ingress/payload-processing-e2e-api-a-7cec02-6d95f44b95-wgspn"
Oct 31 01:27:35.504097 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:27:35.504077    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vd4rr\" (UniqueName: \"kubernetes.io/projected/a26b5198-d0fe-403b-858f-0e6619c15c3b-kube-api-access-vd4rr\") pod \"payload-processing-e2e-api-a-7cec02-6d95f44b95-wgspn\" (UID: \"a26b5198-d0fe-403b-858f-0e6619c15c3b\") " pod="openshift-ingress/payload-processing-e2e-api-a-7cec02-6d95f44b95-wgspn"
Oct 31 01:27:35.562079 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:27:35.562046    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vd4rr\" (UniqueName: \"kubernetes.io/projected/a26b5198-d0fe-403b-858f-0e6619c15c3b-kube-api-access-vd4rr\") pod \"payload-processing-e2e-api-a-7cec02-6d95f44b95-wgspn\" (UID: \"a26b5198-d0fe-403b-858f-0e6619c15c3b\") " pod="openshift-ingress/payload-processing-e2e-api-a-7cec02-6d95f44b95-wgspn"
Oct 31 01:27:35.637796 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:27:35.637729    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-7cec02-6d95f44b95-wgspn"
Oct 31 01:27:35.810140 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:27:35.810106    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-7cec02-64cb84f795-865w7"]
Oct 31 01:27:35.818695 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:27:35.818658    4964 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb9a6e0ab_3025_4363_8ad7_4434733148b7.slice/crio-32e7522530bc35d81fa846c35c91964406257cf3b63d8151b9a8b3dbe64eb20f WatchSource:0}: Error finding container 32e7522530bc35d81fa846c35c91964406257cf3b63d8151b9a8b3dbe64eb20f: Status 404 returned error can't find the container with id 32e7522530bc35d81fa846c35c91964406257cf3b63d8151b9a8b3dbe64eb20f
Oct 31 01:27:36.024042 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:27:36.024010    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-7cec02-6d95f44b95-wgspn"]
Oct 31 01:27:36.025807 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:27:36.025778    4964 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda26b5198_d0fe_403b_858f_0e6619c15c3b.slice/crio-b54f447f16e6e7a0412cfb17317821f3695da7df9106e943b7c7289f1d0bc84a WatchSource:0}: Error finding container b54f447f16e6e7a0412cfb17317821f3695da7df9106e943b7c7289f1d0bc84a: Status 404 returned error can't find the container with id b54f447f16e6e7a0412cfb17317821f3695da7df9106e943b7c7289f1d0bc84a
Oct 31 01:27:36.405464 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:27:36.405361    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-7cec02-6d95f44b95-wgspn" event={"ID":"a26b5198-d0fe-403b-858f-0e6619c15c3b","Type":"ContainerStarted","Data":"3af96a2b131aba33ff6fb070316294ff8effa58cff6e12c4335f18ac167a713a"}
Oct 31 01:27:36.405464 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:27:36.405429    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-7cec02-6d95f44b95-wgspn" event={"ID":"a26b5198-d0fe-403b-858f-0e6619c15c3b","Type":"ContainerStarted","Data":"b54f447f16e6e7a0412cfb17317821f3695da7df9106e943b7c7289f1d0bc84a"}
Oct 31 01:27:36.405681 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:27:36.405491    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-7cec02-6d95f44b95-wgspn"
Oct 31 01:27:36.407080 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:27:36.407054    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-7cec02-64cb84f795-865w7" event={"ID":"b9a6e0ab-3025-4363-8ad7-4434733148b7","Type":"ContainerStarted","Data":"5e2a25f80a5a70f23e8bdbcf22ad6dcf857b166706697945deb90bf4f2a17ab6"}
Oct 31 01:27:36.407225 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:27:36.407093    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-7cec02-64cb84f795-865w7" event={"ID":"b9a6e0ab-3025-4363-8ad7-4434733148b7","Type":"ContainerStarted","Data":"32e7522530bc35d81fa846c35c91964406257cf3b63d8151b9a8b3dbe64eb20f"}
Oct 31 01:27:36.407225 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:27:36.407195    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-7cec02-64cb84f795-865w7"
Oct 31 01:27:36.519921 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:27:36.519864    4964 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-7cec02-6d95f44b95-wgspn" podStartSLOduration=1.519849095 podStartE2EDuration="1.519849095s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:27:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:27:36.518508206 +0000 UTC m=+1424.995074587" watchObservedRunningTime="2026-10-31 01:27:36.519849095 +0000 UTC m=+1424.996415475"
Oct 31 01:27:36.641243 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:27:36.641177    4964 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-7cec02-64cb84f795-865w7" podStartSLOduration=1.641158618 podStartE2EDuration="1.641158618s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:27:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:27:36.635822781 +0000 UTC m=+1425.112389167" watchObservedRunningTime="2026-10-31 01:27:36.641158618 +0000 UTC m=+1425.117724998"
Oct 31 01:27:37.050160 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:27:37.050121    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:27:39.014930 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:27:39.014845    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:27:40.726780 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:27:40.726745    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:27:41.240711 ip-10-0-133-197 kubenswrapper[4964]: E1031 01:27:41.240598    4964 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q56x5" containerName="extractor"
Oct 31 01:27:43.805080 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:27:43.805041    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:27:44.851344 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:27:44.850945    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:27:45.600038 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:27:45.599998    4964 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a8e157/distinct-2-w3-a8e157-kserve-9ddd45d6-ng7j7" podUID="bc7a71df-c523-4590-91df-cf8fdf3ed19e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:27:45.600684 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:27:45.600175    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:27:46.154340 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:27:46.150298    4964 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a8e157/distinct-w3-a8e157-kserve-549559dd5c-t29tw" podUID="4b0fd818-20c2-4dd6-9846-f361c236dbd9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:27:46.187585 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:27:46.187463    4964 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-a8e157/distinct-w3-a8e157-kserve-549559dd5c-t29tw" podUID="4b0fd818-20c2-4dd6-9846-f361c236dbd9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:27:46.749462 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:27:46.749419    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:27:47.323886 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:27:47.323837    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:27:47.417751 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:27:47.417726    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-7cec02-6d95f44b95-wgspn"
Oct 31 01:27:47.417953 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:27:47.417788    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-7cec02-64cb84f795-865w7"
Oct 31 01:27:47.688278 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:27:47.686542    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:27:47.879394 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:27:47.879345    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:27:48.844222 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:27:48.844193    4964 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-flwkb"]
Oct 31 01:27:48.848486 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:27:48.848470    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-flwkb"
Oct 31 01:27:48.893584 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:27:48.893559    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:27:48.904828 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:27:48.904270    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/da53c8d7-cc60-45cd-ab19-365d486484b1-istiod-ca-cert\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-flwkb\" (UID: \"da53c8d7-cc60-45cd-ab19-365d486484b1\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-flwkb"
Oct 31 01:27:48.904828 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:27:48.904375    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nhlrq\" (UniqueName: \"kubernetes.io/projected/da53c8d7-cc60-45cd-ab19-365d486484b1-kube-api-access-nhlrq\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-flwkb\" (UID: \"da53c8d7-cc60-45cd-ab19-365d486484b1\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-flwkb"
Oct 31 01:27:48.904828 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:27:48.904402    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/da53c8d7-cc60-45cd-ab19-365d486484b1-istio-data\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-flwkb\" (UID: \"da53c8d7-cc60-45cd-ab19-365d486484b1\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-flwkb"
Oct 31 01:27:48.904828 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:27:48.904429    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/da53c8d7-cc60-45cd-ab19-365d486484b1-istio-token\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-flwkb\" (UID: \"da53c8d7-cc60-45cd-ab19-365d486484b1\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-flwkb"
Oct 31 01:27:48.904828 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:27:48.904467    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/da53c8d7-cc60-45cd-ab19-365d486484b1-credential-socket\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-flwkb\" (UID: \"da53c8d7-cc60-45cd-ab19-365d486484b1\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-flwkb"
Oct 31 01:27:48.904828 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:27:48.904497    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/da53c8d7-cc60-45cd-ab19-365d486484b1-workload-socket\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-flwkb\" (UID: \"da53c8d7-cc60-45cd-ab19-365d486484b1\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-flwkb"
Oct 31 01:27:48.904828 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:27:48.904539    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/da53c8d7-cc60-45cd-ab19-365d486484b1-istio-envoy\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-flwkb\" (UID: \"da53c8d7-cc60-45cd-ab19-365d486484b1\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-flwkb"
Oct 31 01:27:48.904828 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:27:48.904588    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/da53c8d7-cc60-45cd-ab19-365d486484b1-workload-certs\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-flwkb\" (UID: \"da53c8d7-cc60-45cd-ab19-365d486484b1\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-flwkb"
Oct 31 01:27:48.904828 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:27:48.904605    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/da53c8d7-cc60-45cd-ab19-365d486484b1-istio-podinfo\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-flwkb\" (UID: \"da53c8d7-cc60-45cd-ab19-365d486484b1\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-flwkb"
Oct 31 01:27:48.973076 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:27:48.973040    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-flwkb"]
Oct 31 01:27:49.007000 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:27:49.006965    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/da53c8d7-cc60-45cd-ab19-365d486484b1-workload-certs\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-flwkb\" (UID: \"da53c8d7-cc60-45cd-ab19-365d486484b1\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-flwkb"
Oct 31 01:27:49.007403 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:27:49.007378    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/da53c8d7-cc60-45cd-ab19-365d486484b1-istio-podinfo\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-flwkb\" (UID: \"da53c8d7-cc60-45cd-ab19-365d486484b1\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-flwkb"
Oct 31 01:27:49.007594 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:27:49.007571    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/da53c8d7-cc60-45cd-ab19-365d486484b1-istiod-ca-cert\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-flwkb\" (UID: \"da53c8d7-cc60-45cd-ab19-365d486484b1\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-flwkb"
Oct 31 01:27:49.007594 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:27:49.007584    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/da53c8d7-cc60-45cd-ab19-365d486484b1-workload-certs\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-flwkb\" (UID: \"da53c8d7-cc60-45cd-ab19-365d486484b1\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-flwkb"
Oct 31 01:27:49.007723 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:27:49.007703    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nhlrq\" (UniqueName: \"kubernetes.io/projected/da53c8d7-cc60-45cd-ab19-365d486484b1-kube-api-access-nhlrq\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-flwkb\" (UID: \"da53c8d7-cc60-45cd-ab19-365d486484b1\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-flwkb"
Oct 31 01:27:49.007785 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:27:49.007750    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/da53c8d7-cc60-45cd-ab19-365d486484b1-istio-data\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-flwkb\" (UID: \"da53c8d7-cc60-45cd-ab19-365d486484b1\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-flwkb"
Oct 31 01:27:49.007837 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:27:49.007781    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/da53c8d7-cc60-45cd-ab19-365d486484b1-istio-token\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-flwkb\" (UID: \"da53c8d7-cc60-45cd-ab19-365d486484b1\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-flwkb"
Oct 31 01:27:49.007890 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:27:49.007848    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/da53c8d7-cc60-45cd-ab19-365d486484b1-credential-socket\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-flwkb\" (UID: \"da53c8d7-cc60-45cd-ab19-365d486484b1\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-flwkb"
Oct 31 01:27:49.007945 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:27:49.007890    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/da53c8d7-cc60-45cd-ab19-365d486484b1-workload-socket\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-flwkb\" (UID: \"da53c8d7-cc60-45cd-ab19-365d486484b1\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-flwkb"
Oct 31 01:27:49.008000 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:27:49.007958    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/da53c8d7-cc60-45cd-ab19-365d486484b1-istio-envoy\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-flwkb\" (UID: \"da53c8d7-cc60-45cd-ab19-365d486484b1\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-flwkb"
Oct 31 01:27:49.008495 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:27:49.008258    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/da53c8d7-cc60-45cd-ab19-365d486484b1-istiod-ca-cert\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-flwkb\" (UID: \"da53c8d7-cc60-45cd-ab19-365d486484b1\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-flwkb"
Oct 31 01:27:49.008495 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:27:49.008363    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/da53c8d7-cc60-45cd-ab19-365d486484b1-credential-socket\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-flwkb\" (UID: \"da53c8d7-cc60-45cd-ab19-365d486484b1\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-flwkb"
Oct 31 01:27:49.008495 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:27:49.008462    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/da53c8d7-cc60-45cd-ab19-365d486484b1-workload-socket\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-flwkb\" (UID: \"da53c8d7-cc60-45cd-ab19-365d486484b1\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-flwkb"
Oct 31 01:27:49.008641 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:27:49.008520    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/da53c8d7-cc60-45cd-ab19-365d486484b1-istio-data\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-flwkb\" (UID: \"da53c8d7-cc60-45cd-ab19-365d486484b1\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-flwkb"
Oct 31 01:27:49.010843 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:27:49.010817    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/da53c8d7-cc60-45cd-ab19-365d486484b1-istio-envoy\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-flwkb\" (UID: \"da53c8d7-cc60-45cd-ab19-365d486484b1\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-flwkb"
Oct 31 01:27:49.011802 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:27:49.011780    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/da53c8d7-cc60-45cd-ab19-365d486484b1-istio-podinfo\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-flwkb\" (UID: \"da53c8d7-cc60-45cd-ab19-365d486484b1\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-flwkb"
Oct 31 01:27:49.064193 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:27:49.064163    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/da53c8d7-cc60-45cd-ab19-365d486484b1-istio-token\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-flwkb\" (UID: \"da53c8d7-cc60-45cd-ab19-365d486484b1\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-flwkb"
Oct 31 01:27:49.064380 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:27:49.064212    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nhlrq\" (UniqueName: \"kubernetes.io/projected/da53c8d7-cc60-45cd-ab19-365d486484b1-kube-api-access-nhlrq\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-flwkb\" (UID: \"da53c8d7-cc60-45cd-ab19-365d486484b1\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-flwkb"
Oct 31 01:27:49.162170 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:27:49.162104    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-flwkb"
Oct 31 01:27:49.598791 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:27:49.598713    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-flwkb"]
Oct 31 01:27:49.601655 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:27:49.601621    4964 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 01:27:49.601768 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:27:49.601691    4964 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 01:27:49.601768 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:27:49.601731    4964 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 01:27:49.626895 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:27:49.626864    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-flwkb" event={"ID":"da53c8d7-cc60-45cd-ab19-365d486484b1","Type":"ContainerStarted","Data":"a6b38a7354dbf52e201f388f9aee18d1b1cfb98ed7600edff83b36d677c4f484"}
Oct 31 01:27:50.637863 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:27:50.637824    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-flwkb" event={"ID":"da53c8d7-cc60-45cd-ab19-365d486484b1","Type":"ContainerStarted","Data":"bdcf04a8c954cdaf6e316d4dc62721e3b63f9762d805650794f18632b675f279"}
Oct 31 01:27:50.864083 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:27:50.864024    4964 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-flwkb" podStartSLOduration=2.864007439 podStartE2EDuration="2.864007439s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:27:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:27:50.811780561 +0000 UTC m=+1439.288346942" watchObservedRunningTime="2026-10-31 01:27:50.864007439 +0000 UTC m=+1439.340573819"
Oct 31 01:27:51.162410 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:27:51.162374    4964 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-flwkb"
Oct 31 01:27:51.163695 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:27:51.163671    4964 patch_prober.go:29] interesting pod/e2e-worker-w2-00e840-openshift-default-59b96fb879-flwkb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" start-of-body=
Oct 31 01:27:51.163820 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:27:51.163726    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-flwkb" podUID="da53c8d7-cc60-45cd-ab19-365d486484b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused"
Oct 31 01:27:51.683469 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:27:51.683433    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:27:51.883112 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:27:51.883074    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:27:52.061924 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:27:52.061896    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:27:52.162499 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:27:52.162462    4964 patch_prober.go:29] interesting pod/e2e-worker-w2-00e840-openshift-default-59b96fb879-flwkb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" start-of-body=
Oct 31 01:27:52.162663 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:27:52.162511    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-flwkb" podUID="da53c8d7-cc60-45cd-ab19-365d486484b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused"
Oct 31 01:27:53.162628 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:27:53.162592    4964 patch_prober.go:29] interesting pod/e2e-worker-w2-00e840-openshift-default-59b96fb879-flwkb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" start-of-body=
Oct 31 01:27:53.163042 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:27:53.162659    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-flwkb" podUID="da53c8d7-cc60-45cd-ab19-365d486484b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused"
Oct 31 01:27:54.163433 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:27:54.163393    4964 patch_prober.go:29] interesting pod/e2e-worker-w2-00e840-openshift-default-59b96fb879-flwkb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" start-of-body=
Oct 31 01:27:54.163818 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:27:54.163451    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-flwkb" podUID="da53c8d7-cc60-45cd-ab19-365d486484b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused"
Oct 31 01:27:54.167980 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:27:54.167947    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:27:55.043305 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:27:55.043241    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:27:55.196389 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:27:55.183750    4964 patch_prober.go:29] interesting pod/e2e-worker-w2-00e840-openshift-default-59b96fb879-flwkb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" start-of-body=
Oct 31 01:27:55.201636 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:27:55.201597    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-flwkb" podUID="da53c8d7-cc60-45cd-ab19-365d486484b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused"
Oct 31 01:27:55.271613 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:27:55.271489    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:27:55.417193 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:27:55.417097    4964 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-01cbd8-f74599447-dqk2f"]
Oct 31 01:27:55.442907 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:27:55.442879    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-01cbd8-f74599447-dqk2f"
Oct 31 01:27:55.490069 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:27:55.487095    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-01cbd8-f74599447-dqk2f"]
Oct 31 01:27:55.540624 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:27:55.539765    4964 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-01cbd8-dockercfg-j8zsv\""
Oct 31 01:27:55.540624 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:27:55.540531    4964 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-01cbd8\""
Oct 31 01:27:55.554027 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:27:55.553742    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8225k\" (UniqueName: \"kubernetes.io/projected/68178bce-da5e-4260-bdee-291a5d7bd028-kube-api-access-8225k\") pod \"payload-pre-processing-e2e-api-a-01cbd8-f74599447-dqk2f\" (UID: \"68178bce-da5e-4260-bdee-291a5d7bd028\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-01cbd8-f74599447-dqk2f"
Oct 31 01:27:55.554465 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:27:55.554367    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/68178bce-da5e-4260-bdee-291a5d7bd028-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-01cbd8-f74599447-dqk2f\" (UID: \"68178bce-da5e-4260-bdee-291a5d7bd028\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-01cbd8-f74599447-dqk2f"
Oct 31 01:27:55.565678 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:27:55.565652    4964 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-01cbd8-75c567cb95-tpmh5"]
Oct 31 01:27:55.571241 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:27:55.571219    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-01cbd8-75c567cb95-tpmh5"
Oct 31 01:27:55.654212 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:27:55.648962    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-01cbd8-75c567cb95-tpmh5"]
Oct 31 01:27:55.669241 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:27:55.663919    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pdbjt\" (UniqueName: \"kubernetes.io/projected/086815fd-df1b-4b33-a457-5f67321d8224-kube-api-access-pdbjt\") pod \"payload-processing-e2e-api-a-01cbd8-75c567cb95-tpmh5\" (UID: \"086815fd-df1b-4b33-a457-5f67321d8224\") " pod="openshift-ingress/payload-processing-e2e-api-a-01cbd8-75c567cb95-tpmh5"
Oct 31 01:27:55.669241 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:27:55.663992    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/68178bce-da5e-4260-bdee-291a5d7bd028-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-01cbd8-f74599447-dqk2f\" (UID: \"68178bce-da5e-4260-bdee-291a5d7bd028\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-01cbd8-f74599447-dqk2f"
Oct 31 01:27:55.669241 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:27:55.664163    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8225k\" (UniqueName: \"kubernetes.io/projected/68178bce-da5e-4260-bdee-291a5d7bd028-kube-api-access-8225k\") pod \"payload-pre-processing-e2e-api-a-01cbd8-f74599447-dqk2f\" (UID: \"68178bce-da5e-4260-bdee-291a5d7bd028\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-01cbd8-f74599447-dqk2f"
Oct 31 01:27:55.669241 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:27:55.664238    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/086815fd-df1b-4b33-a457-5f67321d8224-plugins-config-volume\") pod \"payload-processing-e2e-api-a-01cbd8-75c567cb95-tpmh5\" (UID: \"086815fd-df1b-4b33-a457-5f67321d8224\") " pod="openshift-ingress/payload-processing-e2e-api-a-01cbd8-75c567cb95-tpmh5"
Oct 31 01:27:55.669241 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:27:55.665457    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/68178bce-da5e-4260-bdee-291a5d7bd028-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-01cbd8-f74599447-dqk2f\" (UID: \"68178bce-da5e-4260-bdee-291a5d7bd028\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-01cbd8-f74599447-dqk2f"
Oct 31 01:27:55.805234 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:27:55.805195    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8225k\" (UniqueName: \"kubernetes.io/projected/68178bce-da5e-4260-bdee-291a5d7bd028-kube-api-access-8225k\") pod \"payload-pre-processing-e2e-api-a-01cbd8-f74599447-dqk2f\" (UID: \"68178bce-da5e-4260-bdee-291a5d7bd028\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-01cbd8-f74599447-dqk2f"
Oct 31 01:27:55.806146 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:27:55.806123    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-01cbd8-f74599447-dqk2f"
Oct 31 01:27:55.808135 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:27:55.808105    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pdbjt\" (UniqueName: \"kubernetes.io/projected/086815fd-df1b-4b33-a457-5f67321d8224-kube-api-access-pdbjt\") pod \"payload-processing-e2e-api-a-01cbd8-75c567cb95-tpmh5\" (UID: \"086815fd-df1b-4b33-a457-5f67321d8224\") " pod="openshift-ingress/payload-processing-e2e-api-a-01cbd8-75c567cb95-tpmh5"
Oct 31 01:27:55.808399 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:27:55.808376    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/086815fd-df1b-4b33-a457-5f67321d8224-plugins-config-volume\") pod \"payload-processing-e2e-api-a-01cbd8-75c567cb95-tpmh5\" (UID: \"086815fd-df1b-4b33-a457-5f67321d8224\") " pod="openshift-ingress/payload-processing-e2e-api-a-01cbd8-75c567cb95-tpmh5"
Oct 31 01:27:55.809353 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:27:55.809261    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/086815fd-df1b-4b33-a457-5f67321d8224-plugins-config-volume\") pod \"payload-processing-e2e-api-a-01cbd8-75c567cb95-tpmh5\" (UID: \"086815fd-df1b-4b33-a457-5f67321d8224\") " pod="openshift-ingress/payload-processing-e2e-api-a-01cbd8-75c567cb95-tpmh5"
Oct 31 01:27:55.867669 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:27:55.867633    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pdbjt\" (UniqueName: \"kubernetes.io/projected/086815fd-df1b-4b33-a457-5f67321d8224-kube-api-access-pdbjt\") pod \"payload-processing-e2e-api-a-01cbd8-75c567cb95-tpmh5\" (UID: \"086815fd-df1b-4b33-a457-5f67321d8224\") " pod="openshift-ingress/payload-processing-e2e-api-a-01cbd8-75c567cb95-tpmh5"
Oct 31 01:27:55.897296 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:27:55.897260    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-01cbd8-75c567cb95-tpmh5"
Oct 31 01:27:56.163138 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:27:56.163095    4964 patch_prober.go:29] interesting pod/e2e-worker-w2-00e840-openshift-default-59b96fb879-flwkb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" start-of-body=
Oct 31 01:27:56.163304 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:27:56.163163    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-flwkb" podUID="da53c8d7-cc60-45cd-ab19-365d486484b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused"
Oct 31 01:27:56.263146 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:27:56.263114    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-01cbd8-f74599447-dqk2f"]
Oct 31 01:27:56.263942 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:27:56.263917    4964 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod68178bce_da5e_4260_bdee_291a5d7bd028.slice/crio-e15c5b94ba8f79fabb65ec125568045bc9aee8a6e1088c812b965e981a664ada WatchSource:0}: Error finding container e15c5b94ba8f79fabb65ec125568045bc9aee8a6e1088c812b965e981a664ada: Status 404 returned error can't find the container with id e15c5b94ba8f79fabb65ec125568045bc9aee8a6e1088c812b965e981a664ada
Oct 31 01:27:56.371269 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:27:56.371241    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-01cbd8-75c567cb95-tpmh5"]
Oct 31 01:27:56.371579 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:27:56.371545    4964 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod086815fd_df1b_4b33_a457_5f67321d8224.slice/crio-0ced1e5730d293c3c71f1c6b1ffa376661ecafa13ad3ec402d0426db6f67a0b6 WatchSource:0}: Error finding container 0ced1e5730d293c3c71f1c6b1ffa376661ecafa13ad3ec402d0426db6f67a0b6: Status 404 returned error can't find the container with id 0ced1e5730d293c3c71f1c6b1ffa376661ecafa13ad3ec402d0426db6f67a0b6
Oct 31 01:27:56.880354 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:27:56.880299    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-01cbd8-75c567cb95-tpmh5" event={"ID":"086815fd-df1b-4b33-a457-5f67321d8224","Type":"ContainerStarted","Data":"95e1130d0c5f85f21f8b944a35613eb4bcf22513fd52227884b8ff22e332a0f2"}
Oct 31 01:27:56.880531 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:27:56.880375    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-01cbd8-75c567cb95-tpmh5" event={"ID":"086815fd-df1b-4b33-a457-5f67321d8224","Type":"ContainerStarted","Data":"0ced1e5730d293c3c71f1c6b1ffa376661ecafa13ad3ec402d0426db6f67a0b6"}
Oct 31 01:27:56.880531 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:27:56.880410    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-01cbd8-75c567cb95-tpmh5"
Oct 31 01:27:56.882864 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:27:56.882354    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-01cbd8-f74599447-dqk2f" event={"ID":"68178bce-da5e-4260-bdee-291a5d7bd028","Type":"ContainerStarted","Data":"b033908913c9b6f9d5a64503b7f06256b112e8742c763f61262410896d216321"}
Oct 31 01:27:56.882864 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:27:56.882402    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-01cbd8-f74599447-dqk2f" event={"ID":"68178bce-da5e-4260-bdee-291a5d7bd028","Type":"ContainerStarted","Data":"e15c5b94ba8f79fabb65ec125568045bc9aee8a6e1088c812b965e981a664ada"}
Oct 31 01:27:56.882864 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:27:56.882423    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-01cbd8-f74599447-dqk2f"
Oct 31 01:27:57.004107 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:27:57.004046    4964 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-01cbd8-75c567cb95-tpmh5" podStartSLOduration=2.004028582 podStartE2EDuration="2.004028582s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:27:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:27:56.99817641 +0000 UTC m=+1445.474742792" watchObservedRunningTime="2026-10-31 01:27:57.004028582 +0000 UTC m=+1445.480594963"
Oct 31 01:27:57.148397 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:27:57.148231    4964 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-01cbd8-f74599447-dqk2f" podStartSLOduration=2.148211572 podStartE2EDuration="2.148211572s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:27:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:27:57.137340422 +0000 UTC m=+1445.613906803" watchObservedRunningTime="2026-10-31 01:27:57.148211572 +0000 UTC m=+1445.624777955"
Oct 31 01:27:57.175492 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:27:57.175462    4964 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-flwkb"
Oct 31 01:27:57.176213 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:27:57.176189    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-flwkb"
Oct 31 01:27:57.177944 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:27:57.177918    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-flwkb"
Oct 31 01:27:59.197388 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:27:59.197353    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:28:04.203553 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:28:04.202302    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:28:07.894372 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:28:07.894306    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-01cbd8-f74599447-dqk2f"
Oct 31 01:28:07.894849 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:28:07.894554    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-01cbd8-75c567cb95-tpmh5"
Oct 31 01:28:08.250601 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:28:08.250564    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:28:08.709013 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:28:08.708944    4964 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-7cec02-56fb787486-ms9st"]
Oct 31 01:28:08.713343 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:28:08.713297    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-7cec02-56fb787486-ms9st"
Oct 31 01:28:08.768256 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:28:08.768196    4964 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-7cec02-serving-cert\""
Oct 31 01:28:08.772661 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:28:08.772633    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-7cec02-56fb787486-ms9st"]
Oct 31 01:28:08.869208 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:28:08.869170    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c8cba8e6-afc6-459a-89a5-615f812b930c-metrics-certs\") pod \"maas-api-e2e-api-a-7cec02-56fb787486-ms9st\" (UID: \"c8cba8e6-afc6-459a-89a5-615f812b930c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-7cec02-56fb787486-ms9st"
Oct 31 01:28:08.869434 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:28:08.869382    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vc6fs\" (UniqueName: \"kubernetes.io/projected/c8cba8e6-afc6-459a-89a5-615f812b930c-kube-api-access-vc6fs\") pod \"maas-api-e2e-api-a-7cec02-56fb787486-ms9st\" (UID: \"c8cba8e6-afc6-459a-89a5-615f812b930c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-7cec02-56fb787486-ms9st"
Oct 31 01:28:08.869540 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:28:08.869519    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c8cba8e6-afc6-459a-89a5-615f812b930c-maas-api-tls\") pod \"maas-api-e2e-api-a-7cec02-56fb787486-ms9st\" (UID: \"c8cba8e6-afc6-459a-89a5-615f812b930c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-7cec02-56fb787486-ms9st"
Oct 31 01:28:08.869631 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:28:08.869618    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c8cba8e6-afc6-459a-89a5-615f812b930c-openshift-service-ca\") pod \"maas-api-e2e-api-a-7cec02-56fb787486-ms9st\" (UID: \"c8cba8e6-afc6-459a-89a5-615f812b930c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-7cec02-56fb787486-ms9st"
Oct 31 01:28:08.916396 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:28:08.916363    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:28:08.971532 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:28:08.971492    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c8cba8e6-afc6-459a-89a5-615f812b930c-maas-api-tls\") pod \"maas-api-e2e-api-a-7cec02-56fb787486-ms9st\" (UID: \"c8cba8e6-afc6-459a-89a5-615f812b930c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-7cec02-56fb787486-ms9st"
Oct 31 01:28:08.971786 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:28:08.971644    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c8cba8e6-afc6-459a-89a5-615f812b930c-openshift-service-ca\") pod \"maas-api-e2e-api-a-7cec02-56fb787486-ms9st\" (UID: \"c8cba8e6-afc6-459a-89a5-615f812b930c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-7cec02-56fb787486-ms9st"
Oct 31 01:28:08.971786 ip-10-0-133-197 kubenswrapper[4964]: E1031 01:28:08.971675    4964 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-7cec02-serving-cert: secret "maas-api-e2e-api-a-7cec02-serving-cert" not found
Oct 31 01:28:08.971786 ip-10-0-133-197 kubenswrapper[4964]: E1031 01:28:08.971750    4964 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c8cba8e6-afc6-459a-89a5-615f812b930c-maas-api-tls podName:c8cba8e6-afc6-459a-89a5-615f812b930c nodeName:}" failed. No retries permitted until 2026-10-31 01:28:09.471728641 +0000 UTC m=+1457.948295009 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/c8cba8e6-afc6-459a-89a5-615f812b930c-maas-api-tls") pod "maas-api-e2e-api-a-7cec02-56fb787486-ms9st" (UID: "c8cba8e6-afc6-459a-89a5-615f812b930c") : secret "maas-api-e2e-api-a-7cec02-serving-cert" not found
Oct 31 01:28:08.971975 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:28:08.971796    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c8cba8e6-afc6-459a-89a5-615f812b930c-metrics-certs\") pod \"maas-api-e2e-api-a-7cec02-56fb787486-ms9st\" (UID: \"c8cba8e6-afc6-459a-89a5-615f812b930c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-7cec02-56fb787486-ms9st"
Oct 31 01:28:08.971975 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:28:08.971906    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vc6fs\" (UniqueName: \"kubernetes.io/projected/c8cba8e6-afc6-459a-89a5-615f812b930c-kube-api-access-vc6fs\") pod \"maas-api-e2e-api-a-7cec02-56fb787486-ms9st\" (UID: \"c8cba8e6-afc6-459a-89a5-615f812b930c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-7cec02-56fb787486-ms9st"
Oct 31 01:28:08.972350 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:28:08.972332    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c8cba8e6-afc6-459a-89a5-615f812b930c-openshift-service-ca\") pod \"maas-api-e2e-api-a-7cec02-56fb787486-ms9st\" (UID: \"c8cba8e6-afc6-459a-89a5-615f812b930c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-7cec02-56fb787486-ms9st"
Oct 31 01:28:08.975189 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:28:08.975167    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c8cba8e6-afc6-459a-89a5-615f812b930c-metrics-certs\") pod \"maas-api-e2e-api-a-7cec02-56fb787486-ms9st\" (UID: \"c8cba8e6-afc6-459a-89a5-615f812b930c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-7cec02-56fb787486-ms9st"
Oct 31 01:28:09.029592 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:28:09.029557    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vc6fs\" (UniqueName: \"kubernetes.io/projected/c8cba8e6-afc6-459a-89a5-615f812b930c-kube-api-access-vc6fs\") pod \"maas-api-e2e-api-a-7cec02-56fb787486-ms9st\" (UID: \"c8cba8e6-afc6-459a-89a5-615f812b930c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-7cec02-56fb787486-ms9st"
Oct 31 01:28:09.095684 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:28:09.095640    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:28:09.268408 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:28:09.268260    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:28:09.478871 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:28:09.478834    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c8cba8e6-afc6-459a-89a5-615f812b930c-maas-api-tls\") pod \"maas-api-e2e-api-a-7cec02-56fb787486-ms9st\" (UID: \"c8cba8e6-afc6-459a-89a5-615f812b930c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-7cec02-56fb787486-ms9st"
Oct 31 01:28:09.481440 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:28:09.481416    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c8cba8e6-afc6-459a-89a5-615f812b930c-maas-api-tls\") pod \"maas-api-e2e-api-a-7cec02-56fb787486-ms9st\" (UID: \"c8cba8e6-afc6-459a-89a5-615f812b930c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-7cec02-56fb787486-ms9st"
Oct 31 01:28:09.505931 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:28:09.505902    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:28:09.630823 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:28:09.630752    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-7cec02-56fb787486-ms9st"
Oct 31 01:28:10.012302 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:28:10.012266    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-7cec02-56fb787486-ms9st"]
Oct 31 01:28:10.012925 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:28:10.012903    4964 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc8cba8e6_afc6_459a_89a5_615f812b930c.slice/crio-1da8202cb714f5ccc3465c25ff579ad5e85dbb804e0bc48508f0b319b12bb1c5 WatchSource:0}: Error finding container 1da8202cb714f5ccc3465c25ff579ad5e85dbb804e0bc48508f0b319b12bb1c5: Status 404 returned error can't find the container with id 1da8202cb714f5ccc3465c25ff579ad5e85dbb804e0bc48508f0b319b12bb1c5
Oct 31 01:28:10.014546 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:28:10.014528    4964 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 31 01:28:11.022225 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:28:11.022191    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-7cec02-56fb787486-ms9st" event={"ID":"c8cba8e6-afc6-459a-89a5-615f812b930c","Type":"ContainerStarted","Data":"6cf56edb7185751ae6e9c849d46c2c29a2f0ebbde3be50c5fccb8d36c77ab8f9"}
Oct 31 01:28:11.022668 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:28:11.022263    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-7cec02-56fb787486-ms9st" event={"ID":"c8cba8e6-afc6-459a-89a5-615f812b930c","Type":"ContainerStarted","Data":"1da8202cb714f5ccc3465c25ff579ad5e85dbb804e0bc48508f0b319b12bb1c5"}
Oct 31 01:28:11.023437 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:28:11.023416    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-7cec02-56fb787486-ms9st"
Oct 31 01:28:11.175104 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:28:11.175051    4964 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-7cec02-56fb787486-ms9st" podStartSLOduration=2.253122062 podStartE2EDuration="3.175038444s" totalImagesPullingTime="921.916382ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:28:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:28:11.173041459 +0000 UTC m=+1459.649607853" watchObservedRunningTime="2026-10-31 01:28:11.175038444 +0000 UTC m=+1459.651604824"
Oct 31 01:28:11.239082 ip-10-0-133-197 kubenswrapper[4964]: E1031 01:28:11.239043    4964 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q56x5" containerName="extractor"
Oct 31 01:28:13.875839 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:28:13.875808    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:28:18.041923 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:28:18.041891    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-7cec02-56fb787486-ms9st"
Oct 31 01:28:18.711395 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:28:18.711297    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:28:20.168098 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:28:20.168064    4964 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-18bf762c-openshift-default-77b9fdd9b5-z7xxk"]
Oct 31 01:28:20.172854 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:28:20.172836    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-18bf762c-openshift-default-77b9fdd9b5-z7xxk"
Oct 31 01:28:20.225219 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:28:20.225185    4964 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-18bf762c-openshift-default-dockercfg-8g8p2\""
Oct 31 01:28:20.279913 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:28:20.279881    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-18bf762c-openshift-default-77b9fdd9b5-z7xxk"]
Oct 31 01:28:20.331125 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:28:20.330884    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7c25fc09-ee27-46c3-95fc-bfc6bacbc46f-istio-envoy\") pod \"e2e-mt-18bf762c-openshift-default-77b9fdd9b5-z7xxk\" (UID: \"7c25fc09-ee27-46c3-95fc-bfc6bacbc46f\") " pod="openshift-ingress/e2e-mt-18bf762c-openshift-default-77b9fdd9b5-z7xxk"
Oct 31 01:28:20.331522 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:28:20.331496    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dsdcq\" (UniqueName: \"kubernetes.io/projected/7c25fc09-ee27-46c3-95fc-bfc6bacbc46f-kube-api-access-dsdcq\") pod \"e2e-mt-18bf762c-openshift-default-77b9fdd9b5-z7xxk\" (UID: \"7c25fc09-ee27-46c3-95fc-bfc6bacbc46f\") " pod="openshift-ingress/e2e-mt-18bf762c-openshift-default-77b9fdd9b5-z7xxk"
Oct 31 01:28:20.331706 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:28:20.331692    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c25fc09-ee27-46c3-95fc-bfc6bacbc46f-workload-certs\") pod \"e2e-mt-18bf762c-openshift-default-77b9fdd9b5-z7xxk\" (UID: \"7c25fc09-ee27-46c3-95fc-bfc6bacbc46f\") " pod="openshift-ingress/e2e-mt-18bf762c-openshift-default-77b9fdd9b5-z7xxk"
Oct 31 01:28:20.331817 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:28:20.331804    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7c25fc09-ee27-46c3-95fc-bfc6bacbc46f-istio-data\") pod \"e2e-mt-18bf762c-openshift-default-77b9fdd9b5-z7xxk\" (UID: \"7c25fc09-ee27-46c3-95fc-bfc6bacbc46f\") " pod="openshift-ingress/e2e-mt-18bf762c-openshift-default-77b9fdd9b5-z7xxk"
Oct 31 01:28:20.332088 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:28:20.332058    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c25fc09-ee27-46c3-95fc-bfc6bacbc46f-workload-socket\") pod \"e2e-mt-18bf762c-openshift-default-77b9fdd9b5-z7xxk\" (UID: \"7c25fc09-ee27-46c3-95fc-bfc6bacbc46f\") " pod="openshift-ingress/e2e-mt-18bf762c-openshift-default-77b9fdd9b5-z7xxk"
Oct 31 01:28:20.332254 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:28:20.332240    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7c25fc09-ee27-46c3-95fc-bfc6bacbc46f-istiod-ca-cert\") pod \"e2e-mt-18bf762c-openshift-default-77b9fdd9b5-z7xxk\" (UID: \"7c25fc09-ee27-46c3-95fc-bfc6bacbc46f\") " pod="openshift-ingress/e2e-mt-18bf762c-openshift-default-77b9fdd9b5-z7xxk"
Oct 31 01:28:20.332454 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:28:20.332438    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c25fc09-ee27-46c3-95fc-bfc6bacbc46f-credential-socket\") pod \"e2e-mt-18bf762c-openshift-default-77b9fdd9b5-z7xxk\" (UID: \"7c25fc09-ee27-46c3-95fc-bfc6bacbc46f\") " pod="openshift-ingress/e2e-mt-18bf762c-openshift-default-77b9fdd9b5-z7xxk"
Oct 31 01:28:20.332618 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:28:20.332603    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7c25fc09-ee27-46c3-95fc-bfc6bacbc46f-istio-podinfo\") pod \"e2e-mt-18bf762c-openshift-default-77b9fdd9b5-z7xxk\" (UID: \"7c25fc09-ee27-46c3-95fc-bfc6bacbc46f\") " pod="openshift-ingress/e2e-mt-18bf762c-openshift-default-77b9fdd9b5-z7xxk"
Oct 31 01:28:20.332817 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:28:20.332798    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7c25fc09-ee27-46c3-95fc-bfc6bacbc46f-istio-token\") pod \"e2e-mt-18bf762c-openshift-default-77b9fdd9b5-z7xxk\" (UID: \"7c25fc09-ee27-46c3-95fc-bfc6bacbc46f\") " pod="openshift-ingress/e2e-mt-18bf762c-openshift-default-77b9fdd9b5-z7xxk"
Oct 31 01:28:20.435414 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:28:20.435302    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c25fc09-ee27-46c3-95fc-bfc6bacbc46f-credential-socket\") pod \"e2e-mt-18bf762c-openshift-default-77b9fdd9b5-z7xxk\" (UID: \"7c25fc09-ee27-46c3-95fc-bfc6bacbc46f\") " pod="openshift-ingress/e2e-mt-18bf762c-openshift-default-77b9fdd9b5-z7xxk"
Oct 31 01:28:20.435414 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:28:20.435396    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7c25fc09-ee27-46c3-95fc-bfc6bacbc46f-istio-podinfo\") pod \"e2e-mt-18bf762c-openshift-default-77b9fdd9b5-z7xxk\" (UID: \"7c25fc09-ee27-46c3-95fc-bfc6bacbc46f\") " pod="openshift-ingress/e2e-mt-18bf762c-openshift-default-77b9fdd9b5-z7xxk"
Oct 31 01:28:20.435657 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:28:20.435428    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7c25fc09-ee27-46c3-95fc-bfc6bacbc46f-istio-token\") pod \"e2e-mt-18bf762c-openshift-default-77b9fdd9b5-z7xxk\" (UID: \"7c25fc09-ee27-46c3-95fc-bfc6bacbc46f\") " pod="openshift-ingress/e2e-mt-18bf762c-openshift-default-77b9fdd9b5-z7xxk"
Oct 31 01:28:20.435657 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:28:20.435555    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7c25fc09-ee27-46c3-95fc-bfc6bacbc46f-istio-envoy\") pod \"e2e-mt-18bf762c-openshift-default-77b9fdd9b5-z7xxk\" (UID: \"7c25fc09-ee27-46c3-95fc-bfc6bacbc46f\") " pod="openshift-ingress/e2e-mt-18bf762c-openshift-default-77b9fdd9b5-z7xxk"
Oct 31 01:28:20.435657 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:28:20.435586    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dsdcq\" (UniqueName: \"kubernetes.io/projected/7c25fc09-ee27-46c3-95fc-bfc6bacbc46f-kube-api-access-dsdcq\") pod \"e2e-mt-18bf762c-openshift-default-77b9fdd9b5-z7xxk\" (UID: \"7c25fc09-ee27-46c3-95fc-bfc6bacbc46f\") " pod="openshift-ingress/e2e-mt-18bf762c-openshift-default-77b9fdd9b5-z7xxk"
Oct 31 01:28:20.435657 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:28:20.435622    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c25fc09-ee27-46c3-95fc-bfc6bacbc46f-workload-certs\") pod \"e2e-mt-18bf762c-openshift-default-77b9fdd9b5-z7xxk\" (UID: \"7c25fc09-ee27-46c3-95fc-bfc6bacbc46f\") " pod="openshift-ingress/e2e-mt-18bf762c-openshift-default-77b9fdd9b5-z7xxk"
Oct 31 01:28:20.435657 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:28:20.435648    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7c25fc09-ee27-46c3-95fc-bfc6bacbc46f-istio-data\") pod \"e2e-mt-18bf762c-openshift-default-77b9fdd9b5-z7xxk\" (UID: \"7c25fc09-ee27-46c3-95fc-bfc6bacbc46f\") " pod="openshift-ingress/e2e-mt-18bf762c-openshift-default-77b9fdd9b5-z7xxk"
Oct 31 01:28:20.435910 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:28:20.435770    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c25fc09-ee27-46c3-95fc-bfc6bacbc46f-credential-socket\") pod \"e2e-mt-18bf762c-openshift-default-77b9fdd9b5-z7xxk\" (UID: \"7c25fc09-ee27-46c3-95fc-bfc6bacbc46f\") " pod="openshift-ingress/e2e-mt-18bf762c-openshift-default-77b9fdd9b5-z7xxk"
Oct 31 01:28:20.435910 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:28:20.435808    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c25fc09-ee27-46c3-95fc-bfc6bacbc46f-workload-socket\") pod \"e2e-mt-18bf762c-openshift-default-77b9fdd9b5-z7xxk\" (UID: \"7c25fc09-ee27-46c3-95fc-bfc6bacbc46f\") " pod="openshift-ingress/e2e-mt-18bf762c-openshift-default-77b9fdd9b5-z7xxk"
Oct 31 01:28:20.435910 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:28:20.435882    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7c25fc09-ee27-46c3-95fc-bfc6bacbc46f-istiod-ca-cert\") pod \"e2e-mt-18bf762c-openshift-default-77b9fdd9b5-z7xxk\" (UID: \"7c25fc09-ee27-46c3-95fc-bfc6bacbc46f\") " pod="openshift-ingress/e2e-mt-18bf762c-openshift-default-77b9fdd9b5-z7xxk"
Oct 31 01:28:20.436246 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:28:20.436220    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c25fc09-ee27-46c3-95fc-bfc6bacbc46f-workload-socket\") pod \"e2e-mt-18bf762c-openshift-default-77b9fdd9b5-z7xxk\" (UID: \"7c25fc09-ee27-46c3-95fc-bfc6bacbc46f\") " pod="openshift-ingress/e2e-mt-18bf762c-openshift-default-77b9fdd9b5-z7xxk"
Oct 31 01:28:20.436421 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:28:20.436299    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7c25fc09-ee27-46c3-95fc-bfc6bacbc46f-istio-data\") pod \"e2e-mt-18bf762c-openshift-default-77b9fdd9b5-z7xxk\" (UID: \"7c25fc09-ee27-46c3-95fc-bfc6bacbc46f\") " pod="openshift-ingress/e2e-mt-18bf762c-openshift-default-77b9fdd9b5-z7xxk"
Oct 31 01:28:20.436496 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:28:20.436427    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c25fc09-ee27-46c3-95fc-bfc6bacbc46f-workload-certs\") pod \"e2e-mt-18bf762c-openshift-default-77b9fdd9b5-z7xxk\" (UID: \"7c25fc09-ee27-46c3-95fc-bfc6bacbc46f\") " pod="openshift-ingress/e2e-mt-18bf762c-openshift-default-77b9fdd9b5-z7xxk"
Oct 31 01:28:20.436641 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:28:20.436621    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7c25fc09-ee27-46c3-95fc-bfc6bacbc46f-istiod-ca-cert\") pod \"e2e-mt-18bf762c-openshift-default-77b9fdd9b5-z7xxk\" (UID: \"7c25fc09-ee27-46c3-95fc-bfc6bacbc46f\") " pod="openshift-ingress/e2e-mt-18bf762c-openshift-default-77b9fdd9b5-z7xxk"
Oct 31 01:28:20.438465 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:28:20.438437    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7c25fc09-ee27-46c3-95fc-bfc6bacbc46f-istio-envoy\") pod \"e2e-mt-18bf762c-openshift-default-77b9fdd9b5-z7xxk\" (UID: \"7c25fc09-ee27-46c3-95fc-bfc6bacbc46f\") " pod="openshift-ingress/e2e-mt-18bf762c-openshift-default-77b9fdd9b5-z7xxk"
Oct 31 01:28:20.438656 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:28:20.438639    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7c25fc09-ee27-46c3-95fc-bfc6bacbc46f-istio-podinfo\") pod \"e2e-mt-18bf762c-openshift-default-77b9fdd9b5-z7xxk\" (UID: \"7c25fc09-ee27-46c3-95fc-bfc6bacbc46f\") " pod="openshift-ingress/e2e-mt-18bf762c-openshift-default-77b9fdd9b5-z7xxk"
Oct 31 01:28:20.493063 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:28:20.493027    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dsdcq\" (UniqueName: \"kubernetes.io/projected/7c25fc09-ee27-46c3-95fc-bfc6bacbc46f-kube-api-access-dsdcq\") pod \"e2e-mt-18bf762c-openshift-default-77b9fdd9b5-z7xxk\" (UID: \"7c25fc09-ee27-46c3-95fc-bfc6bacbc46f\") " pod="openshift-ingress/e2e-mt-18bf762c-openshift-default-77b9fdd9b5-z7xxk"
Oct 31 01:28:20.493483 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:28:20.493460    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7c25fc09-ee27-46c3-95fc-bfc6bacbc46f-istio-token\") pod \"e2e-mt-18bf762c-openshift-default-77b9fdd9b5-z7xxk\" (UID: \"7c25fc09-ee27-46c3-95fc-bfc6bacbc46f\") " pod="openshift-ingress/e2e-mt-18bf762c-openshift-default-77b9fdd9b5-z7xxk"
Oct 31 01:28:20.787399 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:28:20.787359    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-18bf762c-openshift-default-77b9fdd9b5-z7xxk"
Oct 31 01:28:21.228272 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:28:21.228241    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-18bf762c-openshift-default-77b9fdd9b5-z7xxk"]
Oct 31 01:28:21.229394 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:28:21.229366    4964 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c25fc09_ee27_46c3_95fc_bfc6bacbc46f.slice/crio-67208aa252abb8d485b62d47a0847a996cc12bb19fa4fba6c730b39f68d88e00 WatchSource:0}: Error finding container 67208aa252abb8d485b62d47a0847a996cc12bb19fa4fba6c730b39f68d88e00: Status 404 returned error can't find the container with id 67208aa252abb8d485b62d47a0847a996cc12bb19fa4fba6c730b39f68d88e00
Oct 31 01:28:21.232151 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:28:21.232121    4964 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 01:28:21.232256 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:28:21.232184    4964 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 01:28:21.232256 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:28:21.232232    4964 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 01:28:22.107854 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:28:22.107808    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-18bf762c-openshift-default-77b9fdd9b5-z7xxk" event={"ID":"7c25fc09-ee27-46c3-95fc-bfc6bacbc46f","Type":"ContainerStarted","Data":"c916abb8c68f7d0f205b477f82ba80edcb54699901ab78a106d6c0e02815c9fc"}
Oct 31 01:28:22.108031 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:28:22.107871    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-18bf762c-openshift-default-77b9fdd9b5-z7xxk" event={"ID":"7c25fc09-ee27-46c3-95fc-bfc6bacbc46f","Type":"ContainerStarted","Data":"67208aa252abb8d485b62d47a0847a996cc12bb19fa4fba6c730b39f68d88e00"}
Oct 31 01:28:22.275245 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:28:22.275198    4964 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-18bf762c-openshift-default-77b9fdd9b5-z7xxk" podStartSLOduration=2.27518317 podStartE2EDuration="2.27518317s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:28:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:28:22.273778363 +0000 UTC m=+1470.750344758" watchObservedRunningTime="2026-10-31 01:28:22.27518317 +0000 UTC m=+1470.751749550"
Oct 31 01:28:22.787950 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:28:22.787913    4964 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-18bf762c-openshift-default-77b9fdd9b5-z7xxk"
Oct 31 01:28:22.789950 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:28:22.789923    4964 patch_prober.go:29] interesting pod/e2e-mt-18bf762c-openshift-default-77b9fdd9b5-z7xxk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" start-of-body=
Oct 31 01:28:22.790109 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:28:22.789983    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-18bf762c-openshift-default-77b9fdd9b5-z7xxk" podUID="7c25fc09-ee27-46c3-95fc-bfc6bacbc46f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused"
Oct 31 01:28:23.516341 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:28:23.515119    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:28:23.788063 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:28:23.787981    4964 patch_prober.go:29] interesting pod/e2e-mt-18bf762c-openshift-default-77b9fdd9b5-z7xxk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" start-of-body=
Oct 31 01:28:23.788063 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:28:23.788040    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-18bf762c-openshift-default-77b9fdd9b5-z7xxk" podUID="7c25fc09-ee27-46c3-95fc-bfc6bacbc46f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused"
Oct 31 01:28:24.788004 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:28:24.787965    4964 patch_prober.go:29] interesting pod/e2e-mt-18bf762c-openshift-default-77b9fdd9b5-z7xxk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" start-of-body=
Oct 31 01:28:24.788440 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:28:24.788025    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-18bf762c-openshift-default-77b9fdd9b5-z7xxk" podUID="7c25fc09-ee27-46c3-95fc-bfc6bacbc46f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused"
Oct 31 01:28:25.788571 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:28:25.788538    4964 patch_prober.go:29] interesting pod/e2e-mt-18bf762c-openshift-default-77b9fdd9b5-z7xxk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" start-of-body=
Oct 31 01:28:25.788958 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:28:25.788597    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-18bf762c-openshift-default-77b9fdd9b5-z7xxk" podUID="7c25fc09-ee27-46c3-95fc-bfc6bacbc46f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused"
Oct 31 01:28:26.788471 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:28:26.788438    4964 patch_prober.go:29] interesting pod/e2e-mt-18bf762c-openshift-default-77b9fdd9b5-z7xxk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" start-of-body=
Oct 31 01:28:26.788760 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:28:26.788732    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-18bf762c-openshift-default-77b9fdd9b5-z7xxk" podUID="7c25fc09-ee27-46c3-95fc-bfc6bacbc46f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused"
Oct 31 01:28:27.788145 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:28:27.788111    4964 patch_prober.go:29] interesting pod/e2e-mt-18bf762c-openshift-default-77b9fdd9b5-z7xxk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" start-of-body=
Oct 31 01:28:27.788357 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:28:27.788173    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-18bf762c-openshift-default-77b9fdd9b5-z7xxk" podUID="7c25fc09-ee27-46c3-95fc-bfc6bacbc46f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused"
Oct 31 01:28:28.329600 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:28:28.329555    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:28:28.802457 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:28:28.798900    4964 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-18bf762c-openshift-default-77b9fdd9b5-z7xxk"
Oct 31 01:28:28.802457 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:28:28.800484    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-18bf762c-openshift-default-77b9fdd9b5-z7xxk"
Oct 31 01:28:28.802457 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:28:28.802353    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-18bf762c-openshift-default-77b9fdd9b5-z7xxk"
Oct 31 01:28:33.037166 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:28:33.036829    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:28:33.214825 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:28:33.214776    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:28:33.412845 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:28:33.412757    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:28:36.120920 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:28:36.120881    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:28:36.293099 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:28:36.293065    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:28:36.320535 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:28:36.320495    4964 patch_prober.go:29] interesting pod/e2e-api-a-7cec02-openshift-default-6fc456b49-jn9wm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 01:28:36.320778 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:28:36.320578    4964 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-7cec02-openshift-default-6fc456b49-jn9wm" podUID="45450e7a-2b62-4417-acca-d370251f6d54" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:28:37.997286 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:28:37.997249    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:28:38.307761 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:28:38.307724    4964 generic.go:356] "Generic (PLEG): container finished" podID="45450e7a-2b62-4417-acca-d370251f6d54" containerID="362bac2a8bd4ba033fd5150f6a3973611b86e493d849686055f98cf44a5b996a" exitCode=0
Oct 31 01:28:38.307933 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:28:38.307799    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-7cec02-openshift-default-6fc456b49-jn9wm" event={"ID":"45450e7a-2b62-4417-acca-d370251f6d54","Type":"ContainerDied","Data":"362bac2a8bd4ba033fd5150f6a3973611b86e493d849686055f98cf44a5b996a"}
Oct 31 01:28:38.308292 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:28:38.308276    4964 scope.go:122] "RemoveContainer" containerID="362bac2a8bd4ba033fd5150f6a3973611b86e493d849686055f98cf44a5b996a"
Oct 31 01:28:38.309928 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:28:38.309904    4964 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 01:28:38.310006 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:28:38.309970    4964 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 01:28:38.310006 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:28:38.310002    4964 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 01:28:39.332563 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:28:39.332521    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-7cec02-openshift-default-6fc456b49-jn9wm" event={"ID":"45450e7a-2b62-4417-acca-d370251f6d54","Type":"ContainerStarted","Data":"8c9a9da3abc3f800d88d817b91d0d19a6e8c67d550b7bf80bc28279b921e16f0"}
Oct 31 01:28:39.654596 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:28:39.654514    4964 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-7cec02-openshift-default-6fc456b49-jn9wm"
Oct 31 01:28:39.654780 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:28:39.654756    4964 patch_prober.go:29] interesting pod/e2e-api-a-7cec02-openshift-default-6fc456b49-jn9wm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" start-of-body=
Oct 31 01:28:39.654840 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:28:39.654813    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-7cec02-openshift-default-6fc456b49-jn9wm" podUID="45450e7a-2b62-4417-acca-d370251f6d54" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused"
Oct 31 01:28:40.655053 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:28:40.655022    4964 patch_prober.go:29] interesting pod/e2e-api-a-7cec02-openshift-default-6fc456b49-jn9wm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" start-of-body=
Oct 31 01:28:40.655432 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:28:40.655086    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-7cec02-openshift-default-6fc456b49-jn9wm" podUID="45450e7a-2b62-4417-acca-d370251f6d54" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused"
Oct 31 01:28:41.249631 ip-10-0-133-197 kubenswrapper[4964]: E1031 01:28:41.249586    4964 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q56x5" containerName="extractor"
Oct 31 01:28:41.654697 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:28:41.654615    4964 patch_prober.go:29] interesting pod/e2e-api-a-7cec02-openshift-default-6fc456b49-jn9wm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" start-of-body=
Oct 31 01:28:41.654697 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:28:41.654672    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-7cec02-openshift-default-6fc456b49-jn9wm" podUID="45450e7a-2b62-4417-acca-d370251f6d54" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused"
Oct 31 01:28:42.654762 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:28:42.654730    4964 patch_prober.go:29] interesting pod/e2e-api-a-7cec02-openshift-default-6fc456b49-jn9wm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" start-of-body=
Oct 31 01:28:42.655179 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:28:42.654784    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-7cec02-openshift-default-6fc456b49-jn9wm" podUID="45450e7a-2b62-4417-acca-d370251f6d54" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused"
Oct 31 01:28:42.821697 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:28:42.821660    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:28:43.655114 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:28:43.655023    4964 patch_prober.go:29] interesting pod/e2e-api-a-7cec02-openshift-default-6fc456b49-jn9wm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" start-of-body=
Oct 31 01:28:43.655516 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:28:43.655145    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-7cec02-openshift-default-6fc456b49-jn9wm" podUID="45450e7a-2b62-4417-acca-d370251f6d54" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused"
Oct 31 01:28:44.658876 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:28:44.658840    4964 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-7cec02-openshift-default-6fc456b49-jn9wm"
Oct 31 01:28:44.659366 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:28:44.659133    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-7cec02-openshift-default-6fc456b49-jn9wm"
Oct 31 01:28:44.659863 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:28:44.659846    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-7cec02-openshift-default-6fc456b49-jn9wm"
Oct 31 01:28:47.613511 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:28:47.613478    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:28:47.792083 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:28:47.792049    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:28:51.976628 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:28:51.976595    4964 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-9257n_f0c0ada7-7688-4dc2-97fc-813e4d0c4ef3/console-operator/2.log"
Oct 31 01:28:51.980218 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:28:51.980194    4964 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-9257n_f0c0ada7-7688-4dc2-97fc-813e4d0c4ef3/console-operator/2.log"
Oct 31 01:28:52.397828 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:28:52.397746    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:28:57.286843 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:28:57.286801    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:29:00.246827 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:29:00.246794    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:29:00.821518 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:29:00.821471    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:29:01.308283 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:29:01.308246    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:29:01.532811 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:29:01.532780    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:29:01.922438 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:29:01.922410    4964 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-42c96d-openshift-default-69697959d9-p8ljq"]
Oct 31 01:29:01.933038 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:29:01.933010    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-42c96d-openshift-default-69697959d9-p8ljq"
Oct 31 01:29:02.037810 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:29:02.026381    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/50f61fea-7b83-48aa-ad74-ea2be71f683b-istio-envoy\") pod \"e2e-api-a-42c96d-openshift-default-69697959d9-p8ljq\" (UID: \"50f61fea-7b83-48aa-ad74-ea2be71f683b\") " pod="openshift-ingress/e2e-api-a-42c96d-openshift-default-69697959d9-p8ljq"
Oct 31 01:29:02.037810 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:29:02.026525    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/50f61fea-7b83-48aa-ad74-ea2be71f683b-istio-data\") pod \"e2e-api-a-42c96d-openshift-default-69697959d9-p8ljq\" (UID: \"50f61fea-7b83-48aa-ad74-ea2be71f683b\") " pod="openshift-ingress/e2e-api-a-42c96d-openshift-default-69697959d9-p8ljq"
Oct 31 01:29:02.037810 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:29:02.026563    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/50f61fea-7b83-48aa-ad74-ea2be71f683b-workload-certs\") pod \"e2e-api-a-42c96d-openshift-default-69697959d9-p8ljq\" (UID: \"50f61fea-7b83-48aa-ad74-ea2be71f683b\") " pod="openshift-ingress/e2e-api-a-42c96d-openshift-default-69697959d9-p8ljq"
Oct 31 01:29:02.037810 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:29:02.026609    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/50f61fea-7b83-48aa-ad74-ea2be71f683b-workload-socket\") pod \"e2e-api-a-42c96d-openshift-default-69697959d9-p8ljq\" (UID: \"50f61fea-7b83-48aa-ad74-ea2be71f683b\") " pod="openshift-ingress/e2e-api-a-42c96d-openshift-default-69697959d9-p8ljq"
Oct 31 01:29:02.037810 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:29:02.026647    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/50f61fea-7b83-48aa-ad74-ea2be71f683b-istio-podinfo\") pod \"e2e-api-a-42c96d-openshift-default-69697959d9-p8ljq\" (UID: \"50f61fea-7b83-48aa-ad74-ea2be71f683b\") " pod="openshift-ingress/e2e-api-a-42c96d-openshift-default-69697959d9-p8ljq"
Oct 31 01:29:02.037810 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:29:02.026727    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/50f61fea-7b83-48aa-ad74-ea2be71f683b-credential-socket\") pod \"e2e-api-a-42c96d-openshift-default-69697959d9-p8ljq\" (UID: \"50f61fea-7b83-48aa-ad74-ea2be71f683b\") " pod="openshift-ingress/e2e-api-a-42c96d-openshift-default-69697959d9-p8ljq"
Oct 31 01:29:02.037810 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:29:02.026775    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h4xmz\" (UniqueName: \"kubernetes.io/projected/50f61fea-7b83-48aa-ad74-ea2be71f683b-kube-api-access-h4xmz\") pod \"e2e-api-a-42c96d-openshift-default-69697959d9-p8ljq\" (UID: \"50f61fea-7b83-48aa-ad74-ea2be71f683b\") " pod="openshift-ingress/e2e-api-a-42c96d-openshift-default-69697959d9-p8ljq"
Oct 31 01:29:02.037810 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:29:02.026810    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/50f61fea-7b83-48aa-ad74-ea2be71f683b-istio-token\") pod \"e2e-api-a-42c96d-openshift-default-69697959d9-p8ljq\" (UID: \"50f61fea-7b83-48aa-ad74-ea2be71f683b\") " pod="openshift-ingress/e2e-api-a-42c96d-openshift-default-69697959d9-p8ljq"
Oct 31 01:29:02.037810 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:29:02.026928    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/50f61fea-7b83-48aa-ad74-ea2be71f683b-istiod-ca-cert\") pod \"e2e-api-a-42c96d-openshift-default-69697959d9-p8ljq\" (UID: \"50f61fea-7b83-48aa-ad74-ea2be71f683b\") " pod="openshift-ingress/e2e-api-a-42c96d-openshift-default-69697959d9-p8ljq"
Oct 31 01:29:02.076737 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:29:02.076708    4964 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-42c96d-openshift-default-dockercfg-252ck\""
Oct 31 01:29:02.091586 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:29:02.089996    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:29:02.104380 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:29:02.095361    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-42c96d-openshift-default-69697959d9-p8ljq"]
Oct 31 01:29:02.131150 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:29:02.128153    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/50f61fea-7b83-48aa-ad74-ea2be71f683b-istio-data\") pod \"e2e-api-a-42c96d-openshift-default-69697959d9-p8ljq\" (UID: \"50f61fea-7b83-48aa-ad74-ea2be71f683b\") " pod="openshift-ingress/e2e-api-a-42c96d-openshift-default-69697959d9-p8ljq"
Oct 31 01:29:02.131150 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:29:02.128221    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/50f61fea-7b83-48aa-ad74-ea2be71f683b-workload-certs\") pod \"e2e-api-a-42c96d-openshift-default-69697959d9-p8ljq\" (UID: \"50f61fea-7b83-48aa-ad74-ea2be71f683b\") " pod="openshift-ingress/e2e-api-a-42c96d-openshift-default-69697959d9-p8ljq"
Oct 31 01:29:02.131150 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:29:02.128270    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/50f61fea-7b83-48aa-ad74-ea2be71f683b-workload-socket\") pod \"e2e-api-a-42c96d-openshift-default-69697959d9-p8ljq\" (UID: \"50f61fea-7b83-48aa-ad74-ea2be71f683b\") " pod="openshift-ingress/e2e-api-a-42c96d-openshift-default-69697959d9-p8ljq"
Oct 31 01:29:02.131150 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:29:02.128332    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/50f61fea-7b83-48aa-ad74-ea2be71f683b-istio-podinfo\") pod \"e2e-api-a-42c96d-openshift-default-69697959d9-p8ljq\" (UID: \"50f61fea-7b83-48aa-ad74-ea2be71f683b\") " pod="openshift-ingress/e2e-api-a-42c96d-openshift-default-69697959d9-p8ljq"
Oct 31 01:29:02.131150 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:29:02.128417    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/50f61fea-7b83-48aa-ad74-ea2be71f683b-credential-socket\") pod \"e2e-api-a-42c96d-openshift-default-69697959d9-p8ljq\" (UID: \"50f61fea-7b83-48aa-ad74-ea2be71f683b\") " pod="openshift-ingress/e2e-api-a-42c96d-openshift-default-69697959d9-p8ljq"
Oct 31 01:29:02.131150 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:29:02.128465    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h4xmz\" (UniqueName: \"kubernetes.io/projected/50f61fea-7b83-48aa-ad74-ea2be71f683b-kube-api-access-h4xmz\") pod \"e2e-api-a-42c96d-openshift-default-69697959d9-p8ljq\" (UID: \"50f61fea-7b83-48aa-ad74-ea2be71f683b\") " pod="openshift-ingress/e2e-api-a-42c96d-openshift-default-69697959d9-p8ljq"
Oct 31 01:29:02.131150 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:29:02.128497    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/50f61fea-7b83-48aa-ad74-ea2be71f683b-istio-token\") pod \"e2e-api-a-42c96d-openshift-default-69697959d9-p8ljq\" (UID: \"50f61fea-7b83-48aa-ad74-ea2be71f683b\") " pod="openshift-ingress/e2e-api-a-42c96d-openshift-default-69697959d9-p8ljq"
Oct 31 01:29:02.131150 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:29:02.128945    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/50f61fea-7b83-48aa-ad74-ea2be71f683b-workload-certs\") pod \"e2e-api-a-42c96d-openshift-default-69697959d9-p8ljq\" (UID: \"50f61fea-7b83-48aa-ad74-ea2be71f683b\") " pod="openshift-ingress/e2e-api-a-42c96d-openshift-default-69697959d9-p8ljq"
Oct 31 01:29:02.131150 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:29:02.129098    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/50f61fea-7b83-48aa-ad74-ea2be71f683b-istiod-ca-cert\") pod \"e2e-api-a-42c96d-openshift-default-69697959d9-p8ljq\" (UID: \"50f61fea-7b83-48aa-ad74-ea2be71f683b\") " pod="openshift-ingress/e2e-api-a-42c96d-openshift-default-69697959d9-p8ljq"
Oct 31 01:29:02.131150 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:29:02.129183    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/50f61fea-7b83-48aa-ad74-ea2be71f683b-istio-data\") pod \"e2e-api-a-42c96d-openshift-default-69697959d9-p8ljq\" (UID: \"50f61fea-7b83-48aa-ad74-ea2be71f683b\") " pod="openshift-ingress/e2e-api-a-42c96d-openshift-default-69697959d9-p8ljq"
Oct 31 01:29:02.131150 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:29:02.129199    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/50f61fea-7b83-48aa-ad74-ea2be71f683b-istio-envoy\") pod \"e2e-api-a-42c96d-openshift-default-69697959d9-p8ljq\" (UID: \"50f61fea-7b83-48aa-ad74-ea2be71f683b\") " pod="openshift-ingress/e2e-api-a-42c96d-openshift-default-69697959d9-p8ljq"
Oct 31 01:29:02.131150 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:29:02.129467    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/50f61fea-7b83-48aa-ad74-ea2be71f683b-workload-socket\") pod \"e2e-api-a-42c96d-openshift-default-69697959d9-p8ljq\" (UID: \"50f61fea-7b83-48aa-ad74-ea2be71f683b\") " pod="openshift-ingress/e2e-api-a-42c96d-openshift-default-69697959d9-p8ljq"
Oct 31 01:29:02.131150 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:29:02.129971    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/50f61fea-7b83-48aa-ad74-ea2be71f683b-credential-socket\") pod \"e2e-api-a-42c96d-openshift-default-69697959d9-p8ljq\" (UID: \"50f61fea-7b83-48aa-ad74-ea2be71f683b\") " pod="openshift-ingress/e2e-api-a-42c96d-openshift-default-69697959d9-p8ljq"
Oct 31 01:29:02.131150 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:29:02.130650    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/50f61fea-7b83-48aa-ad74-ea2be71f683b-istiod-ca-cert\") pod \"e2e-api-a-42c96d-openshift-default-69697959d9-p8ljq\" (UID: \"50f61fea-7b83-48aa-ad74-ea2be71f683b\") " pod="openshift-ingress/e2e-api-a-42c96d-openshift-default-69697959d9-p8ljq"
Oct 31 01:29:02.137852 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:29:02.137827    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/50f61fea-7b83-48aa-ad74-ea2be71f683b-istio-podinfo\") pod \"e2e-api-a-42c96d-openshift-default-69697959d9-p8ljq\" (UID: \"50f61fea-7b83-48aa-ad74-ea2be71f683b\") " pod="openshift-ingress/e2e-api-a-42c96d-openshift-default-69697959d9-p8ljq"
Oct 31 01:29:02.138206 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:29:02.138188    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/50f61fea-7b83-48aa-ad74-ea2be71f683b-istio-envoy\") pod \"e2e-api-a-42c96d-openshift-default-69697959d9-p8ljq\" (UID: \"50f61fea-7b83-48aa-ad74-ea2be71f683b\") " pod="openshift-ingress/e2e-api-a-42c96d-openshift-default-69697959d9-p8ljq"
Oct 31 01:29:02.186994 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:29:02.186923    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/50f61fea-7b83-48aa-ad74-ea2be71f683b-istio-token\") pod \"e2e-api-a-42c96d-openshift-default-69697959d9-p8ljq\" (UID: \"50f61fea-7b83-48aa-ad74-ea2be71f683b\") " pod="openshift-ingress/e2e-api-a-42c96d-openshift-default-69697959d9-p8ljq"
Oct 31 01:29:02.187122 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:29:02.187088    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h4xmz\" (UniqueName: \"kubernetes.io/projected/50f61fea-7b83-48aa-ad74-ea2be71f683b-kube-api-access-h4xmz\") pod \"e2e-api-a-42c96d-openshift-default-69697959d9-p8ljq\" (UID: \"50f61fea-7b83-48aa-ad74-ea2be71f683b\") " pod="openshift-ingress/e2e-api-a-42c96d-openshift-default-69697959d9-p8ljq"
Oct 31 01:29:02.251275 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:29:02.251240    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-42c96d-openshift-default-69697959d9-p8ljq"
Oct 31 01:29:02.280973 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:29:02.280715    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:29:02.530454 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:29:02.530406    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:29:02.774872 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:29:02.774836    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-42c96d-openshift-default-69697959d9-p8ljq"]
Oct 31 01:29:02.775416 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:29:02.775387    4964 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50f61fea_7b83_48aa_ad74_ea2be71f683b.slice/crio-3d61406606430beaf52c28bf589ca2f21fbe0b8c1b53cdcb6a730f1a899e1af5 WatchSource:0}: Error finding container 3d61406606430beaf52c28bf589ca2f21fbe0b8c1b53cdcb6a730f1a899e1af5: Status 404 returned error can't find the container with id 3d61406606430beaf52c28bf589ca2f21fbe0b8c1b53cdcb6a730f1a899e1af5
Oct 31 01:29:02.777882 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:29:02.777850    4964 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 01:29:02.777997 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:29:02.777912    4964 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 01:29:02.777997 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:29:02.777944    4964 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 01:29:03.055101 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:29:03.052421    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:29:03.551056 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:29:03.551017    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-42c96d-openshift-default-69697959d9-p8ljq" event={"ID":"50f61fea-7b83-48aa-ad74-ea2be71f683b","Type":"ContainerStarted","Data":"9eeac5cf406f68573cb3279b07b12b7f3068677631a160ef21978be114b06a04"}
Oct 31 01:29:03.551541 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:29:03.551079    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-42c96d-openshift-default-69697959d9-p8ljq" event={"ID":"50f61fea-7b83-48aa-ad74-ea2be71f683b","Type":"ContainerStarted","Data":"3d61406606430beaf52c28bf589ca2f21fbe0b8c1b53cdcb6a730f1a899e1af5"}
Oct 31 01:29:03.671746 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:29:03.671697    4964 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-42c96d-openshift-default-69697959d9-p8ljq" podStartSLOduration=2.671682993 podStartE2EDuration="2.671682993s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:29:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:29:03.667423502 +0000 UTC m=+1512.143989882" watchObservedRunningTime="2026-10-31 01:29:03.671682993 +0000 UTC m=+1512.148249374"
Oct 31 01:29:04.251637 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:29:04.251602    4964 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-42c96d-openshift-default-69697959d9-p8ljq"
Oct 31 01:29:04.257595 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:29:04.257569    4964 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-42c96d-openshift-default-69697959d9-p8ljq"
Oct 31 01:29:04.560627 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:29:04.560558    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-42c96d-openshift-default-69697959d9-p8ljq"
Oct 31 01:29:04.561472 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:29:04.561449    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-42c96d-openshift-default-69697959d9-p8ljq"
Oct 31 01:29:06.528877 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:29:06.528846    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:29:06.755357 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:29:06.755303    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:29:07.753068 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:29:07.753027    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:29:08.587897 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:29:08.587866    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:29:09.768912 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:29:09.768871    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:29:10.006036 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:29:10.005998    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:29:10.557486 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:29:10.557455    4964 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-42c96d-65764fbd8c-84plq"]
Oct 31 01:29:10.577811 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:29:10.577655    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-42c96d-65764fbd8c-84plq"
Oct 31 01:29:10.596562 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:29:10.596533    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-42c96d-65764fbd8c-84plq"]
Oct 31 01:29:10.673279 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:29:10.673247    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4zj4c\" (UniqueName: \"kubernetes.io/projected/065217f5-7fda-424b-a2e7-fed363588c32-kube-api-access-4zj4c\") pod \"payload-pre-processing-e2e-api-a-42c96d-65764fbd8c-84plq\" (UID: \"065217f5-7fda-424b-a2e7-fed363588c32\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-42c96d-65764fbd8c-84plq"
Oct 31 01:29:10.673501 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:29:10.673435    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/065217f5-7fda-424b-a2e7-fed363588c32-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-42c96d-65764fbd8c-84plq\" (UID: \"065217f5-7fda-424b-a2e7-fed363588c32\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-42c96d-65764fbd8c-84plq"
Oct 31 01:29:10.681355 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:29:10.681311    4964 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-42c96d\""
Oct 31 01:29:10.681470 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:29:10.681351    4964 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-42c96d-dockercfg-lgr5w\""
Oct 31 01:29:10.774625 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:29:10.774597    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/065217f5-7fda-424b-a2e7-fed363588c32-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-42c96d-65764fbd8c-84plq\" (UID: \"065217f5-7fda-424b-a2e7-fed363588c32\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-42c96d-65764fbd8c-84plq"
Oct 31 01:29:10.775006 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:29:10.774767    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4zj4c\" (UniqueName: \"kubernetes.io/projected/065217f5-7fda-424b-a2e7-fed363588c32-kube-api-access-4zj4c\") pod \"payload-pre-processing-e2e-api-a-42c96d-65764fbd8c-84plq\" (UID: \"065217f5-7fda-424b-a2e7-fed363588c32\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-42c96d-65764fbd8c-84plq"
Oct 31 01:29:10.775333 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:29:10.775299    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/065217f5-7fda-424b-a2e7-fed363588c32-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-42c96d-65764fbd8c-84plq\" (UID: \"065217f5-7fda-424b-a2e7-fed363588c32\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-42c96d-65764fbd8c-84plq"
Oct 31 01:29:10.831494 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:29:10.831432    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4zj4c\" (UniqueName: \"kubernetes.io/projected/065217f5-7fda-424b-a2e7-fed363588c32-kube-api-access-4zj4c\") pod \"payload-pre-processing-e2e-api-a-42c96d-65764fbd8c-84plq\" (UID: \"065217f5-7fda-424b-a2e7-fed363588c32\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-42c96d-65764fbd8c-84plq"
Oct 31 01:29:10.915721 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:29:10.915686    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-42c96d-65764fbd8c-84plq"
Oct 31 01:29:11.238654 ip-10-0-133-197 kubenswrapper[4964]: E1031 01:29:11.238616    4964 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q56x5" containerName="extractor"
Oct 31 01:29:11.306812 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:29:11.306781    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-42c96d-65764fbd8c-84plq"]
Oct 31 01:29:11.307001 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:29:11.306979    4964 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod065217f5_7fda_424b_a2e7_fed363588c32.slice/crio-5a6d7191b21c05ffab402d134bcaa5fef64d724daad9a1e0399c552c0a63ab76 WatchSource:0}: Error finding container 5a6d7191b21c05ffab402d134bcaa5fef64d724daad9a1e0399c552c0a63ab76: Status 404 returned error can't find the container with id 5a6d7191b21c05ffab402d134bcaa5fef64d724daad9a1e0399c552c0a63ab76
Oct 31 01:29:11.626097 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:29:11.626020    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-42c96d-65764fbd8c-84plq" event={"ID":"065217f5-7fda-424b-a2e7-fed363588c32","Type":"ContainerStarted","Data":"426c3d564ffe278eee6386373d24d17b3d168df9b492f545e353a1d68a07d042"}
Oct 31 01:29:11.626097 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:29:11.626065    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-42c96d-65764fbd8c-84plq" event={"ID":"065217f5-7fda-424b-a2e7-fed363588c32","Type":"ContainerStarted","Data":"5a6d7191b21c05ffab402d134bcaa5fef64d724daad9a1e0399c552c0a63ab76"}
Oct 31 01:29:11.626284 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:29:11.626129    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-42c96d-65764fbd8c-84plq"
Oct 31 01:29:11.744057 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:29:11.744001    4964 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-42c96d-65764fbd8c-84plq" podStartSLOduration=1.743988818 podStartE2EDuration="1.743988818s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:29:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:29:11.741768963 +0000 UTC m=+1520.218335344" watchObservedRunningTime="2026-10-31 01:29:11.743988818 +0000 UTC m=+1520.220555217"
Oct 31 01:29:12.824067 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:29:12.824035    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:29:13.206294 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:29:13.206208    4964 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-18bf762c-6cc59897d9-w4n2s"]
Oct 31 01:29:13.319916 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:29:13.319858    4964 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-18bf762c-c84588867-fcv87"]
Oct 31 01:29:13.612206 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:29:13.612168    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:29:13.793019 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:29:13.792972    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:29:14.214390 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:29:14.214360    4964 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-e7eea9-openshift-default-76c54449f5-c6bvv"]
Oct 31 01:29:14.218894 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:29:14.218877    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-e7eea9-openshift-default-76c54449f5-c6bvv"
Oct 31 01:29:14.273160 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:29:14.273128    4964 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-e7eea9-openshift-default-dockercfg-78glq\""
Oct 31 01:29:14.342086 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:29:14.342051    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eb13aca8-a24f-4fc8-a51f-3d3f45d739ec-istio-token\") pod \"e2e-api-b-e7eea9-openshift-default-76c54449f5-c6bvv\" (UID: \"eb13aca8-a24f-4fc8-a51f-3d3f45d739ec\") " pod="openshift-ingress/e2e-api-b-e7eea9-openshift-default-76c54449f5-c6bvv"
Oct 31 01:29:14.342369 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:29:14.342343    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eb13aca8-a24f-4fc8-a51f-3d3f45d739ec-istiod-ca-cert\") pod \"e2e-api-b-e7eea9-openshift-default-76c54449f5-c6bvv\" (UID: \"eb13aca8-a24f-4fc8-a51f-3d3f45d739ec\") " pod="openshift-ingress/e2e-api-b-e7eea9-openshift-default-76c54449f5-c6bvv"
Oct 31 01:29:14.342573 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:29:14.342555    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eb13aca8-a24f-4fc8-a51f-3d3f45d739ec-istio-envoy\") pod \"e2e-api-b-e7eea9-openshift-default-76c54449f5-c6bvv\" (UID: \"eb13aca8-a24f-4fc8-a51f-3d3f45d739ec\") " pod="openshift-ingress/e2e-api-b-e7eea9-openshift-default-76c54449f5-c6bvv"
Oct 31 01:29:14.342789 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:29:14.342768    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb13aca8-a24f-4fc8-a51f-3d3f45d739ec-credential-socket\") pod \"e2e-api-b-e7eea9-openshift-default-76c54449f5-c6bvv\" (UID: \"eb13aca8-a24f-4fc8-a51f-3d3f45d739ec\") " pod="openshift-ingress/e2e-api-b-e7eea9-openshift-default-76c54449f5-c6bvv"
Oct 31 01:29:14.342923 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:29:14.342907    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r6cvp\" (UniqueName: \"kubernetes.io/projected/eb13aca8-a24f-4fc8-a51f-3d3f45d739ec-kube-api-access-r6cvp\") pod \"e2e-api-b-e7eea9-openshift-default-76c54449f5-c6bvv\" (UID: \"eb13aca8-a24f-4fc8-a51f-3d3f45d739ec\") " pod="openshift-ingress/e2e-api-b-e7eea9-openshift-default-76c54449f5-c6bvv"
Oct 31 01:29:14.343122 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:29:14.343103    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eb13aca8-a24f-4fc8-a51f-3d3f45d739ec-istio-data\") pod \"e2e-api-b-e7eea9-openshift-default-76c54449f5-c6bvv\" (UID: \"eb13aca8-a24f-4fc8-a51f-3d3f45d739ec\") " pod="openshift-ingress/e2e-api-b-e7eea9-openshift-default-76c54449f5-c6bvv"
Oct 31 01:29:14.343297 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:29:14.343282    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eb13aca8-a24f-4fc8-a51f-3d3f45d739ec-istio-podinfo\") pod \"e2e-api-b-e7eea9-openshift-default-76c54449f5-c6bvv\" (UID: \"eb13aca8-a24f-4fc8-a51f-3d3f45d739ec\") " pod="openshift-ingress/e2e-api-b-e7eea9-openshift-default-76c54449f5-c6bvv"
Oct 31 01:29:14.343506 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:29:14.343491    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb13aca8-a24f-4fc8-a51f-3d3f45d739ec-workload-socket\") pod \"e2e-api-b-e7eea9-openshift-default-76c54449f5-c6bvv\" (UID: \"eb13aca8-a24f-4fc8-a51f-3d3f45d739ec\") " pod="openshift-ingress/e2e-api-b-e7eea9-openshift-default-76c54449f5-c6bvv"
Oct 31 01:29:14.343636 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:29:14.343623    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eb13aca8-a24f-4fc8-a51f-3d3f45d739ec-workload-certs\") pod \"e2e-api-b-e7eea9-openshift-default-76c54449f5-c6bvv\" (UID: \"eb13aca8-a24f-4fc8-a51f-3d3f45d739ec\") " pod="openshift-ingress/e2e-api-b-e7eea9-openshift-default-76c54449f5-c6bvv"
Oct 31 01:29:14.347395 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:29:14.347370    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-e7eea9-openshift-default-76c54449f5-c6bvv"]
Oct 31 01:29:14.453247 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:29:14.453217    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eb13aca8-a24f-4fc8-a51f-3d3f45d739ec-istio-podinfo\") pod \"e2e-api-b-e7eea9-openshift-default-76c54449f5-c6bvv\" (UID: \"eb13aca8-a24f-4fc8-a51f-3d3f45d739ec\") " pod="openshift-ingress/e2e-api-b-e7eea9-openshift-default-76c54449f5-c6bvv"
Oct 31 01:29:14.453505 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:29:14.453299    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb13aca8-a24f-4fc8-a51f-3d3f45d739ec-workload-socket\") pod \"e2e-api-b-e7eea9-openshift-default-76c54449f5-c6bvv\" (UID: \"eb13aca8-a24f-4fc8-a51f-3d3f45d739ec\") " pod="openshift-ingress/e2e-api-b-e7eea9-openshift-default-76c54449f5-c6bvv"
Oct 31 01:29:14.453505 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:29:14.453376    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eb13aca8-a24f-4fc8-a51f-3d3f45d739ec-workload-certs\") pod \"e2e-api-b-e7eea9-openshift-default-76c54449f5-c6bvv\" (UID: \"eb13aca8-a24f-4fc8-a51f-3d3f45d739ec\") " pod="openshift-ingress/e2e-api-b-e7eea9-openshift-default-76c54449f5-c6bvv"
Oct 31 01:29:14.453505 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:29:14.453443    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eb13aca8-a24f-4fc8-a51f-3d3f45d739ec-istio-token\") pod \"e2e-api-b-e7eea9-openshift-default-76c54449f5-c6bvv\" (UID: \"eb13aca8-a24f-4fc8-a51f-3d3f45d739ec\") " pod="openshift-ingress/e2e-api-b-e7eea9-openshift-default-76c54449f5-c6bvv"
Oct 31 01:29:14.453505 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:29:14.453469    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eb13aca8-a24f-4fc8-a51f-3d3f45d739ec-istiod-ca-cert\") pod \"e2e-api-b-e7eea9-openshift-default-76c54449f5-c6bvv\" (UID: \"eb13aca8-a24f-4fc8-a51f-3d3f45d739ec\") " pod="openshift-ingress/e2e-api-b-e7eea9-openshift-default-76c54449f5-c6bvv"
Oct 31 01:29:14.453711 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:29:14.453538    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eb13aca8-a24f-4fc8-a51f-3d3f45d739ec-istio-envoy\") pod \"e2e-api-b-e7eea9-openshift-default-76c54449f5-c6bvv\" (UID: \"eb13aca8-a24f-4fc8-a51f-3d3f45d739ec\") " pod="openshift-ingress/e2e-api-b-e7eea9-openshift-default-76c54449f5-c6bvv"
Oct 31 01:29:14.453711 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:29:14.453640    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb13aca8-a24f-4fc8-a51f-3d3f45d739ec-credential-socket\") pod \"e2e-api-b-e7eea9-openshift-default-76c54449f5-c6bvv\" (UID: \"eb13aca8-a24f-4fc8-a51f-3d3f45d739ec\") " pod="openshift-ingress/e2e-api-b-e7eea9-openshift-default-76c54449f5-c6bvv"
Oct 31 01:29:14.453711 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:29:14.453675    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r6cvp\" (UniqueName: \"kubernetes.io/projected/eb13aca8-a24f-4fc8-a51f-3d3f45d739ec-kube-api-access-r6cvp\") pod \"e2e-api-b-e7eea9-openshift-default-76c54449f5-c6bvv\" (UID: \"eb13aca8-a24f-4fc8-a51f-3d3f45d739ec\") " pod="openshift-ingress/e2e-api-b-e7eea9-openshift-default-76c54449f5-c6bvv"
Oct 31 01:29:14.453865 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:29:14.453782    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eb13aca8-a24f-4fc8-a51f-3d3f45d739ec-istio-data\") pod \"e2e-api-b-e7eea9-openshift-default-76c54449f5-c6bvv\" (UID: \"eb13aca8-a24f-4fc8-a51f-3d3f45d739ec\") " pod="openshift-ingress/e2e-api-b-e7eea9-openshift-default-76c54449f5-c6bvv"
Oct 31 01:29:14.456201 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:29:14.454398    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb13aca8-a24f-4fc8-a51f-3d3f45d739ec-workload-socket\") pod \"e2e-api-b-e7eea9-openshift-default-76c54449f5-c6bvv\" (UID: \"eb13aca8-a24f-4fc8-a51f-3d3f45d739ec\") " pod="openshift-ingress/e2e-api-b-e7eea9-openshift-default-76c54449f5-c6bvv"
Oct 31 01:29:14.456201 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:29:14.454543    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eb13aca8-a24f-4fc8-a51f-3d3f45d739ec-workload-certs\") pod \"e2e-api-b-e7eea9-openshift-default-76c54449f5-c6bvv\" (UID: \"eb13aca8-a24f-4fc8-a51f-3d3f45d739ec\") " pod="openshift-ingress/e2e-api-b-e7eea9-openshift-default-76c54449f5-c6bvv"
Oct 31 01:29:14.456201 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:29:14.455658    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eb13aca8-a24f-4fc8-a51f-3d3f45d739ec-istiod-ca-cert\") pod \"e2e-api-b-e7eea9-openshift-default-76c54449f5-c6bvv\" (UID: \"eb13aca8-a24f-4fc8-a51f-3d3f45d739ec\") " pod="openshift-ingress/e2e-api-b-e7eea9-openshift-default-76c54449f5-c6bvv"
Oct 31 01:29:14.456201 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:29:14.455919    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb13aca8-a24f-4fc8-a51f-3d3f45d739ec-credential-socket\") pod \"e2e-api-b-e7eea9-openshift-default-76c54449f5-c6bvv\" (UID: \"eb13aca8-a24f-4fc8-a51f-3d3f45d739ec\") " pod="openshift-ingress/e2e-api-b-e7eea9-openshift-default-76c54449f5-c6bvv"
Oct 31 01:29:14.481288 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:29:14.481230    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eb13aca8-a24f-4fc8-a51f-3d3f45d739ec-istio-podinfo\") pod \"e2e-api-b-e7eea9-openshift-default-76c54449f5-c6bvv\" (UID: \"eb13aca8-a24f-4fc8-a51f-3d3f45d739ec\") " pod="openshift-ingress/e2e-api-b-e7eea9-openshift-default-76c54449f5-c6bvv"
Oct 31 01:29:14.482263 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:29:14.481652    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eb13aca8-a24f-4fc8-a51f-3d3f45d739ec-istio-data\") pod \"e2e-api-b-e7eea9-openshift-default-76c54449f5-c6bvv\" (UID: \"eb13aca8-a24f-4fc8-a51f-3d3f45d739ec\") " pod="openshift-ingress/e2e-api-b-e7eea9-openshift-default-76c54449f5-c6bvv"
Oct 31 01:29:14.484996 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:29:14.484940    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eb13aca8-a24f-4fc8-a51f-3d3f45d739ec-istio-envoy\") pod \"e2e-api-b-e7eea9-openshift-default-76c54449f5-c6bvv\" (UID: \"eb13aca8-a24f-4fc8-a51f-3d3f45d739ec\") " pod="openshift-ingress/e2e-api-b-e7eea9-openshift-default-76c54449f5-c6bvv"
Oct 31 01:29:14.503206 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:29:14.503165    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:29:14.510298 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:29:14.510253    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eb13aca8-a24f-4fc8-a51f-3d3f45d739ec-istio-token\") pod \"e2e-api-b-e7eea9-openshift-default-76c54449f5-c6bvv\" (UID: \"eb13aca8-a24f-4fc8-a51f-3d3f45d739ec\") " pod="openshift-ingress/e2e-api-b-e7eea9-openshift-default-76c54449f5-c6bvv"
Oct 31 01:29:14.514198 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:29:14.514161    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r6cvp\" (UniqueName: \"kubernetes.io/projected/eb13aca8-a24f-4fc8-a51f-3d3f45d739ec-kube-api-access-r6cvp\") pod \"e2e-api-b-e7eea9-openshift-default-76c54449f5-c6bvv\" (UID: \"eb13aca8-a24f-4fc8-a51f-3d3f45d739ec\") " pod="openshift-ingress/e2e-api-b-e7eea9-openshift-default-76c54449f5-c6bvv"
Oct 31 01:29:14.534494 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:29:14.534225    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-e7eea9-openshift-default-76c54449f5-c6bvv"
Oct 31 01:29:14.947298 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:29:14.947209    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:29:15.028719 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:29:15.020408    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-e7eea9-openshift-default-76c54449f5-c6bvv"]
Oct 31 01:29:15.031855 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:29:15.031161    4964 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb13aca8_a24f_4fc8_a51f_3d3f45d739ec.slice/crio-fe310b73fc4bb80eac8e38e3b45b33921d8dcd5293fbc5f35da5fb1730974a7e WatchSource:0}: Error finding container fe310b73fc4bb80eac8e38e3b45b33921d8dcd5293fbc5f35da5fb1730974a7e: Status 404 returned error can't find the container with id fe310b73fc4bb80eac8e38e3b45b33921d8dcd5293fbc5f35da5fb1730974a7e
Oct 31 01:29:15.033914 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:29:15.033867    4964 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 01:29:15.034037 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:29:15.033959    4964 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 01:29:15.034037 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:29:15.034003    4964 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 01:29:15.719924 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:29:15.719890    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-e7eea9-openshift-default-76c54449f5-c6bvv" event={"ID":"eb13aca8-a24f-4fc8-a51f-3d3f45d739ec","Type":"ContainerStarted","Data":"1136ce451c0dddd2c9cbcfcb5f38288098a69c45635846edaf608507b1c4cff3"}
Oct 31 01:29:15.720306 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:29:15.719952    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-e7eea9-openshift-default-76c54449f5-c6bvv" event={"ID":"eb13aca8-a24f-4fc8-a51f-3d3f45d739ec","Type":"ContainerStarted","Data":"fe310b73fc4bb80eac8e38e3b45b33921d8dcd5293fbc5f35da5fb1730974a7e"}
Oct 31 01:29:15.936420 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:29:15.936370    4964 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-e7eea9-openshift-default-76c54449f5-c6bvv" podStartSLOduration=1.9363561740000002 podStartE2EDuration="1.936356174s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:29:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:29:15.885273625 +0000 UTC m=+1524.361840007" watchObservedRunningTime="2026-10-31 01:29:15.936356174 +0000 UTC m=+1524.412922642"
Oct 31 01:29:16.535158 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:29:16.535118    4964 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-e7eea9-openshift-default-76c54449f5-c6bvv"
Oct 31 01:29:16.540999 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:29:16.540973    4964 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-e7eea9-openshift-default-76c54449f5-c6bvv"
Oct 31 01:29:16.732570 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:29:16.732533    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-e7eea9-openshift-default-76c54449f5-c6bvv"
Oct 31 01:29:16.733946 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:29:16.733906    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-e7eea9-openshift-default-76c54449f5-c6bvv"
Oct 31 01:29:17.615073 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:29:17.615035    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:29:18.207271 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:29:18.207219    4964 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-18bf762c-6cc59897d9-w4n2s" podUID="4a70412f-a2ba-4f8c-896d-945e58e057db" containerName="payload-processing" containerID="cri-o://0d826f90a037e14dcd2b85c2a53a3d1ea5ce5e08644cbdb45fa9fa8577e665da" gracePeriod=25
Oct 31 01:29:18.321279 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:29:18.321204    4964 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-18bf762c-c84588867-fcv87" podUID="74f9ca59-18e2-4fb6-b596-08f86168bb39" containerName="payload-pre-processing" containerID="cri-o://d686c8d947f75ed277c24a533e1078c025f9571cd00b15757a1c30871ae644b1" gracePeriod=25
Oct 31 01:29:18.520917 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:29:18.520884    4964 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-18bf762c-6cc59897d9-w4n2s"
Oct 31 01:29:18.631807 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:29:18.631780    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4a70412f-a2ba-4f8c-896d-945e58e057db-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4a70412f-a2ba-4f8c-896d-945e58e057db-plugins-config-volume\") pod \"4a70412f-a2ba-4f8c-896d-945e58e057db\" (UID: \"4a70412f-a2ba-4f8c-896d-945e58e057db\") "
Oct 31 01:29:18.632175 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:29:18.632156    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4a70412f-a2ba-4f8c-896d-945e58e057db-kube-api-access-df7jz\" (UniqueName: \"kubernetes.io/projected/4a70412f-a2ba-4f8c-896d-945e58e057db-kube-api-access-df7jz\") pod \"4a70412f-a2ba-4f8c-896d-945e58e057db\" (UID: \"4a70412f-a2ba-4f8c-896d-945e58e057db\") "
Oct 31 01:29:18.632382 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:29:18.632161    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4a70412f-a2ba-4f8c-896d-945e58e057db-plugins-config-volume" pod "4a70412f-a2ba-4f8c-896d-945e58e057db" (UID: "4a70412f-a2ba-4f8c-896d-945e58e057db"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:29:18.633250 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:29:18.633230    4964 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4a70412f-a2ba-4f8c-896d-945e58e057db-plugins-config-volume\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:29:18.641542 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:29:18.638549    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a70412f-a2ba-4f8c-896d-945e58e057db-kube-api-access-df7jz" pod "4a70412f-a2ba-4f8c-896d-945e58e057db" (UID: "4a70412f-a2ba-4f8c-896d-945e58e057db"). InnerVolumeSpecName "kube-api-access-df7jz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:29:18.644056 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:29:18.644035    4964 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-18bf762c-c84588867-fcv87"
Oct 31 01:29:18.734980 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:29:18.734904    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/74f9ca59-18e2-4fb6-b596-08f86168bb39-kube-api-access-p6vqk\" (UniqueName: \"kubernetes.io/projected/74f9ca59-18e2-4fb6-b596-08f86168bb39-kube-api-access-p6vqk\") pod \"74f9ca59-18e2-4fb6-b596-08f86168bb39\" (UID: \"74f9ca59-18e2-4fb6-b596-08f86168bb39\") "
Oct 31 01:29:18.735135 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:29:18.735037    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/74f9ca59-18e2-4fb6-b596-08f86168bb39-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/74f9ca59-18e2-4fb6-b596-08f86168bb39-plugins-config-volume\") pod \"74f9ca59-18e2-4fb6-b596-08f86168bb39\" (UID: \"74f9ca59-18e2-4fb6-b596-08f86168bb39\") "
Oct 31 01:29:18.735506 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:29:18.735476    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/74f9ca59-18e2-4fb6-b596-08f86168bb39-plugins-config-volume" pod "74f9ca59-18e2-4fb6-b596-08f86168bb39" (UID: "74f9ca59-18e2-4fb6-b596-08f86168bb39"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:29:18.735926 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:29:18.735907    4964 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/74f9ca59-18e2-4fb6-b596-08f86168bb39-plugins-config-volume\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:29:18.735992 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:29:18.735935    4964 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-df7jz\" (UniqueName: \"kubernetes.io/projected/4a70412f-a2ba-4f8c-896d-945e58e057db-kube-api-access-df7jz\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:29:18.738083 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:29:18.738052    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/74f9ca59-18e2-4fb6-b596-08f86168bb39-kube-api-access-p6vqk" pod "74f9ca59-18e2-4fb6-b596-08f86168bb39" (UID: "74f9ca59-18e2-4fb6-b596-08f86168bb39"). InnerVolumeSpecName "kube-api-access-p6vqk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:29:18.751297 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:29:18.751275    4964 generic.go:356] "Generic (PLEG): container finished" podID="4a70412f-a2ba-4f8c-896d-945e58e057db" containerID="0d826f90a037e14dcd2b85c2a53a3d1ea5ce5e08644cbdb45fa9fa8577e665da" exitCode=0
Oct 31 01:29:18.751437 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:29:18.751364    4964 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-18bf762c-6cc59897d9-w4n2s"
Oct 31 01:29:18.751437 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:29:18.751364    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-18bf762c-6cc59897d9-w4n2s" event={"ID":"4a70412f-a2ba-4f8c-896d-945e58e057db","Type":"ContainerDied","Data":"0d826f90a037e14dcd2b85c2a53a3d1ea5ce5e08644cbdb45fa9fa8577e665da"}
Oct 31 01:29:18.751437 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:29:18.751421    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-18bf762c-6cc59897d9-w4n2s" event={"ID":"4a70412f-a2ba-4f8c-896d-945e58e057db","Type":"ContainerDied","Data":"7c280845b05c1d0540cdb32387897eeb970a35b2fec3cfc55d17f690a22aac12"}
Oct 31 01:29:18.751604 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:29:18.751450    4964 scope.go:122] "RemoveContainer" containerID="0d826f90a037e14dcd2b85c2a53a3d1ea5ce5e08644cbdb45fa9fa8577e665da"
Oct 31 01:29:18.753037 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:29:18.753017    4964 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-18bf762c-c84588867-fcv87"
Oct 31 01:29:18.753037 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:29:18.753025    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-18bf762c-c84588867-fcv87" event={"ID":"74f9ca59-18e2-4fb6-b596-08f86168bb39","Type":"ContainerDied","Data":"d686c8d947f75ed277c24a533e1078c025f9571cd00b15757a1c30871ae644b1"}
Oct 31 01:29:18.753189 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:29:18.753010    4964 generic.go:356] "Generic (PLEG): container finished" podID="74f9ca59-18e2-4fb6-b596-08f86168bb39" containerID="d686c8d947f75ed277c24a533e1078c025f9571cd00b15757a1c30871ae644b1" exitCode=0
Oct 31 01:29:18.753229 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:29:18.753206    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-18bf762c-c84588867-fcv87" event={"ID":"74f9ca59-18e2-4fb6-b596-08f86168bb39","Type":"ContainerDied","Data":"c8b4b6f4a48f2ad6de69676f4d7a902fb8f5ec1acfa83905249b6b81afd526cb"}
Oct 31 01:29:18.770782 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:29:18.770761    4964 scope.go:122] "RemoveContainer" containerID="0d826f90a037e14dcd2b85c2a53a3d1ea5ce5e08644cbdb45fa9fa8577e665da"
Oct 31 01:29:18.771109 ip-10-0-133-197 kubenswrapper[4964]: E1031 01:29:18.771090    4964 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0d826f90a037e14dcd2b85c2a53a3d1ea5ce5e08644cbdb45fa9fa8577e665da\": container with ID starting with 0d826f90a037e14dcd2b85c2a53a3d1ea5ce5e08644cbdb45fa9fa8577e665da not found: ID does not exist" containerID="0d826f90a037e14dcd2b85c2a53a3d1ea5ce5e08644cbdb45fa9fa8577e665da"
Oct 31 01:29:18.771178 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:29:18.771119    4964 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0d826f90a037e14dcd2b85c2a53a3d1ea5ce5e08644cbdb45fa9fa8577e665da"} err="failed to get container status \"0d826f90a037e14dcd2b85c2a53a3d1ea5ce5e08644cbdb45fa9fa8577e665da\": rpc error: code = NotFound desc = could not find container \"0d826f90a037e14dcd2b85c2a53a3d1ea5ce5e08644cbdb45fa9fa8577e665da\": container with ID starting with 0d826f90a037e14dcd2b85c2a53a3d1ea5ce5e08644cbdb45fa9fa8577e665da not found: ID does not exist"
Oct 31 01:29:18.771178 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:29:18.771146    4964 scope.go:122] "RemoveContainer" containerID="d686c8d947f75ed277c24a533e1078c025f9571cd00b15757a1c30871ae644b1"
Oct 31 01:29:18.785042 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:29:18.785020    4964 scope.go:122] "RemoveContainer" containerID="d686c8d947f75ed277c24a533e1078c025f9571cd00b15757a1c30871ae644b1"
Oct 31 01:29:18.785412 ip-10-0-133-197 kubenswrapper[4964]: E1031 01:29:18.785385    4964 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d686c8d947f75ed277c24a533e1078c025f9571cd00b15757a1c30871ae644b1\": container with ID starting with d686c8d947f75ed277c24a533e1078c025f9571cd00b15757a1c30871ae644b1 not found: ID does not exist" containerID="d686c8d947f75ed277c24a533e1078c025f9571cd00b15757a1c30871ae644b1"
Oct 31 01:29:18.785483 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:29:18.785423    4964 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d686c8d947f75ed277c24a533e1078c025f9571cd00b15757a1c30871ae644b1"} err="failed to get container status \"d686c8d947f75ed277c24a533e1078c025f9571cd00b15757a1c30871ae644b1\": rpc error: code = NotFound desc = could not find container \"d686c8d947f75ed277c24a533e1078c025f9571cd00b15757a1c30871ae644b1\": container with ID starting with d686c8d947f75ed277c24a533e1078c025f9571cd00b15757a1c30871ae644b1 not found: ID does not exist"
Oct 31 01:29:18.790307 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:29:18.790271    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:29:18.837856 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:29:18.837824    4964 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p6vqk\" (UniqueName: \"kubernetes.io/projected/74f9ca59-18e2-4fb6-b596-08f86168bb39-kube-api-access-p6vqk\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:29:18.974775 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:29:18.974748    4964 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-18bf762c-6cc59897d9-w4n2s"]
Oct 31 01:29:18.975099 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:29:18.975073    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:29:18.979044 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:29:18.979023    4964 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-18bf762c-6cc59897d9-w4n2s"]
Oct 31 01:29:19.083939 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:29:19.083857    4964 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-18bf762c-c84588867-fcv87"]
Oct 31 01:29:19.089593 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:29:19.089567    4964 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-18bf762c-c84588867-fcv87"]
Oct 31 01:29:19.150610 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:29:19.150577    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:29:19.778649 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:29:19.778620    4964 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4a70412f-a2ba-4f8c-896d-945e58e057db" path="/var/lib/kubelet/pods/4a70412f-a2ba-4f8c-896d-945e58e057db/volumes"
Oct 31 01:29:19.779097 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:29:19.779082    4964 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="74f9ca59-18e2-4fb6-b596-08f86168bb39" path="/var/lib/kubelet/pods/74f9ca59-18e2-4fb6-b596-08f86168bb39/volumes"
Oct 31 01:29:22.003311 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:29:22.003254    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:29:22.223372 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:29:22.222299    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:29:22.393641 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:29:22.393547    4964 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-e7eea9-6464f8bcd7-wdd67"]
Oct 31 01:29:22.394289 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:29:22.394273    4964 cpu_manager.go:414] "containerMap: removing container" podUID="4a70412f-a2ba-4f8c-896d-945e58e057db" containerName="payload-processing"
Oct 31 01:29:22.394414 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:29:22.394291    4964 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4a70412f-a2ba-4f8c-896d-945e58e057db" containerName="payload-processing"
Oct 31 01:29:22.394414 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:29:22.394327    4964 cpu_manager.go:414] "containerMap: removing container" podUID="74f9ca59-18e2-4fb6-b596-08f86168bb39" containerName="payload-pre-processing"
Oct 31 01:29:22.394414 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:29:22.394334    4964 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="74f9ca59-18e2-4fb6-b596-08f86168bb39" containerName="payload-pre-processing"
Oct 31 01:29:22.394578 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:29:22.394468    4964 memory_manager.go:365] "RemoveStaleState removing state" podUID="4a70412f-a2ba-4f8c-896d-945e58e057db" containerName="payload-processing"
Oct 31 01:29:22.394578 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:29:22.394479    4964 memory_manager.go:365] "RemoveStaleState removing state" podUID="74f9ca59-18e2-4fb6-b596-08f86168bb39" containerName="payload-pre-processing"
Oct 31 01:29:22.399724 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:29:22.399698    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-e7eea9-6464f8bcd7-wdd67"
Oct 31 01:29:22.456043 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:29:22.456014    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-e7eea9-6464f8bcd7-wdd67"]
Oct 31 01:29:22.496988 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:29:22.496959    4964 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-e7eea9\""
Oct 31 01:29:22.497188 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:29:22.497076    4964 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-e7eea9-dockercfg-hg8br\""
Oct 31 01:29:22.509618 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:29:22.509581    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f1770ebf-036d-4645-b3f2-c1f14574438f-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-e7eea9-6464f8bcd7-wdd67\" (UID: \"f1770ebf-036d-4645-b3f2-c1f14574438f\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-e7eea9-6464f8bcd7-wdd67"
Oct 31 01:29:22.510016 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:29:22.509985    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2ds2m\" (UniqueName: \"kubernetes.io/projected/f1770ebf-036d-4645-b3f2-c1f14574438f-kube-api-access-2ds2m\") pod \"payload-pre-processing-e2e-api-b-e7eea9-6464f8bcd7-wdd67\" (UID: \"f1770ebf-036d-4645-b3f2-c1f14574438f\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-e7eea9-6464f8bcd7-wdd67"
Oct 31 01:29:22.552590 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:29:22.552559    4964 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-e7eea9-7699bc58bf-b8snr"]
Oct 31 01:29:22.557939 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:29:22.557912    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-e7eea9-7699bc58bf-b8snr"
Oct 31 01:29:22.607035 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:29:22.604908    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-e7eea9-7699bc58bf-b8snr"]
Oct 31 01:29:22.616418 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:29:22.615849    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f1770ebf-036d-4645-b3f2-c1f14574438f-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-e7eea9-6464f8bcd7-wdd67\" (UID: \"f1770ebf-036d-4645-b3f2-c1f14574438f\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-e7eea9-6464f8bcd7-wdd67"
Oct 31 01:29:22.616418 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:29:22.615978    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2ds2m\" (UniqueName: \"kubernetes.io/projected/f1770ebf-036d-4645-b3f2-c1f14574438f-kube-api-access-2ds2m\") pod \"payload-pre-processing-e2e-api-b-e7eea9-6464f8bcd7-wdd67\" (UID: \"f1770ebf-036d-4645-b3f2-c1f14574438f\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-e7eea9-6464f8bcd7-wdd67"
Oct 31 01:29:22.625361 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:29:22.624720    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f1770ebf-036d-4645-b3f2-c1f14574438f-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-e7eea9-6464f8bcd7-wdd67\" (UID: \"f1770ebf-036d-4645-b3f2-c1f14574438f\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-e7eea9-6464f8bcd7-wdd67"
Oct 31 01:29:22.654547 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:29:22.654473    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-42c96d-65764fbd8c-84plq"
Oct 31 01:29:22.673714 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:29:22.673683    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2ds2m\" (UniqueName: \"kubernetes.io/projected/f1770ebf-036d-4645-b3f2-c1f14574438f-kube-api-access-2ds2m\") pod \"payload-pre-processing-e2e-api-b-e7eea9-6464f8bcd7-wdd67\" (UID: \"f1770ebf-036d-4645-b3f2-c1f14574438f\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-e7eea9-6464f8bcd7-wdd67"
Oct 31 01:29:22.718365 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:29:22.718335    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-55gqg\" (UniqueName: \"kubernetes.io/projected/fbf50441-0853-4ecf-99ea-38a47e9d5677-kube-api-access-55gqg\") pod \"payload-processing-e2e-api-b-e7eea9-7699bc58bf-b8snr\" (UID: \"fbf50441-0853-4ecf-99ea-38a47e9d5677\") " pod="openshift-ingress/payload-processing-e2e-api-b-e7eea9-7699bc58bf-b8snr"
Oct 31 01:29:22.718614 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:29:22.718525    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fbf50441-0853-4ecf-99ea-38a47e9d5677-plugins-config-volume\") pod \"payload-processing-e2e-api-b-e7eea9-7699bc58bf-b8snr\" (UID: \"fbf50441-0853-4ecf-99ea-38a47e9d5677\") " pod="openshift-ingress/payload-processing-e2e-api-b-e7eea9-7699bc58bf-b8snr"
Oct 31 01:29:22.724553 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:29:22.724520    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-e7eea9-6464f8bcd7-wdd67"
Oct 31 01:29:22.820060 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:29:22.820001    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fbf50441-0853-4ecf-99ea-38a47e9d5677-plugins-config-volume\") pod \"payload-processing-e2e-api-b-e7eea9-7699bc58bf-b8snr\" (UID: \"fbf50441-0853-4ecf-99ea-38a47e9d5677\") " pod="openshift-ingress/payload-processing-e2e-api-b-e7eea9-7699bc58bf-b8snr"
Oct 31 01:29:22.820254 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:29:22.820241    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-55gqg\" (UniqueName: \"kubernetes.io/projected/fbf50441-0853-4ecf-99ea-38a47e9d5677-kube-api-access-55gqg\") pod \"payload-processing-e2e-api-b-e7eea9-7699bc58bf-b8snr\" (UID: \"fbf50441-0853-4ecf-99ea-38a47e9d5677\") " pod="openshift-ingress/payload-processing-e2e-api-b-e7eea9-7699bc58bf-b8snr"
Oct 31 01:29:22.821727 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:29:22.821667    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fbf50441-0853-4ecf-99ea-38a47e9d5677-plugins-config-volume\") pod \"payload-processing-e2e-api-b-e7eea9-7699bc58bf-b8snr\" (UID: \"fbf50441-0853-4ecf-99ea-38a47e9d5677\") " pod="openshift-ingress/payload-processing-e2e-api-b-e7eea9-7699bc58bf-b8snr"
Oct 31 01:29:22.899568 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:29:22.896738    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-55gqg\" (UniqueName: \"kubernetes.io/projected/fbf50441-0853-4ecf-99ea-38a47e9d5677-kube-api-access-55gqg\") pod \"payload-processing-e2e-api-b-e7eea9-7699bc58bf-b8snr\" (UID: \"fbf50441-0853-4ecf-99ea-38a47e9d5677\") " pod="openshift-ingress/payload-processing-e2e-api-b-e7eea9-7699bc58bf-b8snr"
Oct 31 01:29:23.024339 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:29:23.011487    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:29:23.130228 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:29:23.130201    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-e7eea9-6464f8bcd7-wdd67"]
Oct 31 01:29:23.131278 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:29:23.131252    4964 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1770ebf_036d_4645_b3f2_c1f14574438f.slice/crio-d4d10aa2ba02745117656a513ec3558c8a9701a63c73df6a80fe036193e84baa WatchSource:0}: Error finding container d4d10aa2ba02745117656a513ec3558c8a9701a63c73df6a80fe036193e84baa: Status 404 returned error can't find the container with id d4d10aa2ba02745117656a513ec3558c8a9701a63c73df6a80fe036193e84baa
Oct 31 01:29:23.180127 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:29:23.180064    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-e7eea9-7699bc58bf-b8snr"
Oct 31 01:29:23.589176 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:29:23.589125    4964 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfbf50441_0853_4ecf_99ea_38a47e9d5677.slice/crio-7f872630e42b957495e3769566b657beb4e3c54106318da91b4cfaf33efd5bf5 WatchSource:0}: Error finding container 7f872630e42b957495e3769566b657beb4e3c54106318da91b4cfaf33efd5bf5: Status 404 returned error can't find the container with id 7f872630e42b957495e3769566b657beb4e3c54106318da91b4cfaf33efd5bf5
Oct 31 01:29:23.589731 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:29:23.589709    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-e7eea9-7699bc58bf-b8snr"]
Oct 31 01:29:23.802406 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:29:23.802369    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-e7eea9-7699bc58bf-b8snr" event={"ID":"fbf50441-0853-4ecf-99ea-38a47e9d5677","Type":"ContainerStarted","Data":"790cd26a5733b2b9f30b0b221059c2786bd948b7717a861fc45ec55b164ffee7"}
Oct 31 01:29:23.802600 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:29:23.802433    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-e7eea9-7699bc58bf-b8snr" event={"ID":"fbf50441-0853-4ecf-99ea-38a47e9d5677","Type":"ContainerStarted","Data":"7f872630e42b957495e3769566b657beb4e3c54106318da91b4cfaf33efd5bf5"}
Oct 31 01:29:23.802697 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:29:23.802594    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-e7eea9-7699bc58bf-b8snr"
Oct 31 01:29:23.804109 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:29:23.804082    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-e7eea9-6464f8bcd7-wdd67" event={"ID":"f1770ebf-036d-4645-b3f2-c1f14574438f","Type":"ContainerStarted","Data":"75b89ffa93193cdda1bfd503acfef5a3c5dba50f75e0a7b9a6c9b36f6a0d05ca"}
Oct 31 01:29:23.804237 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:29:23.804135    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-e7eea9-6464f8bcd7-wdd67"
Oct 31 01:29:23.804237 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:29:23.804157    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-e7eea9-6464f8bcd7-wdd67" event={"ID":"f1770ebf-036d-4645-b3f2-c1f14574438f","Type":"ContainerStarted","Data":"d4d10aa2ba02745117656a513ec3558c8a9701a63c73df6a80fe036193e84baa"}
Oct 31 01:29:23.916518 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:29:23.916405    4964 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-e7eea9-7699bc58bf-b8snr" podStartSLOduration=1.9163858839999999 podStartE2EDuration="1.916385884s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:29:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:29:23.9141664 +0000 UTC m=+1532.390732782" watchObservedRunningTime="2026-10-31 01:29:23.916385884 +0000 UTC m=+1532.392952265"
Oct 31 01:29:24.030730 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:29:24.030681    4964 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-e7eea9-6464f8bcd7-wdd67" podStartSLOduration=2.030665021 podStartE2EDuration="2.030665021s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:29:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:29:24.025928945 +0000 UTC m=+1532.502495326" watchObservedRunningTime="2026-10-31 01:29:24.030665021 +0000 UTC m=+1532.507231412"
Oct 31 01:29:24.520693 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:29:24.520655    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:29:25.268642 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:29:25.268610    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:29:25.769028 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:29:25.768990    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:29:25.994909 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:29:25.994863    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:29:26.308341 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:29:26.306864    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:29:26.492112 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:29:26.492071    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:29:28.358781 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:29:28.358740    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:29:29.141025 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:29:29.140991    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:29:29.308167 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:29:29.308130    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:29:29.518702 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:29:29.518659    4964 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-2c70e7-5d586ccdb7-l82mc"]
Oct 31 01:29:29.531977 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:29:29.531949    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-2c70e7-5d586ccdb7-l82mc"
Oct 31 01:29:29.586613 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:29:29.586577    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-2c70e7-5d586ccdb7-l82mc"]
Oct 31 01:29:29.588872 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:29:29.588827    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2708a82e-32b4-443c-8cfe-97815ae5f85b-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-2c70e7-5d586ccdb7-l82mc\" (UID: \"2708a82e-32b4-443c-8cfe-97815ae5f85b\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-2c70e7-5d586ccdb7-l82mc"
Oct 31 01:29:29.589070 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:29:29.589045    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fncqm\" (UniqueName: \"kubernetes.io/projected/2708a82e-32b4-443c-8cfe-97815ae5f85b-kube-api-access-fncqm\") pod \"payload-pre-processing-e2e-api-a-2c70e7-5d586ccdb7-l82mc\" (UID: \"2708a82e-32b4-443c-8cfe-97815ae5f85b\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-2c70e7-5d586ccdb7-l82mc"
Oct 31 01:29:29.697365 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:29:29.695764    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fncqm\" (UniqueName: \"kubernetes.io/projected/2708a82e-32b4-443c-8cfe-97815ae5f85b-kube-api-access-fncqm\") pod \"payload-pre-processing-e2e-api-a-2c70e7-5d586ccdb7-l82mc\" (UID: \"2708a82e-32b4-443c-8cfe-97815ae5f85b\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-2c70e7-5d586ccdb7-l82mc"
Oct 31 01:29:29.697365 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:29:29.695914    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2708a82e-32b4-443c-8cfe-97815ae5f85b-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-2c70e7-5d586ccdb7-l82mc\" (UID: \"2708a82e-32b4-443c-8cfe-97815ae5f85b\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-2c70e7-5d586ccdb7-l82mc"
Oct 31 01:29:29.725982 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:29:29.725949    4964 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-2c70e7\""
Oct 31 01:29:29.726356 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:29:29.726337    4964 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-2c70e7-dockercfg-c5npv\""
Oct 31 01:29:29.764401 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:29:29.753692    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2708a82e-32b4-443c-8cfe-97815ae5f85b-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-2c70e7-5d586ccdb7-l82mc\" (UID: \"2708a82e-32b4-443c-8cfe-97815ae5f85b\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-2c70e7-5d586ccdb7-l82mc"
Oct 31 01:29:29.769386 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:29:29.769283    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fncqm\" (UniqueName: \"kubernetes.io/projected/2708a82e-32b4-443c-8cfe-97815ae5f85b-kube-api-access-fncqm\") pod \"payload-pre-processing-e2e-api-a-2c70e7-5d586ccdb7-l82mc\" (UID: \"2708a82e-32b4-443c-8cfe-97815ae5f85b\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-2c70e7-5d586ccdb7-l82mc"
Oct 31 01:29:29.852798 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:29:29.852768    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-2c70e7-5d586ccdb7-l82mc"
Oct 31 01:29:30.267004 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:29:30.266972    4964 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2708a82e_32b4_443c_8cfe_97815ae5f85b.slice/crio-c203321453896b258b0e2cf120ef4ddb92b8e28ee63b31ebf818a2a23e1b5184 WatchSource:0}: Error finding container c203321453896b258b0e2cf120ef4ddb92b8e28ee63b31ebf818a2a23e1b5184: Status 404 returned error can't find the container with id c203321453896b258b0e2cf120ef4ddb92b8e28ee63b31ebf818a2a23e1b5184
Oct 31 01:29:30.269225 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:29:30.269193    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-2c70e7-5d586ccdb7-l82mc"]
Oct 31 01:29:30.386622 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:29:30.386589    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:29:30.620785 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:29:30.620687    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:29:30.862476 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:29:30.862442    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-2c70e7-5d586ccdb7-l82mc" event={"ID":"2708a82e-32b4-443c-8cfe-97815ae5f85b","Type":"ContainerStarted","Data":"c4b8f4383237145d9dc69e41d9af5c1cedeea41c16fc57d57dfec38c5cafa94b"}
Oct 31 01:29:30.862651 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:29:30.862489    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-2c70e7-5d586ccdb7-l82mc" event={"ID":"2708a82e-32b4-443c-8cfe-97815ae5f85b","Type":"ContainerStarted","Data":"c203321453896b258b0e2cf120ef4ddb92b8e28ee63b31ebf818a2a23e1b5184"}
Oct 31 01:29:30.862651 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:29:30.862532    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-2c70e7-5d586ccdb7-l82mc"
Oct 31 01:29:30.979039 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:29:30.978989    4964 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-2c70e7-5d586ccdb7-l82mc" podStartSLOduration=1.978976989 podStartE2EDuration="1.978976989s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:29:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:29:30.975168034 +0000 UTC m=+1539.451734415" watchObservedRunningTime="2026-10-31 01:29:30.978976989 +0000 UTC m=+1539.455543368"
Oct 31 01:29:33.641106 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:29:33.641067    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:29:34.816004 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:29:34.815958    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-e7eea9-7699bc58bf-b8snr"
Oct 31 01:29:34.816515 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:29:34.816125    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-e7eea9-6464f8bcd7-wdd67"
Oct 31 01:29:36.247564 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:29:36.247532    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:29:36.443785 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:29:36.443751    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:29:36.780884 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:29:36.780848    4964 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-14ca09-59f575c7d9-ttflk"]
Oct 31 01:29:36.798604 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:29:36.798571    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-14ca09-59f575c7d9-ttflk"
Oct 31 01:29:36.873119 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:29:36.873078    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-14ca09-59f575c7d9-ttflk"]
Oct 31 01:29:36.889520 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:29:36.889488    4964 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-14ca09-dockercfg-lhbcx\""
Oct 31 01:29:36.890029 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:29:36.890009    4964 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-14ca09\""
Oct 31 01:29:36.908506 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:29:36.907698    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d8871960-8a84-44b1-b24a-a6447283f329-plugins-config-volume\") pod \"payload-processing-e2e-api-b-14ca09-59f575c7d9-ttflk\" (UID: \"d8871960-8a84-44b1-b24a-a6447283f329\") " pod="openshift-ingress/payload-processing-e2e-api-b-14ca09-59f575c7d9-ttflk"
Oct 31 01:29:36.913852 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:29:36.913822    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ttgbh\" (UniqueName: \"kubernetes.io/projected/d8871960-8a84-44b1-b24a-a6447283f329-kube-api-access-ttgbh\") pod \"payload-processing-e2e-api-b-14ca09-59f575c7d9-ttflk\" (UID: \"d8871960-8a84-44b1-b24a-a6447283f329\") " pod="openshift-ingress/payload-processing-e2e-api-b-14ca09-59f575c7d9-ttflk"
Oct 31 01:29:37.016604 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:29:37.016569    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d8871960-8a84-44b1-b24a-a6447283f329-plugins-config-volume\") pod \"payload-processing-e2e-api-b-14ca09-59f575c7d9-ttflk\" (UID: \"d8871960-8a84-44b1-b24a-a6447283f329\") " pod="openshift-ingress/payload-processing-e2e-api-b-14ca09-59f575c7d9-ttflk"
Oct 31 01:29:37.018214 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:29:37.017853    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ttgbh\" (UniqueName: \"kubernetes.io/projected/d8871960-8a84-44b1-b24a-a6447283f329-kube-api-access-ttgbh\") pod \"payload-processing-e2e-api-b-14ca09-59f575c7d9-ttflk\" (UID: \"d8871960-8a84-44b1-b24a-a6447283f329\") " pod="openshift-ingress/payload-processing-e2e-api-b-14ca09-59f575c7d9-ttflk"
Oct 31 01:29:37.018674 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:29:37.017774    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d8871960-8a84-44b1-b24a-a6447283f329-plugins-config-volume\") pod \"payload-processing-e2e-api-b-14ca09-59f575c7d9-ttflk\" (UID: \"d8871960-8a84-44b1-b24a-a6447283f329\") " pod="openshift-ingress/payload-processing-e2e-api-b-14ca09-59f575c7d9-ttflk"
Oct 31 01:29:37.074601 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:29:37.074535    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ttgbh\" (UniqueName: \"kubernetes.io/projected/d8871960-8a84-44b1-b24a-a6447283f329-kube-api-access-ttgbh\") pod \"payload-processing-e2e-api-b-14ca09-59f575c7d9-ttflk\" (UID: \"d8871960-8a84-44b1-b24a-a6447283f329\") " pod="openshift-ingress/payload-processing-e2e-api-b-14ca09-59f575c7d9-ttflk"
Oct 31 01:29:37.145764 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:29:37.145725    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-14ca09-59f575c7d9-ttflk"
Oct 31 01:29:37.549624 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:29:37.549597    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-14ca09-59f575c7d9-ttflk"]
Oct 31 01:29:37.550807 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:29:37.550780    4964 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd8871960_8a84_44b1_b24a_a6447283f329.slice/crio-5922efb67f2965c56d392e0e328ec2c787c7dceb2c5bb21e83ff65c494acca13 WatchSource:0}: Error finding container 5922efb67f2965c56d392e0e328ec2c787c7dceb2c5bb21e83ff65c494acca13: Status 404 returned error can't find the container with id 5922efb67f2965c56d392e0e328ec2c787c7dceb2c5bb21e83ff65c494acca13
Oct 31 01:29:37.958758 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:29:37.958662    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-14ca09-59f575c7d9-ttflk" event={"ID":"d8871960-8a84-44b1-b24a-a6447283f329","Type":"ContainerStarted","Data":"5bf89c629df41376d9ed36fef057240cf348680d0ed8a93f6d86556fd2b22006"}
Oct 31 01:29:37.958758 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:29:37.958727    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-14ca09-59f575c7d9-ttflk" event={"ID":"d8871960-8a84-44b1-b24a-a6447283f329","Type":"ContainerStarted","Data":"5922efb67f2965c56d392e0e328ec2c787c7dceb2c5bb21e83ff65c494acca13"}
Oct 31 01:29:37.958758 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:29:37.958755    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-14ca09-59f575c7d9-ttflk"
Oct 31 01:29:38.073798 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:29:38.073749    4964 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-14ca09-59f575c7d9-ttflk" podStartSLOduration=2.073731589 podStartE2EDuration="2.073731589s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:29:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:29:38.07057621 +0000 UTC m=+1546.547142591" watchObservedRunningTime="2026-10-31 01:29:38.073731589 +0000 UTC m=+1546.550297972"
Oct 31 01:29:38.834156 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:29:38.834123    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:29:41.240143 ip-10-0-133-197 kubenswrapper[4964]: E1031 01:29:41.240106    4964 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q56x5" containerName="extractor"
Oct 31 01:29:41.876527 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:29:41.876496    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-2c70e7-5d586ccdb7-l82mc"
Oct 31 01:29:44.088932 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:29:44.088864    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:29:48.762661 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:29:48.762632    4964 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-42c96d-6d9dd859d6-z258p"]
Oct 31 01:29:48.768386 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:29:48.768368    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-42c96d-6d9dd859d6-z258p"
Oct 31 01:29:48.805498 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:29:48.805467    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0d4e0035-30d3-472e-96e5-04df99e26e48-maas-api-tls\") pod \"maas-api-e2e-api-a-42c96d-6d9dd859d6-z258p\" (UID: \"0d4e0035-30d3-472e-96e5-04df99e26e48\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-42c96d-6d9dd859d6-z258p"
Oct 31 01:29:48.805628 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:29:48.805535    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0d4e0035-30d3-472e-96e5-04df99e26e48-metrics-certs\") pod \"maas-api-e2e-api-a-42c96d-6d9dd859d6-z258p\" (UID: \"0d4e0035-30d3-472e-96e5-04df99e26e48\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-42c96d-6d9dd859d6-z258p"
Oct 31 01:29:48.805701 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:29:48.805632    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0d4e0035-30d3-472e-96e5-04df99e26e48-openshift-service-ca\") pod \"maas-api-e2e-api-a-42c96d-6d9dd859d6-z258p\" (UID: \"0d4e0035-30d3-472e-96e5-04df99e26e48\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-42c96d-6d9dd859d6-z258p"
Oct 31 01:29:48.805701 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:29:48.805664    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-54vq4\" (UniqueName: \"kubernetes.io/projected/0d4e0035-30d3-472e-96e5-04df99e26e48-kube-api-access-54vq4\") pod \"maas-api-e2e-api-a-42c96d-6d9dd859d6-z258p\" (UID: \"0d4e0035-30d3-472e-96e5-04df99e26e48\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-42c96d-6d9dd859d6-z258p"
Oct 31 01:29:48.824556 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:29:48.824530    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-42c96d-6d9dd859d6-z258p"]
Oct 31 01:29:48.868575 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:29:48.868545    4964 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-42c96d-serving-cert\""
Oct 31 01:29:48.907463 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:29:48.907436    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0d4e0035-30d3-472e-96e5-04df99e26e48-maas-api-tls\") pod \"maas-api-e2e-api-a-42c96d-6d9dd859d6-z258p\" (UID: \"0d4e0035-30d3-472e-96e5-04df99e26e48\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-42c96d-6d9dd859d6-z258p"
Oct 31 01:29:48.907594 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:29:48.907515    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0d4e0035-30d3-472e-96e5-04df99e26e48-metrics-certs\") pod \"maas-api-e2e-api-a-42c96d-6d9dd859d6-z258p\" (UID: \"0d4e0035-30d3-472e-96e5-04df99e26e48\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-42c96d-6d9dd859d6-z258p"
Oct 31 01:29:48.907594 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:29:48.907549    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0d4e0035-30d3-472e-96e5-04df99e26e48-openshift-service-ca\") pod \"maas-api-e2e-api-a-42c96d-6d9dd859d6-z258p\" (UID: \"0d4e0035-30d3-472e-96e5-04df99e26e48\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-42c96d-6d9dd859d6-z258p"
Oct 31 01:29:48.907594 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:29:48.907569    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-54vq4\" (UniqueName: \"kubernetes.io/projected/0d4e0035-30d3-472e-96e5-04df99e26e48-kube-api-access-54vq4\") pod \"maas-api-e2e-api-a-42c96d-6d9dd859d6-z258p\" (UID: \"0d4e0035-30d3-472e-96e5-04df99e26e48\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-42c96d-6d9dd859d6-z258p"
Oct 31 01:29:48.907594 ip-10-0-133-197 kubenswrapper[4964]: E1031 01:29:48.907573    4964 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-42c96d-serving-cert: secret "maas-api-e2e-api-a-42c96d-serving-cert" not found
Oct 31 01:29:48.907800 ip-10-0-133-197 kubenswrapper[4964]: E1031 01:29:48.907644    4964 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0d4e0035-30d3-472e-96e5-04df99e26e48-maas-api-tls podName:0d4e0035-30d3-472e-96e5-04df99e26e48 nodeName:}" failed. No retries permitted until 2026-10-31 01:29:49.407625215 +0000 UTC m=+1557.884191573 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/0d4e0035-30d3-472e-96e5-04df99e26e48-maas-api-tls") pod "maas-api-e2e-api-a-42c96d-6d9dd859d6-z258p" (UID: "0d4e0035-30d3-472e-96e5-04df99e26e48") : secret "maas-api-e2e-api-a-42c96d-serving-cert" not found
Oct 31 01:29:48.908247 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:29:48.908220    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0d4e0035-30d3-472e-96e5-04df99e26e48-openshift-service-ca\") pod \"maas-api-e2e-api-a-42c96d-6d9dd859d6-z258p\" (UID: \"0d4e0035-30d3-472e-96e5-04df99e26e48\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-42c96d-6d9dd859d6-z258p"
Oct 31 01:29:48.910154 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:29:48.910137    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0d4e0035-30d3-472e-96e5-04df99e26e48-metrics-certs\") pod \"maas-api-e2e-api-a-42c96d-6d9dd859d6-z258p\" (UID: \"0d4e0035-30d3-472e-96e5-04df99e26e48\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-42c96d-6d9dd859d6-z258p"
Oct 31 01:29:48.969110 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:29:48.969083    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-14ca09-59f575c7d9-ttflk"
Oct 31 01:29:48.982089 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:29:48.982053    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-54vq4\" (UniqueName: \"kubernetes.io/projected/0d4e0035-30d3-472e-96e5-04df99e26e48-kube-api-access-54vq4\") pod \"maas-api-e2e-api-a-42c96d-6d9dd859d6-z258p\" (UID: \"0d4e0035-30d3-472e-96e5-04df99e26e48\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-42c96d-6d9dd859d6-z258p"
Oct 31 01:29:49.049398 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:29:49.049299    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:29:49.226664 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:29:49.226622    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:29:49.422042 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:29:49.421954    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0d4e0035-30d3-472e-96e5-04df99e26e48-maas-api-tls\") pod \"maas-api-e2e-api-a-42c96d-6d9dd859d6-z258p\" (UID: \"0d4e0035-30d3-472e-96e5-04df99e26e48\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-42c96d-6d9dd859d6-z258p"
Oct 31 01:29:49.424572 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:29:49.424547    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0d4e0035-30d3-472e-96e5-04df99e26e48-maas-api-tls\") pod \"maas-api-e2e-api-a-42c96d-6d9dd859d6-z258p\" (UID: \"0d4e0035-30d3-472e-96e5-04df99e26e48\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-42c96d-6d9dd859d6-z258p"
Oct 31 01:29:49.682201 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:29:49.682120    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-42c96d-6d9dd859d6-z258p"
Oct 31 01:29:50.072583 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:29:50.072556    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-42c96d-6d9dd859d6-z258p"]
Oct 31 01:29:50.073240 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:29:50.073211    4964 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d4e0035_30d3_472e_96e5_04df99e26e48.slice/crio-e098cea11a8b504e646a0f9f6e0f8e9c691e1374e3af9896aa2f1c7b91745c2b WatchSource:0}: Error finding container e098cea11a8b504e646a0f9f6e0f8e9c691e1374e3af9896aa2f1c7b91745c2b: Status 404 returned error can't find the container with id e098cea11a8b504e646a0f9f6e0f8e9c691e1374e3af9896aa2f1c7b91745c2b
Oct 31 01:29:51.081843 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:29:51.081804    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-42c96d-6d9dd859d6-z258p" event={"ID":"0d4e0035-30d3-472e-96e5-04df99e26e48","Type":"ContainerStarted","Data":"e098cea11a8b504e646a0f9f6e0f8e9c691e1374e3af9896aa2f1c7b91745c2b"}
Oct 31 01:29:52.095134 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:29:52.095091    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-42c96d-6d9dd859d6-z258p" event={"ID":"0d4e0035-30d3-472e-96e5-04df99e26e48","Type":"ContainerStarted","Data":"d078e600265fc8fafd5da89422fad7f6151a3b69c8f3786419f67837836ef3a6"}
Oct 31 01:29:52.095893 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:29:52.095169    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-42c96d-6d9dd859d6-z258p"
Oct 31 01:29:52.211331 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:29:52.211270    4964 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-42c96d-6d9dd859d6-z258p" podStartSLOduration=3.286464355 podStartE2EDuration="4.211256866s" totalImagesPullingTime="924.792511ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:29:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:29:52.209360479 +0000 UTC m=+1560.685926851" watchObservedRunningTime="2026-10-31 01:29:52.211256866 +0000 UTC m=+1560.687823245"
Oct 31 01:29:53.352928 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:29:53.352896    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:29:53.526772 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:29:53.526741    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:29:54.153579 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:29:54.153539    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:29:57.987542 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:29:57.987504    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:29:58.110018 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:29:58.109989    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-42c96d-6d9dd859d6-z258p"
Oct 31 01:29:58.976737 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:29:58.976698    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:30:00.186607 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:30:00.186413    4964 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-7cec02-29890170-vn2k2"]
Oct 31 01:30:00.205645 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:30:00.205612    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-7cec02-29890170-vn2k2"
Oct 31 01:30:00.277546 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:30:00.277509    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-7cec02-29890170-vn2k2"]
Oct 31 01:30:00.293491 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:30:00.293455    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fssjg\" (UniqueName: \"kubernetes.io/projected/fb021400-cb4b-494d-8967-5d0a7b35a2a7-kube-api-access-fssjg\") pod \"maas-api-key-cleanup-e2e-api-a-7cec02-29890170-vn2k2\" (UID: \"fb021400-cb4b-494d-8967-5d0a7b35a2a7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-7cec02-29890170-vn2k2"
Oct 31 01:30:00.293671 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:30:00.293602    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fb021400-cb4b-494d-8967-5d0a7b35a2a7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-a-7cec02-29890170-vn2k2\" (UID: \"fb021400-cb4b-494d-8967-5d0a7b35a2a7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-7cec02-29890170-vn2k2"
Oct 31 01:30:00.322476 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:30:00.322449    4964 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zzj8k\""
Oct 31 01:30:00.394815 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:30:00.394785    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fssjg\" (UniqueName: \"kubernetes.io/projected/fb021400-cb4b-494d-8967-5d0a7b35a2a7-kube-api-access-fssjg\") pod \"maas-api-key-cleanup-e2e-api-a-7cec02-29890170-vn2k2\" (UID: \"fb021400-cb4b-494d-8967-5d0a7b35a2a7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-7cec02-29890170-vn2k2"
Oct 31 01:30:00.394975 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:30:00.394842    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fb021400-cb4b-494d-8967-5d0a7b35a2a7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-a-7cec02-29890170-vn2k2\" (UID: \"fb021400-cb4b-494d-8967-5d0a7b35a2a7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-7cec02-29890170-vn2k2"
Oct 31 01:30:00.395462 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:30:00.395442    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fb021400-cb4b-494d-8967-5d0a7b35a2a7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-a-7cec02-29890170-vn2k2\" (UID: \"fb021400-cb4b-494d-8967-5d0a7b35a2a7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-7cec02-29890170-vn2k2"
Oct 31 01:30:00.451198 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:30:00.451134    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fssjg\" (UniqueName: \"kubernetes.io/projected/fb021400-cb4b-494d-8967-5d0a7b35a2a7-kube-api-access-fssjg\") pod \"maas-api-key-cleanup-e2e-api-a-7cec02-29890170-vn2k2\" (UID: \"fb021400-cb4b-494d-8967-5d0a7b35a2a7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-7cec02-29890170-vn2k2"
Oct 31 01:30:00.525524 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:30:00.525496    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-7cec02-29890170-vn2k2"
Oct 31 01:30:00.907371 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:30:00.907062    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-7cec02-29890170-vn2k2"]
Oct 31 01:30:00.915813 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:30:00.915785    4964 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb021400_cb4b_494d_8967_5d0a7b35a2a7.slice/crio-3eeb086c10b4b802754774401667dfdc5ab8bd0f772d97a4aae11e3beafe8678 WatchSource:0}: Error finding container 3eeb086c10b4b802754774401667dfdc5ab8bd0f772d97a4aae11e3beafe8678: Status 404 returned error can't find the container with id 3eeb086c10b4b802754774401667dfdc5ab8bd0f772d97a4aae11e3beafe8678
Oct 31 01:30:01.166042 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:30:01.165966    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-7cec02-29890170-vn2k2" event={"ID":"fb021400-cb4b-494d-8967-5d0a7b35a2a7","Type":"ContainerStarted","Data":"3eeb086c10b4b802754774401667dfdc5ab8bd0f772d97a4aae11e3beafe8678"}
Oct 31 01:30:03.421943 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:30:03.421910    4964 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-e7eea9-74d6c5586f-hp7tc"]
Oct 31 01:30:03.429221 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:30:03.429190    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e7eea9-74d6c5586f-hp7tc"
Oct 31 01:30:03.455064 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:30:03.455039    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a4153b03-5882-419a-bb5c-f1c9592b23fa-maas-api-tls\") pod \"maas-api-e2e-api-b-e7eea9-74d6c5586f-hp7tc\" (UID: \"a4153b03-5882-419a-bb5c-f1c9592b23fa\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e7eea9-74d6c5586f-hp7tc"
Oct 31 01:30:03.455193 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:30:03.455109    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jp2r8\" (UniqueName: \"kubernetes.io/projected/a4153b03-5882-419a-bb5c-f1c9592b23fa-kube-api-access-jp2r8\") pod \"maas-api-e2e-api-b-e7eea9-74d6c5586f-hp7tc\" (UID: \"a4153b03-5882-419a-bb5c-f1c9592b23fa\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e7eea9-74d6c5586f-hp7tc"
Oct 31 01:30:03.455255 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:30:03.455240    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a4153b03-5882-419a-bb5c-f1c9592b23fa-openshift-service-ca\") pod \"maas-api-e2e-api-b-e7eea9-74d6c5586f-hp7tc\" (UID: \"a4153b03-5882-419a-bb5c-f1c9592b23fa\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e7eea9-74d6c5586f-hp7tc"
Oct 31 01:30:03.455348 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:30:03.455299    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a4153b03-5882-419a-bb5c-f1c9592b23fa-metrics-certs\") pod \"maas-api-e2e-api-b-e7eea9-74d6c5586f-hp7tc\" (UID: \"a4153b03-5882-419a-bb5c-f1c9592b23fa\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e7eea9-74d6c5586f-hp7tc"
Oct 31 01:30:03.481178 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:30:03.481152    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-e7eea9-74d6c5586f-hp7tc"]
Oct 31 01:30:03.525211 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:30:03.525176    4964 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-e7eea9-serving-cert\""
Oct 31 01:30:03.556521 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:30:03.556479    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a4153b03-5882-419a-bb5c-f1c9592b23fa-metrics-certs\") pod \"maas-api-e2e-api-b-e7eea9-74d6c5586f-hp7tc\" (UID: \"a4153b03-5882-419a-bb5c-f1c9592b23fa\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e7eea9-74d6c5586f-hp7tc"
Oct 31 01:30:03.556692 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:30:03.556637    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a4153b03-5882-419a-bb5c-f1c9592b23fa-maas-api-tls\") pod \"maas-api-e2e-api-b-e7eea9-74d6c5586f-hp7tc\" (UID: \"a4153b03-5882-419a-bb5c-f1c9592b23fa\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e7eea9-74d6c5586f-hp7tc"
Oct 31 01:30:03.556765 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:30:03.556692    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jp2r8\" (UniqueName: \"kubernetes.io/projected/a4153b03-5882-419a-bb5c-f1c9592b23fa-kube-api-access-jp2r8\") pod \"maas-api-e2e-api-b-e7eea9-74d6c5586f-hp7tc\" (UID: \"a4153b03-5882-419a-bb5c-f1c9592b23fa\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e7eea9-74d6c5586f-hp7tc"
Oct 31 01:30:03.556833 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:30:03.556788    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a4153b03-5882-419a-bb5c-f1c9592b23fa-openshift-service-ca\") pod \"maas-api-e2e-api-b-e7eea9-74d6c5586f-hp7tc\" (UID: \"a4153b03-5882-419a-bb5c-f1c9592b23fa\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e7eea9-74d6c5586f-hp7tc"
Oct 31 01:30:03.557455 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:30:03.557431    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a4153b03-5882-419a-bb5c-f1c9592b23fa-openshift-service-ca\") pod \"maas-api-e2e-api-b-e7eea9-74d6c5586f-hp7tc\" (UID: \"a4153b03-5882-419a-bb5c-f1c9592b23fa\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e7eea9-74d6c5586f-hp7tc"
Oct 31 01:30:03.560055 ip-10-0-133-197 kubenswrapper[4964]: E1031 01:30:03.560036    4964 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-e7eea9-serving-cert: secret "maas-api-e2e-api-b-e7eea9-serving-cert" not found
Oct 31 01:30:03.560197 ip-10-0-133-197 kubenswrapper[4964]: E1031 01:30:03.560101    4964 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a4153b03-5882-419a-bb5c-f1c9592b23fa-maas-api-tls podName:a4153b03-5882-419a-bb5c-f1c9592b23fa nodeName:}" failed. No retries permitted until 2026-10-31 01:30:04.060082234 +0000 UTC m=+1572.536648607 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/a4153b03-5882-419a-bb5c-f1c9592b23fa-maas-api-tls") pod "maas-api-e2e-api-b-e7eea9-74d6c5586f-hp7tc" (UID: "a4153b03-5882-419a-bb5c-f1c9592b23fa") : secret "maas-api-e2e-api-b-e7eea9-serving-cert" not found
Oct 31 01:30:03.560956 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:30:03.560932    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a4153b03-5882-419a-bb5c-f1c9592b23fa-metrics-certs\") pod \"maas-api-e2e-api-b-e7eea9-74d6c5586f-hp7tc\" (UID: \"a4153b03-5882-419a-bb5c-f1c9592b23fa\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e7eea9-74d6c5586f-hp7tc"
Oct 31 01:30:03.617503 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:30:03.617473    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jp2r8\" (UniqueName: \"kubernetes.io/projected/a4153b03-5882-419a-bb5c-f1c9592b23fa-kube-api-access-jp2r8\") pod \"maas-api-e2e-api-b-e7eea9-74d6c5586f-hp7tc\" (UID: \"a4153b03-5882-419a-bb5c-f1c9592b23fa\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e7eea9-74d6c5586f-hp7tc"
Oct 31 01:30:03.672244 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:30:03.672154    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:30:03.855356 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:30:03.855278    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:30:04.064639 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:30:04.064602    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a4153b03-5882-419a-bb5c-f1c9592b23fa-maas-api-tls\") pod \"maas-api-e2e-api-b-e7eea9-74d6c5586f-hp7tc\" (UID: \"a4153b03-5882-419a-bb5c-f1c9592b23fa\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e7eea9-74d6c5586f-hp7tc"
Oct 31 01:30:04.067540 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:30:04.067512    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a4153b03-5882-419a-bb5c-f1c9592b23fa-maas-api-tls\") pod \"maas-api-e2e-api-b-e7eea9-74d6c5586f-hp7tc\" (UID: \"a4153b03-5882-419a-bb5c-f1c9592b23fa\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e7eea9-74d6c5586f-hp7tc"
Oct 31 01:30:04.347636 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:30:04.347545    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e7eea9-74d6c5586f-hp7tc"
Oct 31 01:30:04.813778 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:30:04.813741    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-e7eea9-74d6c5586f-hp7tc"]
Oct 31 01:30:04.814285 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:30:04.814258    4964 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda4153b03_5882_419a_bb5c_f1c9592b23fa.slice/crio-f2a84fa368f19fcc2914b0238f38730e9a5af216fdd4d60b682b74f1055aaed8 WatchSource:0}: Error finding container f2a84fa368f19fcc2914b0238f38730e9a5af216fdd4d60b682b74f1055aaed8: Status 404 returned error can't find the container with id f2a84fa368f19fcc2914b0238f38730e9a5af216fdd4d60b682b74f1055aaed8
Oct 31 01:30:05.197633 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:30:05.197488    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e7eea9-74d6c5586f-hp7tc" event={"ID":"a4153b03-5882-419a-bb5c-f1c9592b23fa","Type":"ContainerStarted","Data":"f2a84fa368f19fcc2914b0238f38730e9a5af216fdd4d60b682b74f1055aaed8"}
Oct 31 01:30:06.216339 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:30:06.211920    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e7eea9-74d6c5586f-hp7tc" event={"ID":"a4153b03-5882-419a-bb5c-f1c9592b23fa","Type":"ContainerStarted","Data":"058a52c23059c902e390ae4d0854ae7c11f69aaa3322ded8bc91de289aef211f"}
Oct 31 01:30:06.216339 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:30:06.212238    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e7eea9-74d6c5586f-hp7tc"
Oct 31 01:30:08.237419 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:30:08.237383    4964 generic.go:356] "Generic (PLEG): container finished" podID="fb021400-cb4b-494d-8967-5d0a7b35a2a7" containerID="b5d7a641136f7ee2afc21931dfe87e3ab16f73603bcc3424fa60a21014bad786" exitCode=60
Oct 31 01:30:08.237864 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:30:08.237463    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-7cec02-29890170-vn2k2" event={"ID":"fb021400-cb4b-494d-8967-5d0a7b35a2a7","Type":"ContainerDied","Data":"b5d7a641136f7ee2afc21931dfe87e3ab16f73603bcc3424fa60a21014bad786"}
Oct 31 01:30:08.237864 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:30:08.237732    4964 scope.go:122] "RemoveContainer" containerID="b5d7a641136f7ee2afc21931dfe87e3ab16f73603bcc3424fa60a21014bad786"
Oct 31 01:30:08.352841 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:30:08.352749    4964 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e7eea9-74d6c5586f-hp7tc" podStartSLOduration=4.157764988 podStartE2EDuration="5.352735239s" totalImagesPullingTime="1.194970251s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:30:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:30:06.353522813 +0000 UTC m=+1574.830089192" watchObservedRunningTime="2026-10-31 01:30:08.352735239 +0000 UTC m=+1576.829301659"
Oct 31 01:30:08.681384 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:30:08.681278    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:30:09.256932 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:30:09.256892    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-7cec02-29890170-vn2k2" event={"ID":"fb021400-cb4b-494d-8967-5d0a7b35a2a7","Type":"ContainerStarted","Data":"b19372f37642f5f9eb6e761178d1a88f0dd5da33ba5c39fb97c204be5ceb8025"}
Oct 31 01:30:09.378334 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:30:09.378275    4964 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-7cec02-29890170-vn2k2" podStartSLOduration=3.267948775 podStartE2EDuration="9.378259264s" totalImagesPullingTime="6.110310489s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:30:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:30:09.375619606 +0000 UTC m=+1577.852186000" watchObservedRunningTime="2026-10-31 01:30:09.378259264 +0000 UTC m=+1577.854825683"
Oct 31 01:30:09.599950 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:30:09.599804    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:30:10.266329 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:30:10.266292    4964 generic.go:356] "Generic (PLEG): container finished" podID="fb021400-cb4b-494d-8967-5d0a7b35a2a7" containerID="b19372f37642f5f9eb6e761178d1a88f0dd5da33ba5c39fb97c204be5ceb8025" exitCode=60
Oct 31 01:30:10.266783 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:30:10.266374    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-7cec02-29890170-vn2k2" event={"ID":"fb021400-cb4b-494d-8967-5d0a7b35a2a7","Type":"ContainerDied","Data":"b19372f37642f5f9eb6e761178d1a88f0dd5da33ba5c39fb97c204be5ceb8025"}
Oct 31 01:30:10.266783 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:30:10.266435    4964 scope.go:122] "RemoveContainer" containerID="b5d7a641136f7ee2afc21931dfe87e3ab16f73603bcc3424fa60a21014bad786"
Oct 31 01:30:10.266783 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:30:10.266760    4964 scope.go:122] "RemoveContainer" containerID="b19372f37642f5f9eb6e761178d1a88f0dd5da33ba5c39fb97c204be5ceb8025"
Oct 31 01:30:10.267170 ip-10-0-133-197 kubenswrapper[4964]: E1031 01:30:10.267143    4964 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-a-7cec02-29890170-vn2k2_odh-ai-gateway-infra(fb021400-cb4b-494d-8967-5d0a7b35a2a7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-7cec02-29890170-vn2k2" podUID="fb021400-cb4b-494d-8967-5d0a7b35a2a7"
Oct 31 01:30:11.249774 ip-10-0-133-197 kubenswrapper[4964]: E1031 01:30:11.249740    4964 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q56x5" containerName="extractor"
Oct 31 01:30:11.275920 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:30:11.275899    4964 scope.go:122] "RemoveContainer" containerID="b19372f37642f5f9eb6e761178d1a88f0dd5da33ba5c39fb97c204be5ceb8025"
Oct 31 01:30:11.276252 ip-10-0-133-197 kubenswrapper[4964]: E1031 01:30:11.276210    4964 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-a-7cec02-29890170-vn2k2_odh-ai-gateway-infra(fb021400-cb4b-494d-8967-5d0a7b35a2a7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-7cec02-29890170-vn2k2" podUID="fb021400-cb4b-494d-8967-5d0a7b35a2a7"
Oct 31 01:30:12.235391 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:30:12.235361    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e7eea9-74d6c5586f-hp7tc"
Oct 31 01:30:12.846579 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:30:12.846537    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:30:14.402415 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:30:14.402373    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:30:14.627480 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:30:14.627445    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:30:14.798971 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:30:14.798910    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:30:15.319948 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:30:15.319905    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:30:16.324857 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:30:16.324825    4964 generic.go:356] "Generic (PLEG): container finished" podID="50f61fea-7b83-48aa-ad74-ea2be71f683b" containerID="9eeac5cf406f68573cb3279b07b12b7f3068677631a160ef21978be114b06a04" exitCode=0
Oct 31 01:30:16.325356 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:30:16.324870    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-42c96d-openshift-default-69697959d9-p8ljq" event={"ID":"50f61fea-7b83-48aa-ad74-ea2be71f683b","Type":"ContainerDied","Data":"9eeac5cf406f68573cb3279b07b12b7f3068677631a160ef21978be114b06a04"}
Oct 31 01:30:16.325356 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:30:16.325295    4964 scope.go:122] "RemoveContainer" containerID="9eeac5cf406f68573cb3279b07b12b7f3068677631a160ef21978be114b06a04"
Oct 31 01:30:16.327159 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:30:16.327127    4964 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 01:30:16.327277 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:30:16.327218    4964 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 01:30:16.327277 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:30:16.327265    4964 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 01:30:17.333969 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:30:17.333934    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-42c96d-openshift-default-69697959d9-p8ljq" event={"ID":"50f61fea-7b83-48aa-ad74-ea2be71f683b","Type":"ContainerStarted","Data":"e949c066245e8dcc03f440670b60a4128e1960e39914711ffcf7327c1fcdf930"}
Oct 31 01:30:18.251777 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:30:18.251740    4964 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-42c96d-openshift-default-69697959d9-p8ljq"
Oct 31 01:30:18.253655 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:30:18.253614    4964 patch_prober.go:29] interesting pod/e2e-api-a-42c96d-openshift-default-69697959d9-p8ljq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" start-of-body=
Oct 31 01:30:18.253804 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:30:18.253685    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-42c96d-openshift-default-69697959d9-p8ljq" podUID="50f61fea-7b83-48aa-ad74-ea2be71f683b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused"
Oct 31 01:30:19.013341 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:30:19.012305    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:30:19.252043 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:30:19.251999    4964 patch_prober.go:29] interesting pod/e2e-api-a-42c96d-openshift-default-69697959d9-p8ljq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" start-of-body=
Oct 31 01:30:19.252252 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:30:19.252075    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-42c96d-openshift-default-69697959d9-p8ljq" podUID="50f61fea-7b83-48aa-ad74-ea2be71f683b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused"
Oct 31 01:30:19.561451 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:30:19.561416    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-42c96d-openshift-default-69697959d9-p8ljq"
Oct 31 01:30:19.643369 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:30:19.643296    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:30:19.875948 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:30:19.875861    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:30:20.251890 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:30:20.251845    4964 patch_prober.go:29] interesting pod/e2e-api-a-42c96d-openshift-default-69697959d9-p8ljq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" start-of-body=
Oct 31 01:30:20.252350 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:30:20.251903    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-42c96d-openshift-default-69697959d9-p8ljq" podUID="50f61fea-7b83-48aa-ad74-ea2be71f683b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused"
Oct 31 01:30:20.344351 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:30:20.344276    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:30:21.252008 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:30:21.251973    4964 patch_prober.go:29] interesting pod/e2e-api-a-42c96d-openshift-default-69697959d9-p8ljq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" start-of-body=
Oct 31 01:30:21.252456 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:30:21.252038    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-42c96d-openshift-default-69697959d9-p8ljq" podUID="50f61fea-7b83-48aa-ad74-ea2be71f683b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused"
Oct 31 01:30:21.774685 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:30:21.774660    4964 scope.go:122] "RemoveContainer" containerID="b19372f37642f5f9eb6e761178d1a88f0dd5da33ba5c39fb97c204be5ceb8025"
Oct 31 01:30:22.251798 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:30:22.251759    4964 patch_prober.go:29] interesting pod/e2e-api-a-42c96d-openshift-default-69697959d9-p8ljq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" start-of-body=
Oct 31 01:30:22.251996 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:30:22.251831    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-42c96d-openshift-default-69697959d9-p8ljq" podUID="50f61fea-7b83-48aa-ad74-ea2be71f683b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused"
Oct 31 01:30:22.378377 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:30:22.378296    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-7cec02-29890170-vn2k2" event={"ID":"fb021400-cb4b-494d-8967-5d0a7b35a2a7","Type":"ContainerStarted","Data":"5a1927b5deb3060832e9e44ae329d941de24ff4970e7c40d1dfd7832a3d8a550"}
Oct 31 01:30:23.257049 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:30:23.257021    4964 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-42c96d-openshift-default-69697959d9-p8ljq"
Oct 31 01:30:23.258147 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:30:23.258124    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-42c96d-openshift-default-69697959d9-p8ljq"
Oct 31 01:30:23.388850 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:30:23.388810    4964 generic.go:356] "Generic (PLEG): container finished" podID="fb021400-cb4b-494d-8967-5d0a7b35a2a7" containerID="5a1927b5deb3060832e9e44ae329d941de24ff4970e7c40d1dfd7832a3d8a550" exitCode=60
Oct 31 01:30:23.389397 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:30:23.388894    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-7cec02-29890170-vn2k2" event={"ID":"fb021400-cb4b-494d-8967-5d0a7b35a2a7","Type":"ContainerDied","Data":"5a1927b5deb3060832e9e44ae329d941de24ff4970e7c40d1dfd7832a3d8a550"}
Oct 31 01:30:23.389397 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:30:23.388972    4964 scope.go:122] "RemoveContainer" containerID="b19372f37642f5f9eb6e761178d1a88f0dd5da33ba5c39fb97c204be5ceb8025"
Oct 31 01:30:23.389630 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:30:23.389609    4964 scope.go:122] "RemoveContainer" containerID="5a1927b5deb3060832e9e44ae329d941de24ff4970e7c40d1dfd7832a3d8a550"
Oct 31 01:30:23.389965 ip-10-0-133-197 kubenswrapper[4964]: E1031 01:30:23.389943    4964 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-a-7cec02-29890170-vn2k2_odh-ai-gateway-infra(fb021400-cb4b-494d-8967-5d0a7b35a2a7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-7cec02-29890170-vn2k2" podUID="fb021400-cb4b-494d-8967-5d0a7b35a2a7"
Oct 31 01:30:23.501660 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:30:23.501617    4964 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-7cec02-29890170-vn2k2"]
Oct 31 01:30:24.210474 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:30:24.210442    4964 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-jz4tz"]
Oct 31 01:30:24.211043 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:30:24.210975    4964 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-jz4tz" podUID="a2038dbc-765f-488b-9701-c4081448ac4a" containerName="istio-proxy" containerID="cri-o://43a463f81497888dd942f1d16df68e152238a91b7f29fabdf658061d6481f1bc" gracePeriod=30
Oct 31 01:30:24.217775 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:30:24.217750    4964 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-jhg75"]
Oct 31 01:30:24.217951 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:30:24.217929    4964 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-jhg75" podUID="e190985a-446c-409d-87bb-7926a945e43c" containerName="istio-proxy" containerID="cri-o://e85cb26c6dce5fd1a4789974b9eccfbd2810499bb7f59de8a345a59164f6e2f2" gracePeriod=30
Oct 31 01:30:24.221816 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:30:24.221792    4964 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-hppbk"]
Oct 31 01:30:24.221971 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:30:24.221953    4964 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-hppbk" podUID="479904d8-8280-451e-9182-e7d15c28c13d" containerName="istio-proxy" containerID="cri-o://d3a14bedd81cb4cee73bdcd48999d61050eb6b4d9883e179b34b6c3cd0377ac8" gracePeriod=30
Oct 31 01:30:24.250445 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:30:24.250413    4964 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-ls7ft"]
Oct 31 01:30:24.250615 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:30:24.250590    4964 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-ls7ft" podUID="38e3a193-4f84-478a-910e-bed4fac148e2" containerName="istio-proxy" containerID="cri-o://b74fe5fc9f28cac3591d90f5d7113bf442f66534f83451f94e39505a09b5db56" gracePeriod=30
Oct 31 01:30:24.583406 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:30:24.583379    4964 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-7cec02-29890170-vn2k2"
Oct 31 01:30:24.597369 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:30:24.597336    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:30:24.736395 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:30:24.736365    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fb021400-cb4b-494d-8967-5d0a7b35a2a7-kube-api-access-fssjg\" (UniqueName: \"kubernetes.io/projected/fb021400-cb4b-494d-8967-5d0a7b35a2a7-kube-api-access-fssjg\") pod \"fb021400-cb4b-494d-8967-5d0a7b35a2a7\" (UID: \"fb021400-cb4b-494d-8967-5d0a7b35a2a7\") "
Oct 31 01:30:24.736591 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:30:24.736571    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fb021400-cb4b-494d-8967-5d0a7b35a2a7-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fb021400-cb4b-494d-8967-5d0a7b35a2a7-maas-api-service-ca\") pod \"fb021400-cb4b-494d-8967-5d0a7b35a2a7\" (UID: \"fb021400-cb4b-494d-8967-5d0a7b35a2a7\") "
Oct 31 01:30:24.736967 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:30:24.736923    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fb021400-cb4b-494d-8967-5d0a7b35a2a7-maas-api-service-ca" pod "fb021400-cb4b-494d-8967-5d0a7b35a2a7" (UID: "fb021400-cb4b-494d-8967-5d0a7b35a2a7"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:30:24.737334 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:30:24.737292    4964 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fb021400-cb4b-494d-8967-5d0a7b35a2a7-maas-api-service-ca\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:30:24.738800 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:30:24.738782    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb021400-cb4b-494d-8967-5d0a7b35a2a7-kube-api-access-fssjg" pod "fb021400-cb4b-494d-8967-5d0a7b35a2a7" (UID: "fb021400-cb4b-494d-8967-5d0a7b35a2a7"). InnerVolumeSpecName "kube-api-access-fssjg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:30:24.843380 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:30:24.843241    4964 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fssjg\" (UniqueName: \"kubernetes.io/projected/fb021400-cb4b-494d-8967-5d0a7b35a2a7-kube-api-access-fssjg\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:30:25.407858 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:30:25.407828    4964 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-7cec02-29890170-vn2k2"
Oct 31 01:30:25.408051 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:30:25.407863    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-7cec02-29890170-vn2k2" event={"ID":"fb021400-cb4b-494d-8967-5d0a7b35a2a7","Type":"ContainerDied","Data":"3eeb086c10b4b802754774401667dfdc5ab8bd0f772d97a4aae11e3beafe8678"}
Oct 31 01:30:25.408051 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:30:25.407927    4964 scope.go:122] "RemoveContainer" containerID="5a1927b5deb3060832e9e44ae329d941de24ff4970e7c40d1dfd7832a3d8a550"
Oct 31 01:30:25.575180 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:30:25.575147    4964 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-7cec02-29890170-vn2k2"]
Oct 31 01:30:25.580611 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:30:25.580578    4964 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-7cec02-29890170-vn2k2"]
Oct 31 01:30:25.778772 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:30:25.778733    4964 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fb021400-cb4b-494d-8967-5d0a7b35a2a7" path="/var/lib/kubelet/pods/fb021400-cb4b-494d-8967-5d0a7b35a2a7/volumes"
Oct 31 01:30:25.958023 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:30:25.957993    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:30:26.139121 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:30:26.139034    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:30:27.009492 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:30:27.009455    4964 patch_prober.go:29] interesting pod/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-jhg75 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 01:30:27.009890 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:30:27.009519    4964 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-jhg75" podUID="e190985a-446c-409d-87bb-7926a945e43c" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 01:30:29.090608 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:30:29.090576    4964 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-14ca09-54b56b9987-h9ws9"]
Oct 31 01:30:29.091491 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:30:29.091471    4964 cpu_manager.go:414] "containerMap: removing container" podUID="fb021400-cb4b-494d-8967-5d0a7b35a2a7" containerName="cleanup"
Oct 31 01:30:29.091491 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:30:29.091491    4964 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fb021400-cb4b-494d-8967-5d0a7b35a2a7" containerName="cleanup"
Oct 31 01:30:29.091614 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:30:29.091546    4964 cpu_manager.go:414] "containerMap: removing container" podUID="fb021400-cb4b-494d-8967-5d0a7b35a2a7" containerName="cleanup"
Oct 31 01:30:29.091614 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:30:29.091551    4964 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fb021400-cb4b-494d-8967-5d0a7b35a2a7" containerName="cleanup"
Oct 31 01:30:29.091682 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:30:29.091658    4964 memory_manager.go:365] "RemoveStaleState removing state" podUID="fb021400-cb4b-494d-8967-5d0a7b35a2a7" containerName="cleanup"
Oct 31 01:30:29.091682 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:30:29.091675    4964 memory_manager.go:365] "RemoveStaleState removing state" podUID="fb021400-cb4b-494d-8967-5d0a7b35a2a7" containerName="cleanup"
Oct 31 01:30:29.096979 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:30:29.096961    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-14ca09-54b56b9987-h9ws9"
Oct 31 01:30:29.152186 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:30:29.152150    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-14ca09-54b56b9987-h9ws9"]
Oct 31 01:30:29.196224 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:30:29.196196    4964 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-14ca09-serving-cert\""
Oct 31 01:30:29.208212 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:30:29.208173    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/21025738-82f8-45ef-a53f-f7761d96ca2f-openshift-service-ca\") pod \"maas-api-e2e-api-b-14ca09-54b56b9987-h9ws9\" (UID: \"21025738-82f8-45ef-a53f-f7761d96ca2f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-14ca09-54b56b9987-h9ws9"
Oct 31 01:30:29.208434 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:30:29.208247    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lknnw\" (UniqueName: \"kubernetes.io/projected/21025738-82f8-45ef-a53f-f7761d96ca2f-kube-api-access-lknnw\") pod \"maas-api-e2e-api-b-14ca09-54b56b9987-h9ws9\" (UID: \"21025738-82f8-45ef-a53f-f7761d96ca2f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-14ca09-54b56b9987-h9ws9"
Oct 31 01:30:29.208434 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:30:29.208402    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/21025738-82f8-45ef-a53f-f7761d96ca2f-maas-api-tls\") pod \"maas-api-e2e-api-b-14ca09-54b56b9987-h9ws9\" (UID: \"21025738-82f8-45ef-a53f-f7761d96ca2f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-14ca09-54b56b9987-h9ws9"
Oct 31 01:30:29.208579 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:30:29.208526    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/21025738-82f8-45ef-a53f-f7761d96ca2f-metrics-certs\") pod \"maas-api-e2e-api-b-14ca09-54b56b9987-h9ws9\" (UID: \"21025738-82f8-45ef-a53f-f7761d96ca2f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-14ca09-54b56b9987-h9ws9"
Oct 31 01:30:29.310253 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:30:29.310220    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/21025738-82f8-45ef-a53f-f7761d96ca2f-maas-api-tls\") pod \"maas-api-e2e-api-b-14ca09-54b56b9987-h9ws9\" (UID: \"21025738-82f8-45ef-a53f-f7761d96ca2f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-14ca09-54b56b9987-h9ws9"
Oct 31 01:30:29.310439 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:30:29.310348    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/21025738-82f8-45ef-a53f-f7761d96ca2f-metrics-certs\") pod \"maas-api-e2e-api-b-14ca09-54b56b9987-h9ws9\" (UID: \"21025738-82f8-45ef-a53f-f7761d96ca2f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-14ca09-54b56b9987-h9ws9"
Oct 31 01:30:29.310439 ip-10-0-133-197 kubenswrapper[4964]: E1031 01:30:29.310402    4964 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-14ca09-serving-cert: secret "maas-api-e2e-api-b-14ca09-serving-cert" not found
Oct 31 01:30:29.310554 ip-10-0-133-197 kubenswrapper[4964]: E1031 01:30:29.310477    4964 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/21025738-82f8-45ef-a53f-f7761d96ca2f-maas-api-tls podName:21025738-82f8-45ef-a53f-f7761d96ca2f nodeName:}" failed. No retries permitted until 2026-10-31 01:30:29.810455813 +0000 UTC m=+1598.287022175 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/21025738-82f8-45ef-a53f-f7761d96ca2f-maas-api-tls") pod "maas-api-e2e-api-b-14ca09-54b56b9987-h9ws9" (UID: "21025738-82f8-45ef-a53f-f7761d96ca2f") : secret "maas-api-e2e-api-b-14ca09-serving-cert" not found
Oct 31 01:30:29.310646 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:30:29.310610    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/21025738-82f8-45ef-a53f-f7761d96ca2f-openshift-service-ca\") pod \"maas-api-e2e-api-b-14ca09-54b56b9987-h9ws9\" (UID: \"21025738-82f8-45ef-a53f-f7761d96ca2f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-14ca09-54b56b9987-h9ws9"
Oct 31 01:30:29.310711 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:30:29.310678    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lknnw\" (UniqueName: \"kubernetes.io/projected/21025738-82f8-45ef-a53f-f7761d96ca2f-kube-api-access-lknnw\") pod \"maas-api-e2e-api-b-14ca09-54b56b9987-h9ws9\" (UID: \"21025738-82f8-45ef-a53f-f7761d96ca2f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-14ca09-54b56b9987-h9ws9"
Oct 31 01:30:29.311609 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:30:29.311580    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/21025738-82f8-45ef-a53f-f7761d96ca2f-openshift-service-ca\") pod \"maas-api-e2e-api-b-14ca09-54b56b9987-h9ws9\" (UID: \"21025738-82f8-45ef-a53f-f7761d96ca2f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-14ca09-54b56b9987-h9ws9"
Oct 31 01:30:29.313541 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:30:29.313519    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/21025738-82f8-45ef-a53f-f7761d96ca2f-metrics-certs\") pod \"maas-api-e2e-api-b-14ca09-54b56b9987-h9ws9\" (UID: \"21025738-82f8-45ef-a53f-f7761d96ca2f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-14ca09-54b56b9987-h9ws9"
Oct 31 01:30:29.328586 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:30:29.328537    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:30:29.376002 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:30:29.375892    4964 patch_prober.go:29] interesting pod/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-jz4tz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body=
Oct 31 01:30:29.376111 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:30:29.376061    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lknnw\" (UniqueName: \"kubernetes.io/projected/21025738-82f8-45ef-a53f-f7761d96ca2f-kube-api-access-lknnw\") pod \"maas-api-e2e-api-b-14ca09-54b56b9987-h9ws9\" (UID: \"21025738-82f8-45ef-a53f-f7761d96ca2f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-14ca09-54b56b9987-h9ws9"
Oct 31 01:30:29.376676 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:30:29.376209    4964 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-jz4tz" podUID="a2038dbc-765f-488b-9701-c4081448ac4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused"
Oct 31 01:30:29.454382 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:30:29.454149    4964 generic.go:356] "Generic (PLEG): container finished" podID="479904d8-8280-451e-9182-e7d15c28c13d" containerID="d3a14bedd81cb4cee73bdcd48999d61050eb6b4d9883e179b34b6c3cd0377ac8" exitCode=0
Oct 31 01:30:29.454382 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:30:29.454275    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-hppbk" event={"ID":"479904d8-8280-451e-9182-e7d15c28c13d","Type":"ContainerDied","Data":"d3a14bedd81cb4cee73bdcd48999d61050eb6b4d9883e179b34b6c3cd0377ac8"}
Oct 31 01:30:29.456890 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:30:29.456750    4964 generic.go:356] "Generic (PLEG): container finished" podID="a2038dbc-765f-488b-9701-c4081448ac4a" containerID="43a463f81497888dd942f1d16df68e152238a91b7f29fabdf658061d6481f1bc" exitCode=0
Oct 31 01:30:29.456890 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:30:29.456828    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-jz4tz" event={"ID":"a2038dbc-765f-488b-9701-c4081448ac4a","Type":"ContainerDied","Data":"43a463f81497888dd942f1d16df68e152238a91b7f29fabdf658061d6481f1bc"}
Oct 31 01:30:29.459474 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:30:29.459079    4964 generic.go:356] "Generic (PLEG): container finished" podID="38e3a193-4f84-478a-910e-bed4fac148e2" containerID="b74fe5fc9f28cac3591d90f5d7113bf442f66534f83451f94e39505a09b5db56" exitCode=0
Oct 31 01:30:29.459474 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:30:29.459389    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-ls7ft" event={"ID":"38e3a193-4f84-478a-910e-bed4fac148e2","Type":"ContainerDied","Data":"b74fe5fc9f28cac3591d90f5d7113bf442f66534f83451f94e39505a09b5db56"}
Oct 31 01:30:29.461946 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:30:29.461867    4964 generic.go:356] "Generic (PLEG): container finished" podID="e190985a-446c-409d-87bb-7926a945e43c" containerID="e85cb26c6dce5fd1a4789974b9eccfbd2810499bb7f59de8a345a59164f6e2f2" exitCode=0
Oct 31 01:30:29.461946 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:30:29.461906    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-jhg75" event={"ID":"e190985a-446c-409d-87bb-7926a945e43c","Type":"ContainerDied","Data":"e85cb26c6dce5fd1a4789974b9eccfbd2810499bb7f59de8a345a59164f6e2f2"}
Oct 31 01:30:29.514423 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:30:29.511153    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:30:29.694967 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:30:29.694934    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:30:29.710768 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:30:29.710728    4964 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-jz4tz"
Oct 31 01:30:29.749517 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:30:29.749491    4964 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-ls7ft"
Oct 31 01:30:29.757012 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:30:29.756990    4964 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-jhg75"
Oct 31 01:30:29.759120 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:30:29.759100    4964 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-hppbk"
Oct 31 01:30:29.842502 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:30:29.842471    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a2038dbc-765f-488b-9701-c4081448ac4a-istio-token\" (UniqueName: \"kubernetes.io/projected/a2038dbc-765f-488b-9701-c4081448ac4a-istio-token\") pod \"a2038dbc-765f-488b-9701-c4081448ac4a\" (UID: \"a2038dbc-765f-488b-9701-c4081448ac4a\") "
Oct 31 01:30:29.842843 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:30:29.842820    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a2038dbc-765f-488b-9701-c4081448ac4a-kube-api-access-c7jcs\" (UniqueName: \"kubernetes.io/projected/a2038dbc-765f-488b-9701-c4081448ac4a-kube-api-access-c7jcs\") pod \"a2038dbc-765f-488b-9701-c4081448ac4a\" (UID: \"a2038dbc-765f-488b-9701-c4081448ac4a\") "
Oct 31 01:30:29.842920 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:30:29.842894    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a2038dbc-765f-488b-9701-c4081448ac4a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a2038dbc-765f-488b-9701-c4081448ac4a-workload-certs\") pod \"a2038dbc-765f-488b-9701-c4081448ac4a\" (UID: \"a2038dbc-765f-488b-9701-c4081448ac4a\") "
Oct 31 01:30:29.843246 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:30:29.843225    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a2038dbc-765f-488b-9701-c4081448ac4a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2038dbc-765f-488b-9701-c4081448ac4a-credential-socket\") pod \"a2038dbc-765f-488b-9701-c4081448ac4a\" (UID: \"a2038dbc-765f-488b-9701-c4081448ac4a\") "
Oct 31 01:30:29.843480 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:30:29.843460    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a2038dbc-765f-488b-9701-c4081448ac4a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2038dbc-765f-488b-9701-c4081448ac4a-workload-socket\") pod \"a2038dbc-765f-488b-9701-c4081448ac4a\" (UID: \"a2038dbc-765f-488b-9701-c4081448ac4a\") "
Oct 31 01:30:29.843610 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:30:29.843519    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a2038dbc-765f-488b-9701-c4081448ac4a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a2038dbc-765f-488b-9701-c4081448ac4a-istio-podinfo\") pod \"a2038dbc-765f-488b-9701-c4081448ac4a\" (UID: \"a2038dbc-765f-488b-9701-c4081448ac4a\") "
Oct 31 01:30:29.843610 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:30:29.843588    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a2038dbc-765f-488b-9701-c4081448ac4a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a2038dbc-765f-488b-9701-c4081448ac4a-istio-envoy\") pod \"a2038dbc-765f-488b-9701-c4081448ac4a\" (UID: \"a2038dbc-765f-488b-9701-c4081448ac4a\") "
Oct 31 01:30:29.843766 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:30:29.843617    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a2038dbc-765f-488b-9701-c4081448ac4a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a2038dbc-765f-488b-9701-c4081448ac4a-istio-data\") pod \"a2038dbc-765f-488b-9701-c4081448ac4a\" (UID: \"a2038dbc-765f-488b-9701-c4081448ac4a\") "
Oct 31 01:30:29.843766 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:30:29.843636    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2038dbc-765f-488b-9701-c4081448ac4a-credential-socket" pod "a2038dbc-765f-488b-9701-c4081448ac4a" (UID: "a2038dbc-765f-488b-9701-c4081448ac4a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:30:29.843766 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:30:29.843658    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a2038dbc-765f-488b-9701-c4081448ac4a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a2038dbc-765f-488b-9701-c4081448ac4a-istiod-ca-cert\") pod \"a2038dbc-765f-488b-9701-c4081448ac4a\" (UID: \"a2038dbc-765f-488b-9701-c4081448ac4a\") "
Oct 31 01:30:29.843996 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:30:29.843852    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2038dbc-765f-488b-9701-c4081448ac4a-workload-certs" pod "a2038dbc-765f-488b-9701-c4081448ac4a" (UID: "a2038dbc-765f-488b-9701-c4081448ac4a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:30:29.846344 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:30:29.844339    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/21025738-82f8-45ef-a53f-f7761d96ca2f-maas-api-tls\") pod \"maas-api-e2e-api-b-14ca09-54b56b9987-h9ws9\" (UID: \"21025738-82f8-45ef-a53f-f7761d96ca2f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-14ca09-54b56b9987-h9ws9"
Oct 31 01:30:29.846344 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:30:29.845000    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a2038dbc-765f-488b-9701-c4081448ac4a-istiod-ca-cert" pod "a2038dbc-765f-488b-9701-c4081448ac4a" (UID: "a2038dbc-765f-488b-9701-c4081448ac4a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:30:29.846834 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:30:29.846810    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2038dbc-765f-488b-9701-c4081448ac4a-workload-socket" pod "a2038dbc-765f-488b-9701-c4081448ac4a" (UID: "a2038dbc-765f-488b-9701-c4081448ac4a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:30:29.846893 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:30:29.846879    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2038dbc-765f-488b-9701-c4081448ac4a-istio-data" pod "a2038dbc-765f-488b-9701-c4081448ac4a" (UID: "a2038dbc-765f-488b-9701-c4081448ac4a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:30:29.847195 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:30:29.847170    4964 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a2038dbc-765f-488b-9701-c4081448ac4a-workload-certs\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:30:29.847308 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:30:29.847200    4964 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2038dbc-765f-488b-9701-c4081448ac4a-credential-socket\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:30:29.849672 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:30:29.849632    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a2038dbc-765f-488b-9701-c4081448ac4a-istio-podinfo" pod "a2038dbc-765f-488b-9701-c4081448ac4a" (UID: "a2038dbc-765f-488b-9701-c4081448ac4a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 01:30:29.849949 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:30:29.849918    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a2038dbc-765f-488b-9701-c4081448ac4a-istio-token" pod "a2038dbc-765f-488b-9701-c4081448ac4a" (UID: "a2038dbc-765f-488b-9701-c4081448ac4a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:30:29.850266 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:30:29.850242    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2038dbc-765f-488b-9701-c4081448ac4a-istio-envoy" pod "a2038dbc-765f-488b-9701-c4081448ac4a" (UID: "a2038dbc-765f-488b-9701-c4081448ac4a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:30:29.850448 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:30:29.850426    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a2038dbc-765f-488b-9701-c4081448ac4a-kube-api-access-c7jcs" pod "a2038dbc-765f-488b-9701-c4081448ac4a" (UID: "a2038dbc-765f-488b-9701-c4081448ac4a"). InnerVolumeSpecName "kube-api-access-c7jcs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:30:29.850665 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:30:29.850641    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/21025738-82f8-45ef-a53f-f7761d96ca2f-maas-api-tls\") pod \"maas-api-e2e-api-b-14ca09-54b56b9987-h9ws9\" (UID: \"21025738-82f8-45ef-a53f-f7761d96ca2f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-14ca09-54b56b9987-h9ws9"
Oct 31 01:30:29.951213 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:30:29.948277    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/479904d8-8280-451e-9182-e7d15c28c13d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/479904d8-8280-451e-9182-e7d15c28c13d-istio-data\") pod \"479904d8-8280-451e-9182-e7d15c28c13d\" (UID: \"479904d8-8280-451e-9182-e7d15c28c13d\") "
Oct 31 01:30:29.951213 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:30:29.948384    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/479904d8-8280-451e-9182-e7d15c28c13d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/479904d8-8280-451e-9182-e7d15c28c13d-istio-envoy\") pod \"479904d8-8280-451e-9182-e7d15c28c13d\" (UID: \"479904d8-8280-451e-9182-e7d15c28c13d\") "
Oct 31 01:30:29.951213 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:30:29.948422    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e190985a-446c-409d-87bb-7926a945e43c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e190985a-446c-409d-87bb-7926a945e43c-istio-envoy\") pod \"e190985a-446c-409d-87bb-7926a945e43c\" (UID: \"e190985a-446c-409d-87bb-7926a945e43c\") "
Oct 31 01:30:29.951213 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:30:29.948479    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e190985a-446c-409d-87bb-7926a945e43c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e190985a-446c-409d-87bb-7926a945e43c-workload-socket\") pod \"e190985a-446c-409d-87bb-7926a945e43c\" (UID: \"e190985a-446c-409d-87bb-7926a945e43c\") "
Oct 31 01:30:29.951213 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:30:29.948507    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e190985a-446c-409d-87bb-7926a945e43c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e190985a-446c-409d-87bb-7926a945e43c-credential-socket\") pod \"e190985a-446c-409d-87bb-7926a945e43c\" (UID: \"e190985a-446c-409d-87bb-7926a945e43c\") "
Oct 31 01:30:29.951213 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:30:29.948549    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/38e3a193-4f84-478a-910e-bed4fac148e2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/38e3a193-4f84-478a-910e-bed4fac148e2-credential-socket\") pod \"38e3a193-4f84-478a-910e-bed4fac148e2\" (UID: \"38e3a193-4f84-478a-910e-bed4fac148e2\") "
Oct 31 01:30:29.951213 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:30:29.948578    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e190985a-446c-409d-87bb-7926a945e43c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e190985a-446c-409d-87bb-7926a945e43c-istiod-ca-cert\") pod \"e190985a-446c-409d-87bb-7926a945e43c\" (UID: \"e190985a-446c-409d-87bb-7926a945e43c\") "
Oct 31 01:30:29.951213 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:30:29.948629    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/38e3a193-4f84-478a-910e-bed4fac148e2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/38e3a193-4f84-478a-910e-bed4fac148e2-istio-data\") pod \"38e3a193-4f84-478a-910e-bed4fac148e2\" (UID: \"38e3a193-4f84-478a-910e-bed4fac148e2\") "
Oct 31 01:30:29.951213 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:30:29.948670    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/479904d8-8280-451e-9182-e7d15c28c13d-istio-token\" (UniqueName: \"kubernetes.io/projected/479904d8-8280-451e-9182-e7d15c28c13d-istio-token\") pod \"479904d8-8280-451e-9182-e7d15c28c13d\" (UID: \"479904d8-8280-451e-9182-e7d15c28c13d\") "
Oct 31 01:30:29.951213 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:30:29.948668    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/479904d8-8280-451e-9182-e7d15c28c13d-istio-data" pod "479904d8-8280-451e-9182-e7d15c28c13d" (UID: "479904d8-8280-451e-9182-e7d15c28c13d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:30:29.951213 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:30:29.948720    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e190985a-446c-409d-87bb-7926a945e43c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e190985a-446c-409d-87bb-7926a945e43c-istio-podinfo\") pod \"e190985a-446c-409d-87bb-7926a945e43c\" (UID: \"e190985a-446c-409d-87bb-7926a945e43c\") "
Oct 31 01:30:29.951213 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:30:29.948750    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/479904d8-8280-451e-9182-e7d15c28c13d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/479904d8-8280-451e-9182-e7d15c28c13d-credential-socket\") pod \"479904d8-8280-451e-9182-e7d15c28c13d\" (UID: \"479904d8-8280-451e-9182-e7d15c28c13d\") "
Oct 31 01:30:29.951213 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:30:29.948776    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/38e3a193-4f84-478a-910e-bed4fac148e2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/38e3a193-4f84-478a-910e-bed4fac148e2-istiod-ca-cert\") pod \"38e3a193-4f84-478a-910e-bed4fac148e2\" (UID: \"38e3a193-4f84-478a-910e-bed4fac148e2\") "
Oct 31 01:30:29.951213 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:30:29.948814    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/479904d8-8280-451e-9182-e7d15c28c13d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/479904d8-8280-451e-9182-e7d15c28c13d-istiod-ca-cert\") pod \"479904d8-8280-451e-9182-e7d15c28c13d\" (UID: \"479904d8-8280-451e-9182-e7d15c28c13d\") "
Oct 31 01:30:29.951213 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:30:29.948841    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/38e3a193-4f84-478a-910e-bed4fac148e2-istio-token\" (UniqueName: \"kubernetes.io/projected/38e3a193-4f84-478a-910e-bed4fac148e2-istio-token\") pod \"38e3a193-4f84-478a-910e-bed4fac148e2\" (UID: \"38e3a193-4f84-478a-910e-bed4fac148e2\") "
Oct 31 01:30:29.951213 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:30:29.948874    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/38e3a193-4f84-478a-910e-bed4fac148e2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/38e3a193-4f84-478a-910e-bed4fac148e2-istio-envoy\") pod \"38e3a193-4f84-478a-910e-bed4fac148e2\" (UID: \"38e3a193-4f84-478a-910e-bed4fac148e2\") "
Oct 31 01:30:29.951213 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:30:29.948880    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e190985a-446c-409d-87bb-7926a945e43c-credential-socket" pod "e190985a-446c-409d-87bb-7926a945e43c" (UID: "e190985a-446c-409d-87bb-7926a945e43c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:30:29.951213 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:30:29.948910    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/38e3a193-4f84-478a-910e-bed4fac148e2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/38e3a193-4f84-478a-910e-bed4fac148e2-workload-socket\") pod \"38e3a193-4f84-478a-910e-bed4fac148e2\" (UID: \"38e3a193-4f84-478a-910e-bed4fac148e2\") "
Oct 31 01:30:29.951213 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:30:29.948983    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e190985a-446c-409d-87bb-7926a945e43c-kube-api-access-ldhx6\" (UniqueName: \"kubernetes.io/projected/e190985a-446c-409d-87bb-7926a945e43c-kube-api-access-ldhx6\") pod \"e190985a-446c-409d-87bb-7926a945e43c\" (UID: \"e190985a-446c-409d-87bb-7926a945e43c\") "
Oct 31 01:30:29.951213 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:30:29.949048    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/479904d8-8280-451e-9182-e7d15c28c13d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/479904d8-8280-451e-9182-e7d15c28c13d-workload-certs\") pod \"479904d8-8280-451e-9182-e7d15c28c13d\" (UID: \"479904d8-8280-451e-9182-e7d15c28c13d\") "
Oct 31 01:30:29.951213 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:30:29.949075    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e190985a-446c-409d-87bb-7926a945e43c-istio-token\" (UniqueName: \"kubernetes.io/projected/e190985a-446c-409d-87bb-7926a945e43c-istio-token\") pod \"e190985a-446c-409d-87bb-7926a945e43c\" (UID: \"e190985a-446c-409d-87bb-7926a945e43c\") "
Oct 31 01:30:29.951213 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:30:29.949139    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/38e3a193-4f84-478a-910e-bed4fac148e2-kube-api-access-ln4sn\" (UniqueName: \"kubernetes.io/projected/38e3a193-4f84-478a-910e-bed4fac148e2-kube-api-access-ln4sn\") pod \"38e3a193-4f84-478a-910e-bed4fac148e2\" (UID: \"38e3a193-4f84-478a-910e-bed4fac148e2\") "
Oct 31 01:30:29.951213 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:30:29.949184    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/38e3a193-4f84-478a-910e-bed4fac148e2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/38e3a193-4f84-478a-910e-bed4fac148e2-istio-podinfo\") pod \"38e3a193-4f84-478a-910e-bed4fac148e2\" (UID: \"38e3a193-4f84-478a-910e-bed4fac148e2\") "
Oct 31 01:30:29.951213 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:30:29.949225    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e190985a-446c-409d-87bb-7926a945e43c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e190985a-446c-409d-87bb-7926a945e43c-istio-data\") pod \"e190985a-446c-409d-87bb-7926a945e43c\" (UID: \"e190985a-446c-409d-87bb-7926a945e43c\") "
Oct 31 01:30:29.951213 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:30:29.949263    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/479904d8-8280-451e-9182-e7d15c28c13d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/479904d8-8280-451e-9182-e7d15c28c13d-workload-socket\") pod \"479904d8-8280-451e-9182-e7d15c28c13d\" (UID: \"479904d8-8280-451e-9182-e7d15c28c13d\") "
Oct 31 01:30:29.951213 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:30:29.949294    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/479904d8-8280-451e-9182-e7d15c28c13d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/479904d8-8280-451e-9182-e7d15c28c13d-istio-podinfo\") pod \"479904d8-8280-451e-9182-e7d15c28c13d\" (UID: \"479904d8-8280-451e-9182-e7d15c28c13d\") "
Oct 31 01:30:29.951213 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:30:29.949438    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e190985a-446c-409d-87bb-7926a945e43c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e190985a-446c-409d-87bb-7926a945e43c-workload-certs\") pod \"e190985a-446c-409d-87bb-7926a945e43c\" (UID: \"e190985a-446c-409d-87bb-7926a945e43c\") "
Oct 31 01:30:29.951213 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:30:29.949470    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/479904d8-8280-451e-9182-e7d15c28c13d-kube-api-access-b8cxw\" (UniqueName: \"kubernetes.io/projected/479904d8-8280-451e-9182-e7d15c28c13d-kube-api-access-b8cxw\") pod \"479904d8-8280-451e-9182-e7d15c28c13d\" (UID: \"479904d8-8280-451e-9182-e7d15c28c13d\") "
Oct 31 01:30:29.951213 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:30:29.949534    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/38e3a193-4f84-478a-910e-bed4fac148e2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/38e3a193-4f84-478a-910e-bed4fac148e2-workload-certs\") pod \"38e3a193-4f84-478a-910e-bed4fac148e2\" (UID: \"38e3a193-4f84-478a-910e-bed4fac148e2\") "
Oct 31 01:30:29.951213 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:30:29.949931    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/38e3a193-4f84-478a-910e-bed4fac148e2-istiod-ca-cert" pod "38e3a193-4f84-478a-910e-bed4fac148e2" (UID: "38e3a193-4f84-478a-910e-bed4fac148e2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:30:29.951213 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:30:29.950339    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e190985a-446c-409d-87bb-7926a945e43c-workload-socket" pod "e190985a-446c-409d-87bb-7926a945e43c" (UID: "e190985a-446c-409d-87bb-7926a945e43c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:30:29.951213 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:30:29.950959    4964 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a2038dbc-765f-488b-9701-c4081448ac4a-istio-token\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:30:29.951213 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:30:29.950983    4964 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c7jcs\" (UniqueName: \"kubernetes.io/projected/a2038dbc-765f-488b-9701-c4081448ac4a-kube-api-access-c7jcs\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:30:29.951213 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:30:29.950997    4964 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/479904d8-8280-451e-9182-e7d15c28c13d-istio-data\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:30:29.951213 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:30:29.951010    4964 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e190985a-446c-409d-87bb-7926a945e43c-credential-socket\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:30:29.951213 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:30:29.951022    4964 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/38e3a193-4f84-478a-910e-bed4fac148e2-istiod-ca-cert\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:30:29.951213 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:30:29.951035    4964 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2038dbc-765f-488b-9701-c4081448ac4a-workload-socket\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:30:29.951213 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:30:29.951050    4964 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a2038dbc-765f-488b-9701-c4081448ac4a-istio-podinfo\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:30:29.951213 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:30:29.951064    4964 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a2038dbc-765f-488b-9701-c4081448ac4a-istio-envoy\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:30:29.951213 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:30:29.951078    4964 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a2038dbc-765f-488b-9701-c4081448ac4a-istio-data\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:30:29.951213 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:30:29.951092    4964 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a2038dbc-765f-488b-9701-c4081448ac4a-istiod-ca-cert\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:30:29.953479 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:30:29.951529    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/479904d8-8280-451e-9182-e7d15c28c13d-workload-socket" pod "479904d8-8280-451e-9182-e7d15c28c13d" (UID: "479904d8-8280-451e-9182-e7d15c28c13d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:30:29.953479 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:30:29.951791    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38e3a193-4f84-478a-910e-bed4fac148e2-credential-socket" pod "38e3a193-4f84-478a-910e-bed4fac148e2" (UID: "38e3a193-4f84-478a-910e-bed4fac148e2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:30:29.953479 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:30:29.951925    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38e3a193-4f84-478a-910e-bed4fac148e2-istio-data" pod "38e3a193-4f84-478a-910e-bed4fac148e2" (UID: "38e3a193-4f84-478a-910e-bed4fac148e2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:30:29.953479 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:30:29.951955    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e190985a-446c-409d-87bb-7926a945e43c-istio-envoy" pod "e190985a-446c-409d-87bb-7926a945e43c" (UID: "e190985a-446c-409d-87bb-7926a945e43c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:30:29.953479 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:30:29.951976    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/479904d8-8280-451e-9182-e7d15c28c13d-credential-socket" pod "479904d8-8280-451e-9182-e7d15c28c13d" (UID: "479904d8-8280-451e-9182-e7d15c28c13d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:30:29.953479 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:30:29.952430    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e190985a-446c-409d-87bb-7926a945e43c-istiod-ca-cert" pod "e190985a-446c-409d-87bb-7926a945e43c" (UID: "e190985a-446c-409d-87bb-7926a945e43c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:30:29.954068 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:30:29.953803    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38e3a193-4f84-478a-910e-bed4fac148e2-workload-certs" pod "38e3a193-4f84-478a-910e-bed4fac148e2" (UID: "38e3a193-4f84-478a-910e-bed4fac148e2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:30:29.954635 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:30:29.954436    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e190985a-446c-409d-87bb-7926a945e43c-istio-podinfo" pod "e190985a-446c-409d-87bb-7926a945e43c" (UID: "e190985a-446c-409d-87bb-7926a945e43c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 01:30:29.954788 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:30:29.954759    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/479904d8-8280-451e-9182-e7d15c28c13d-workload-certs" pod "479904d8-8280-451e-9182-e7d15c28c13d" (UID: "479904d8-8280-451e-9182-e7d15c28c13d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:30:29.954902 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:30:29.954783    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38e3a193-4f84-478a-910e-bed4fac148e2-istio-envoy" pod "38e3a193-4f84-478a-910e-bed4fac148e2" (UID: "38e3a193-4f84-478a-910e-bed4fac148e2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:30:29.955148 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:30:29.955121    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/479904d8-8280-451e-9182-e7d15c28c13d-istiod-ca-cert" pod "479904d8-8280-451e-9182-e7d15c28c13d" (UID: "479904d8-8280-451e-9182-e7d15c28c13d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:30:29.955731 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:30:29.955700    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/479904d8-8280-451e-9182-e7d15c28c13d-istio-envoy" pod "479904d8-8280-451e-9182-e7d15c28c13d" (UID: "479904d8-8280-451e-9182-e7d15c28c13d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:30:29.955895 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:30:29.955879    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e190985a-446c-409d-87bb-7926a945e43c-istio-data" pod "e190985a-446c-409d-87bb-7926a945e43c" (UID: "e190985a-446c-409d-87bb-7926a945e43c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:30:29.956067 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:30:29.956049    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e190985a-446c-409d-87bb-7926a945e43c-workload-certs" pod "e190985a-446c-409d-87bb-7926a945e43c" (UID: "e190985a-446c-409d-87bb-7926a945e43c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:30:29.956131 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:30:29.956048    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/479904d8-8280-451e-9182-e7d15c28c13d-istio-token" pod "479904d8-8280-451e-9182-e7d15c28c13d" (UID: "479904d8-8280-451e-9182-e7d15c28c13d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:30:29.956239 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:30:29.956220    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38e3a193-4f84-478a-910e-bed4fac148e2-workload-socket" pod "38e3a193-4f84-478a-910e-bed4fac148e2" (UID: "38e3a193-4f84-478a-910e-bed4fac148e2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:30:29.957247 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:30:29.957222    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/38e3a193-4f84-478a-910e-bed4fac148e2-istio-podinfo" pod "38e3a193-4f84-478a-910e-bed4fac148e2" (UID: "38e3a193-4f84-478a-910e-bed4fac148e2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 01:30:29.958983 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:30:29.958946    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e190985a-446c-409d-87bb-7926a945e43c-kube-api-access-ldhx6" pod "e190985a-446c-409d-87bb-7926a945e43c" (UID: "e190985a-446c-409d-87bb-7926a945e43c"). InnerVolumeSpecName "kube-api-access-ldhx6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:30:29.959095 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:30:29.958985    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/38e3a193-4f84-478a-910e-bed4fac148e2-istio-token" pod "38e3a193-4f84-478a-910e-bed4fac148e2" (UID: "38e3a193-4f84-478a-910e-bed4fac148e2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:30:29.959095 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:30:29.959020    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/479904d8-8280-451e-9182-e7d15c28c13d-istio-podinfo" pod "479904d8-8280-451e-9182-e7d15c28c13d" (UID: "479904d8-8280-451e-9182-e7d15c28c13d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 01:30:29.959217 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:30:29.959201    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/479904d8-8280-451e-9182-e7d15c28c13d-kube-api-access-b8cxw" pod "479904d8-8280-451e-9182-e7d15c28c13d" (UID: "479904d8-8280-451e-9182-e7d15c28c13d"). InnerVolumeSpecName "kube-api-access-b8cxw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:30:29.959528 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:30:29.959503    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e190985a-446c-409d-87bb-7926a945e43c-istio-token" pod "e190985a-446c-409d-87bb-7926a945e43c" (UID: "e190985a-446c-409d-87bb-7926a945e43c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:30:29.959528 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:30:29.959519    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/38e3a193-4f84-478a-910e-bed4fac148e2-kube-api-access-ln4sn" pod "38e3a193-4f84-478a-910e-bed4fac148e2" (UID: "38e3a193-4f84-478a-910e-bed4fac148e2"). InnerVolumeSpecName "kube-api-access-ln4sn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:30:30.013185 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:30:30.013139    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-14ca09-54b56b9987-h9ws9"
Oct 31 01:30:30.058276 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:30:30.058148    4964 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e190985a-446c-409d-87bb-7926a945e43c-workload-certs\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:30:30.058276 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:30:30.058194    4964 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b8cxw\" (UniqueName: \"kubernetes.io/projected/479904d8-8280-451e-9182-e7d15c28c13d-kube-api-access-b8cxw\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:30:30.058276 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:30:30.058212    4964 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/38e3a193-4f84-478a-910e-bed4fac148e2-workload-certs\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:30:30.058276 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:30:30.058226    4964 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/479904d8-8280-451e-9182-e7d15c28c13d-istio-envoy\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:30:30.058276 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:30:30.058240    4964 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e190985a-446c-409d-87bb-7926a945e43c-istio-envoy\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:30:30.058276 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:30:30.058255    4964 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e190985a-446c-409d-87bb-7926a945e43c-workload-socket\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:30:30.058276 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:30:30.058270    4964 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/38e3a193-4f84-478a-910e-bed4fac148e2-credential-socket\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:30:30.058276 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:30:30.058284    4964 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e190985a-446c-409d-87bb-7926a945e43c-istiod-ca-cert\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:30:30.058862 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:30:30.058298    4964 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/38e3a193-4f84-478a-910e-bed4fac148e2-istio-data\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:30:30.058862 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:30:30.058311    4964 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/479904d8-8280-451e-9182-e7d15c28c13d-istio-token\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:30:30.058862 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:30:30.058375    4964 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e190985a-446c-409d-87bb-7926a945e43c-istio-podinfo\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:30:30.058862 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:30:30.058389    4964 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/479904d8-8280-451e-9182-e7d15c28c13d-credential-socket\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:30:30.058862 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:30:30.058402    4964 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/479904d8-8280-451e-9182-e7d15c28c13d-istiod-ca-cert\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:30:30.058862 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:30:30.058417    4964 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/38e3a193-4f84-478a-910e-bed4fac148e2-istio-token\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:30:30.058862 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:30:30.058432    4964 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/38e3a193-4f84-478a-910e-bed4fac148e2-istio-envoy\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:30:30.058862 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:30:30.058467    4964 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/38e3a193-4f84-478a-910e-bed4fac148e2-workload-socket\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:30:30.058862 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:30:30.058484    4964 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ldhx6\" (UniqueName: \"kubernetes.io/projected/e190985a-446c-409d-87bb-7926a945e43c-kube-api-access-ldhx6\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:30:30.058862 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:30:30.058499    4964 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/479904d8-8280-451e-9182-e7d15c28c13d-workload-certs\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:30:30.058862 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:30:30.058512    4964 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e190985a-446c-409d-87bb-7926a945e43c-istio-token\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:30:30.058862 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:30:30.058526    4964 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ln4sn\" (UniqueName: \"kubernetes.io/projected/38e3a193-4f84-478a-910e-bed4fac148e2-kube-api-access-ln4sn\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:30:30.058862 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:30:30.058539    4964 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/38e3a193-4f84-478a-910e-bed4fac148e2-istio-podinfo\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:30:30.058862 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:30:30.058553    4964 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e190985a-446c-409d-87bb-7926a945e43c-istio-data\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:30:30.058862 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:30:30.058570    4964 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/479904d8-8280-451e-9182-e7d15c28c13d-workload-socket\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:30:30.058862 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:30:30.058586    4964 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/479904d8-8280-451e-9182-e7d15c28c13d-istio-podinfo\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:30:30.417565 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:30:30.417529    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-14ca09-54b56b9987-h9ws9"]
Oct 31 01:30:30.418453 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:30:30.418423    4964 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod21025738_82f8_45ef_a53f_f7761d96ca2f.slice/crio-1a04f380ec23d3bf336b0603ceeacfa61dec31bb420b7e417f32aa111b3d1ec1 WatchSource:0}: Error finding container 1a04f380ec23d3bf336b0603ceeacfa61dec31bb420b7e417f32aa111b3d1ec1: Status 404 returned error can't find the container with id 1a04f380ec23d3bf336b0603ceeacfa61dec31bb420b7e417f32aa111b3d1ec1
Oct 31 01:30:30.471295 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:30:30.471255    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-14ca09-54b56b9987-h9ws9" event={"ID":"21025738-82f8-45ef-a53f-f7761d96ca2f","Type":"ContainerStarted","Data":"1a04f380ec23d3bf336b0603ceeacfa61dec31bb420b7e417f32aa111b3d1ec1"}
Oct 31 01:30:30.472788 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:30:30.472770    4964 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-jhg75"
Oct 31 01:30:30.472788 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:30:30.472778    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-jhg75" event={"ID":"e190985a-446c-409d-87bb-7926a945e43c","Type":"ContainerDied","Data":"07b42df801c0b2ac6b7af98d3f2a78cdea57542ce466ad9ae025fa72af34b375"}
Oct 31 01:30:30.472974 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:30:30.472837    4964 scope.go:122] "RemoveContainer" containerID="e85cb26c6dce5fd1a4789974b9eccfbd2810499bb7f59de8a345a59164f6e2f2"
Oct 31 01:30:30.474487 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:30:30.474458    4964 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-hppbk"
Oct 31 01:30:30.474594 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:30:30.474495    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-hppbk" event={"ID":"479904d8-8280-451e-9182-e7d15c28c13d","Type":"ContainerDied","Data":"8ee1886b623fa4f2cd4cb3a002f01aab919ba08ee32d49a545ef9bc4bdd9018a"}
Oct 31 01:30:30.476391 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:30:30.476366    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-jz4tz" event={"ID":"a2038dbc-765f-488b-9701-c4081448ac4a","Type":"ContainerDied","Data":"aa800ca2997518d7b698bafe7c650562eddc3423f27183cfadae060415313546"}
Oct 31 01:30:30.476795 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:30:30.476442    4964 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-jz4tz"
Oct 31 01:30:30.478132 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:30:30.478102    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-ls7ft" event={"ID":"38e3a193-4f84-478a-910e-bed4fac148e2","Type":"ContainerDied","Data":"a84d06b5dda177894ba2429e7f9162c8b1d58c4bdc5e603bfa06284673f176eb"}
Oct 31 01:30:30.478245 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:30:30.478105    4964 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-ls7ft"
Oct 31 01:30:30.487622 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:30:30.487603    4964 scope.go:122] "RemoveContainer" containerID="d3a14bedd81cb4cee73bdcd48999d61050eb6b4d9883e179b34b6c3cd0377ac8"
Oct 31 01:30:30.505927 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:30:30.505905    4964 scope.go:122] "RemoveContainer" containerID="43a463f81497888dd942f1d16df68e152238a91b7f29fabdf658061d6481f1bc"
Oct 31 01:30:30.517149 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:30:30.517122    4964 scope.go:122] "RemoveContainer" containerID="b74fe5fc9f28cac3591d90f5d7113bf442f66534f83451f94e39505a09b5db56"
Oct 31 01:30:30.637627 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:30:30.637534    4964 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-ls7ft"]
Oct 31 01:30:30.648618 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:30:30.648568    4964 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-ls7ft"]
Oct 31 01:30:30.753409 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:30:30.753380    4964 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-jhg75"]
Oct 31 01:30:30.756899 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:30:30.756873    4964 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-jhg75"]
Oct 31 01:30:30.863114 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:30:30.863068    4964 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-jz4tz"]
Oct 31 01:30:30.867280 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:30:30.867252    4964 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-jz4tz"]
Oct 31 01:30:30.974581 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:30:30.974546    4964 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-hppbk"]
Oct 31 01:30:30.978354 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:30:30.978305    4964 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-hppbk"]
Oct 31 01:30:31.487747 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:30:31.487716    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-14ca09-54b56b9987-h9ws9" event={"ID":"21025738-82f8-45ef-a53f-f7761d96ca2f","Type":"ContainerStarted","Data":"f449f0541bca60268967a885fb2a45fa7ae04797f042e45c7ba6607f66bbd664"}
Oct 31 01:30:31.488169 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:30:31.487788    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-14ca09-54b56b9987-h9ws9"
Oct 31 01:30:31.605786 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:30:31.605664    4964 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-14ca09-54b56b9987-h9ws9" podStartSLOduration=1.6983332450000002 podStartE2EDuration="2.605645334s" totalImagesPullingTime="907.312089ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:30:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:30:31.602050618 +0000 UTC m=+1600.078617000" watchObservedRunningTime="2026-10-31 01:30:31.605645334 +0000 UTC m=+1600.082211715"
Oct 31 01:30:31.782734 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:30:31.782701    4964 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="38e3a193-4f84-478a-910e-bed4fac148e2" path="/var/lib/kubelet/pods/38e3a193-4f84-478a-910e-bed4fac148e2/volumes"
Oct 31 01:30:31.783399 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:30:31.783379    4964 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="479904d8-8280-451e-9182-e7d15c28c13d" path="/var/lib/kubelet/pods/479904d8-8280-451e-9182-e7d15c28c13d/volumes"
Oct 31 01:30:31.783866 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:30:31.783852    4964 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a2038dbc-765f-488b-9701-c4081448ac4a" path="/var/lib/kubelet/pods/a2038dbc-765f-488b-9701-c4081448ac4a/volumes"
Oct 31 01:30:31.784636 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:30:31.784615    4964 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e190985a-446c-409d-87bb-7926a945e43c" path="/var/lib/kubelet/pods/e190985a-446c-409d-87bb-7926a945e43c/volumes"
Oct 31 01:30:34.284548 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:30:34.284512    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:30:36.916636 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:30:36.916597    4964 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-qvt9t"]
Oct 31 01:30:36.917429 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:30:36.916821    4964 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-qvt9t" podUID="247208c2-428e-4f58-8dff-55939802b345" containerName="istio-proxy" containerID="cri-o://257eafd62630265281c3a7219833e8dc6bb23f8dd14c0b1eaa914618b6196b32" gracePeriod=30
Oct 31 01:30:36.964398 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:30:36.964363    4964 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-46rnw"]
Oct 31 01:30:36.964657 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:30:36.964608    4964 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-46rnw" podUID="30c81ef3-5c85-411a-ac4e-524dbd5e3d3d" containerName="istio-proxy" containerID="cri-o://187f6077ebc1061103fc3aedc2a74a0807d6f6780d5f7a473b50836b977d3c61" gracePeriod=30
Oct 31 01:30:37.522221 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:30:37.522192    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-14ca09-54b56b9987-h9ws9"
Oct 31 01:30:39.111344 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:30:39.102886    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:30:41.252017 ip-10-0-133-197 kubenswrapper[4964]: E1031 01:30:41.251979    4964 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q56x5" containerName="extractor"
Oct 31 01:30:42.138375 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:30:42.135689    4964 patch_prober.go:29] interesting pod/e2e-worker-w1-baacee-openshift-default-57f845d5bf-qvt9t container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body=
Oct 31 01:30:42.138375 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:30:42.135763    4964 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-qvt9t" podUID="247208c2-428e-4f58-8dff-55939802b345" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused"
Oct 31 01:30:42.285113 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:30:42.285078    4964 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-qvt9t"
Oct 31 01:30:42.338172 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:30:42.338144    4964 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-46rnw"
Oct 31 01:30:42.376036 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:30:42.376006    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/30c81ef3-5c85-411a-ac4e-524dbd5e3d3d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/30c81ef3-5c85-411a-ac4e-524dbd5e3d3d-istio-podinfo\") pod \"30c81ef3-5c85-411a-ac4e-524dbd5e3d3d\" (UID: \"30c81ef3-5c85-411a-ac4e-524dbd5e3d3d\") "
Oct 31 01:30:42.376208 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:30:42.376089    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/30c81ef3-5c85-411a-ac4e-524dbd5e3d3d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/30c81ef3-5c85-411a-ac4e-524dbd5e3d3d-istio-envoy\") pod \"30c81ef3-5c85-411a-ac4e-524dbd5e3d3d\" (UID: \"30c81ef3-5c85-411a-ac4e-524dbd5e3d3d\") "
Oct 31 01:30:42.376208 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:30:42.376141    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/247208c2-428e-4f58-8dff-55939802b345-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/247208c2-428e-4f58-8dff-55939802b345-istio-data\") pod \"247208c2-428e-4f58-8dff-55939802b345\" (UID: \"247208c2-428e-4f58-8dff-55939802b345\") "
Oct 31 01:30:42.376208 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:30:42.376165    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/30c81ef3-5c85-411a-ac4e-524dbd5e3d3d-istio-token\" (UniqueName: \"kubernetes.io/projected/30c81ef3-5c85-411a-ac4e-524dbd5e3d3d-istio-token\") pod \"30c81ef3-5c85-411a-ac4e-524dbd5e3d3d\" (UID: \"30c81ef3-5c85-411a-ac4e-524dbd5e3d3d\") "
Oct 31 01:30:42.376344 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:30:42.376208    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/247208c2-428e-4f58-8dff-55939802b345-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/247208c2-428e-4f58-8dff-55939802b345-istio-envoy\") pod \"247208c2-428e-4f58-8dff-55939802b345\" (UID: \"247208c2-428e-4f58-8dff-55939802b345\") "
Oct 31 01:30:42.376344 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:30:42.376229    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/30c81ef3-5c85-411a-ac4e-524dbd5e3d3d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/30c81ef3-5c85-411a-ac4e-524dbd5e3d3d-istiod-ca-cert\") pod \"30c81ef3-5c85-411a-ac4e-524dbd5e3d3d\" (UID: \"30c81ef3-5c85-411a-ac4e-524dbd5e3d3d\") "
Oct 31 01:30:42.376344 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:30:42.376251    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/247208c2-428e-4f58-8dff-55939802b345-istio-token\" (UniqueName: \"kubernetes.io/projected/247208c2-428e-4f58-8dff-55939802b345-istio-token\") pod \"247208c2-428e-4f58-8dff-55939802b345\" (UID: \"247208c2-428e-4f58-8dff-55939802b345\") "
Oct 31 01:30:42.376344 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:30:42.376297    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/247208c2-428e-4f58-8dff-55939802b345-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/247208c2-428e-4f58-8dff-55939802b345-workload-certs\") pod \"247208c2-428e-4f58-8dff-55939802b345\" (UID: \"247208c2-428e-4f58-8dff-55939802b345\") "
Oct 31 01:30:42.376510 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:30:42.376405    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/247208c2-428e-4f58-8dff-55939802b345-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/247208c2-428e-4f58-8dff-55939802b345-istio-podinfo\") pod \"247208c2-428e-4f58-8dff-55939802b345\" (UID: \"247208c2-428e-4f58-8dff-55939802b345\") "
Oct 31 01:30:42.376510 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:30:42.376448    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/247208c2-428e-4f58-8dff-55939802b345-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/247208c2-428e-4f58-8dff-55939802b345-istiod-ca-cert\") pod \"247208c2-428e-4f58-8dff-55939802b345\" (UID: \"247208c2-428e-4f58-8dff-55939802b345\") "
Oct 31 01:30:42.376510 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:30:42.376444    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/247208c2-428e-4f58-8dff-55939802b345-istio-data" pod "247208c2-428e-4f58-8dff-55939802b345" (UID: "247208c2-428e-4f58-8dff-55939802b345"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:30:42.376658 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:30:42.376535    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/30c81ef3-5c85-411a-ac4e-524dbd5e3d3d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/30c81ef3-5c85-411a-ac4e-524dbd5e3d3d-workload-certs\") pod \"30c81ef3-5c85-411a-ac4e-524dbd5e3d3d\" (UID: \"30c81ef3-5c85-411a-ac4e-524dbd5e3d3d\") "
Oct 31 01:30:42.376658 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:30:42.376592    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/247208c2-428e-4f58-8dff-55939802b345-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/247208c2-428e-4f58-8dff-55939802b345-workload-socket\") pod \"247208c2-428e-4f58-8dff-55939802b345\" (UID: \"247208c2-428e-4f58-8dff-55939802b345\") "
Oct 31 01:30:42.376658 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:30:42.376634    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/247208c2-428e-4f58-8dff-55939802b345-kube-api-access-pbpmh\" (UniqueName: \"kubernetes.io/projected/247208c2-428e-4f58-8dff-55939802b345-kube-api-access-pbpmh\") pod \"247208c2-428e-4f58-8dff-55939802b345\" (UID: \"247208c2-428e-4f58-8dff-55939802b345\") "
Oct 31 01:30:42.376814 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:30:42.376719    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/30c81ef3-5c85-411a-ac4e-524dbd5e3d3d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/30c81ef3-5c85-411a-ac4e-524dbd5e3d3d-credential-socket\") pod \"30c81ef3-5c85-411a-ac4e-524dbd5e3d3d\" (UID: \"30c81ef3-5c85-411a-ac4e-524dbd5e3d3d\") "
Oct 31 01:30:42.376814 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:30:42.376751    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/247208c2-428e-4f58-8dff-55939802b345-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/247208c2-428e-4f58-8dff-55939802b345-credential-socket\") pod \"247208c2-428e-4f58-8dff-55939802b345\" (UID: \"247208c2-428e-4f58-8dff-55939802b345\") "
Oct 31 01:30:42.376814 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:30:42.376782    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/30c81ef3-5c85-411a-ac4e-524dbd5e3d3d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/30c81ef3-5c85-411a-ac4e-524dbd5e3d3d-workload-socket\") pod \"30c81ef3-5c85-411a-ac4e-524dbd5e3d3d\" (UID: \"30c81ef3-5c85-411a-ac4e-524dbd5e3d3d\") "
Oct 31 01:30:42.376968 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:30:42.376816    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/247208c2-428e-4f58-8dff-55939802b345-istiod-ca-cert" pod "247208c2-428e-4f58-8dff-55939802b345" (UID: "247208c2-428e-4f58-8dff-55939802b345"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:30:42.376968 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:30:42.376825    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/30c81ef3-5c85-411a-ac4e-524dbd5e3d3d-istiod-ca-cert" pod "30c81ef3-5c85-411a-ac4e-524dbd5e3d3d" (UID: "30c81ef3-5c85-411a-ac4e-524dbd5e3d3d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:30:42.377066 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:30:42.377017    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/30c81ef3-5c85-411a-ac4e-524dbd5e3d3d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/30c81ef3-5c85-411a-ac4e-524dbd5e3d3d-istio-data\") pod \"30c81ef3-5c85-411a-ac4e-524dbd5e3d3d\" (UID: \"30c81ef3-5c85-411a-ac4e-524dbd5e3d3d\") "
Oct 31 01:30:42.377066 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:30:42.377050    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/30c81ef3-5c85-411a-ac4e-524dbd5e3d3d-kube-api-access-s7xvt\" (UniqueName: \"kubernetes.io/projected/30c81ef3-5c85-411a-ac4e-524dbd5e3d3d-kube-api-access-s7xvt\") pod \"30c81ef3-5c85-411a-ac4e-524dbd5e3d3d\" (UID: \"30c81ef3-5c85-411a-ac4e-524dbd5e3d3d\") "
Oct 31 01:30:42.377163 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:30:42.377098    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/247208c2-428e-4f58-8dff-55939802b345-workload-certs" pod "247208c2-428e-4f58-8dff-55939802b345" (UID: "247208c2-428e-4f58-8dff-55939802b345"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:30:42.377344 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:30:42.377303    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30c81ef3-5c85-411a-ac4e-524dbd5e3d3d-credential-socket" pod "30c81ef3-5c85-411a-ac4e-524dbd5e3d3d" (UID: "30c81ef3-5c85-411a-ac4e-524dbd5e3d3d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:30:42.378039 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:30:42.378012    4964 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/247208c2-428e-4f58-8dff-55939802b345-istio-data\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:30:42.378135 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:30:42.378045    4964 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/30c81ef3-5c85-411a-ac4e-524dbd5e3d3d-istiod-ca-cert\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:30:42.378135 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:30:42.378061    4964 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/247208c2-428e-4f58-8dff-55939802b345-workload-certs\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:30:42.378135 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:30:42.378075    4964 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/247208c2-428e-4f58-8dff-55939802b345-istiod-ca-cert\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:30:42.378135 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:30:42.378089    4964 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/30c81ef3-5c85-411a-ac4e-524dbd5e3d3d-credential-socket\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:30:42.378463 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:30:42.378331    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30c81ef3-5c85-411a-ac4e-524dbd5e3d3d-workload-certs" pod "30c81ef3-5c85-411a-ac4e-524dbd5e3d3d" (UID: "30c81ef3-5c85-411a-ac4e-524dbd5e3d3d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:30:42.378528 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:30:42.378514    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/247208c2-428e-4f58-8dff-55939802b345-workload-socket" pod "247208c2-428e-4f58-8dff-55939802b345" (UID: "247208c2-428e-4f58-8dff-55939802b345"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:30:42.379312 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:30:42.379100    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30c81ef3-5c85-411a-ac4e-524dbd5e3d3d-istio-envoy" pod "30c81ef3-5c85-411a-ac4e-524dbd5e3d3d" (UID: "30c81ef3-5c85-411a-ac4e-524dbd5e3d3d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:30:42.379312 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:30:42.379199    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/30c81ef3-5c85-411a-ac4e-524dbd5e3d3d-istio-podinfo" pod "30c81ef3-5c85-411a-ac4e-524dbd5e3d3d" (UID: "30c81ef3-5c85-411a-ac4e-524dbd5e3d3d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 01:30:42.379498 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:30:42.379472    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/247208c2-428e-4f58-8dff-55939802b345-credential-socket" pod "247208c2-428e-4f58-8dff-55939802b345" (UID: "247208c2-428e-4f58-8dff-55939802b345"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:30:42.379689 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:30:42.379625    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30c81ef3-5c85-411a-ac4e-524dbd5e3d3d-workload-socket" pod "30c81ef3-5c85-411a-ac4e-524dbd5e3d3d" (UID: "30c81ef3-5c85-411a-ac4e-524dbd5e3d3d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:30:42.379689 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:30:42.379638    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/247208c2-428e-4f58-8dff-55939802b345-istio-token" pod "247208c2-428e-4f58-8dff-55939802b345" (UID: "247208c2-428e-4f58-8dff-55939802b345"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:30:42.379689 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:30:42.379670    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30c81ef3-5c85-411a-ac4e-524dbd5e3d3d-istio-data" pod "30c81ef3-5c85-411a-ac4e-524dbd5e3d3d" (UID: "30c81ef3-5c85-411a-ac4e-524dbd5e3d3d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:30:42.379895 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:30:42.379719    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/247208c2-428e-4f58-8dff-55939802b345-istio-podinfo" pod "247208c2-428e-4f58-8dff-55939802b345" (UID: "247208c2-428e-4f58-8dff-55939802b345"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 01:30:42.379895 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:30:42.379855    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/30c81ef3-5c85-411a-ac4e-524dbd5e3d3d-kube-api-access-s7xvt" pod "30c81ef3-5c85-411a-ac4e-524dbd5e3d3d" (UID: "30c81ef3-5c85-411a-ac4e-524dbd5e3d3d"). InnerVolumeSpecName "kube-api-access-s7xvt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:30:42.380501 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:30:42.380478    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/247208c2-428e-4f58-8dff-55939802b345-istio-envoy" pod "247208c2-428e-4f58-8dff-55939802b345" (UID: "247208c2-428e-4f58-8dff-55939802b345"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:30:42.381105 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:30:42.381087    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/30c81ef3-5c85-411a-ac4e-524dbd5e3d3d-istio-token" pod "30c81ef3-5c85-411a-ac4e-524dbd5e3d3d" (UID: "30c81ef3-5c85-411a-ac4e-524dbd5e3d3d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:30:42.381613 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:30:42.381595    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/247208c2-428e-4f58-8dff-55939802b345-kube-api-access-pbpmh" pod "247208c2-428e-4f58-8dff-55939802b345" (UID: "247208c2-428e-4f58-8dff-55939802b345"). InnerVolumeSpecName "kube-api-access-pbpmh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:30:42.481580 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:30:42.480222    4964 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/30c81ef3-5c85-411a-ac4e-524dbd5e3d3d-istio-data\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:30:42.481580 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:30:42.480283    4964 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s7xvt\" (UniqueName: \"kubernetes.io/projected/30c81ef3-5c85-411a-ac4e-524dbd5e3d3d-kube-api-access-s7xvt\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:30:42.481580 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:30:42.480301    4964 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/30c81ef3-5c85-411a-ac4e-524dbd5e3d3d-istio-podinfo\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:30:42.481580 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:30:42.480334    4964 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/30c81ef3-5c85-411a-ac4e-524dbd5e3d3d-istio-envoy\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:30:42.481580 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:30:42.480349    4964 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/30c81ef3-5c85-411a-ac4e-524dbd5e3d3d-istio-token\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:30:42.481580 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:30:42.480372    4964 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/247208c2-428e-4f58-8dff-55939802b345-istio-envoy\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:30:42.481580 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:30:42.480386    4964 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/247208c2-428e-4f58-8dff-55939802b345-istio-token\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:30:42.481580 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:30:42.480401    4964 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/247208c2-428e-4f58-8dff-55939802b345-istio-podinfo\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:30:42.481580 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:30:42.480416    4964 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/30c81ef3-5c85-411a-ac4e-524dbd5e3d3d-workload-certs\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:30:42.481580 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:30:42.480436    4964 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/247208c2-428e-4f58-8dff-55939802b345-workload-socket\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:30:42.481580 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:30:42.480452    4964 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pbpmh\" (UniqueName: \"kubernetes.io/projected/247208c2-428e-4f58-8dff-55939802b345-kube-api-access-pbpmh\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:30:42.481580 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:30:42.480469    4964 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/247208c2-428e-4f58-8dff-55939802b345-credential-socket\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:30:42.481580 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:30:42.480488    4964 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/30c81ef3-5c85-411a-ac4e-524dbd5e3d3d-workload-socket\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:30:42.583513 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:30:42.583473    4964 generic.go:356] "Generic (PLEG): container finished" podID="247208c2-428e-4f58-8dff-55939802b345" containerID="257eafd62630265281c3a7219833e8dc6bb23f8dd14c0b1eaa914618b6196b32" exitCode=0
Oct 31 01:30:42.583715 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:30:42.583609    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-qvt9t" event={"ID":"247208c2-428e-4f58-8dff-55939802b345","Type":"ContainerDied","Data":"257eafd62630265281c3a7219833e8dc6bb23f8dd14c0b1eaa914618b6196b32"}
Oct 31 01:30:42.583715 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:30:42.583691    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-qvt9t" event={"ID":"247208c2-428e-4f58-8dff-55939802b345","Type":"ContainerDied","Data":"45722704ca95cf9b8e22389fae1ea4c2b330c7485ff7c24cea51129cbfb2454e"}
Oct 31 01:30:42.583845 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:30:42.583719    4964 scope.go:122] "RemoveContainer" containerID="257eafd62630265281c3a7219833e8dc6bb23f8dd14c0b1eaa914618b6196b32"
Oct 31 01:30:42.583845 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:30:42.583726    4964 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-qvt9t"
Oct 31 01:30:42.586795 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:30:42.586767    4964 generic.go:356] "Generic (PLEG): container finished" podID="30c81ef3-5c85-411a-ac4e-524dbd5e3d3d" containerID="187f6077ebc1061103fc3aedc2a74a0807d6f6780d5f7a473b50836b977d3c61" exitCode=0
Oct 31 01:30:42.587009 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:30:42.586985    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-46rnw" event={"ID":"30c81ef3-5c85-411a-ac4e-524dbd5e3d3d","Type":"ContainerDied","Data":"187f6077ebc1061103fc3aedc2a74a0807d6f6780d5f7a473b50836b977d3c61"}
Oct 31 01:30:42.587083 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:30:42.587028    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-46rnw" event={"ID":"30c81ef3-5c85-411a-ac4e-524dbd5e3d3d","Type":"ContainerDied","Data":"3a2522da97df75337b3c7a8587d855bcc128e1d9acb53ac725a04c141531694b"}
Oct 31 01:30:42.587215 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:30:42.587201    4964 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-46rnw"
Oct 31 01:30:42.601148 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:30:42.601124    4964 scope.go:122] "RemoveContainer" containerID="257eafd62630265281c3a7219833e8dc6bb23f8dd14c0b1eaa914618b6196b32"
Oct 31 01:30:42.601634 ip-10-0-133-197 kubenswrapper[4964]: E1031 01:30:42.601601    4964 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"257eafd62630265281c3a7219833e8dc6bb23f8dd14c0b1eaa914618b6196b32\": container with ID starting with 257eafd62630265281c3a7219833e8dc6bb23f8dd14c0b1eaa914618b6196b32 not found: ID does not exist" containerID="257eafd62630265281c3a7219833e8dc6bb23f8dd14c0b1eaa914618b6196b32"
Oct 31 01:30:42.601737 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:30:42.601642    4964 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"257eafd62630265281c3a7219833e8dc6bb23f8dd14c0b1eaa914618b6196b32"} err="failed to get container status \"257eafd62630265281c3a7219833e8dc6bb23f8dd14c0b1eaa914618b6196b32\": rpc error: code = NotFound desc = could not find container \"257eafd62630265281c3a7219833e8dc6bb23f8dd14c0b1eaa914618b6196b32\": container with ID starting with 257eafd62630265281c3a7219833e8dc6bb23f8dd14c0b1eaa914618b6196b32 not found: ID does not exist"
Oct 31 01:30:42.601737 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:30:42.601667    4964 scope.go:122] "RemoveContainer" containerID="187f6077ebc1061103fc3aedc2a74a0807d6f6780d5f7a473b50836b977d3c61"
Oct 31 01:30:42.629537 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:30:42.628953    4964 scope.go:122] "RemoveContainer" containerID="187f6077ebc1061103fc3aedc2a74a0807d6f6780d5f7a473b50836b977d3c61"
Oct 31 01:30:42.630300 ip-10-0-133-197 kubenswrapper[4964]: E1031 01:30:42.629681    4964 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"187f6077ebc1061103fc3aedc2a74a0807d6f6780d5f7a473b50836b977d3c61\": container with ID starting with 187f6077ebc1061103fc3aedc2a74a0807d6f6780d5f7a473b50836b977d3c61 not found: ID does not exist" containerID="187f6077ebc1061103fc3aedc2a74a0807d6f6780d5f7a473b50836b977d3c61"
Oct 31 01:30:42.630300 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:30:42.629720    4964 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"187f6077ebc1061103fc3aedc2a74a0807d6f6780d5f7a473b50836b977d3c61"} err="failed to get container status \"187f6077ebc1061103fc3aedc2a74a0807d6f6780d5f7a473b50836b977d3c61\": rpc error: code = NotFound desc = could not find container \"187f6077ebc1061103fc3aedc2a74a0807d6f6780d5f7a473b50836b977d3c61\": container with ID starting with 187f6077ebc1061103fc3aedc2a74a0807d6f6780d5f7a473b50836b977d3c61 not found: ID does not exist"
Oct 31 01:30:42.756047 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:30:42.755951    4964 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-qvt9t"]
Oct 31 01:30:42.760301 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:30:42.760274    4964 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-qvt9t"]
Oct 31 01:30:42.867660 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:30:42.867622    4964 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-46rnw"]
Oct 31 01:30:42.875498 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:30:42.875463    4964 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-46rnw"]
Oct 31 01:30:43.780069 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:30:43.780020    4964 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="247208c2-428e-4f58-8dff-55939802b345" path="/var/lib/kubelet/pods/247208c2-428e-4f58-8dff-55939802b345/volumes"
Oct 31 01:30:43.780801 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:30:43.780781    4964 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="30c81ef3-5c85-411a-ac4e-524dbd5e3d3d" path="/var/lib/kubelet/pods/30c81ef3-5c85-411a-ac4e-524dbd5e3d3d/volumes"
Oct 31 01:30:43.818184 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:30:43.818149    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:30:46.633942 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:30:46.633910    4964 generic.go:356] "Generic (PLEG): container finished" podID="eb13aca8-a24f-4fc8-a51f-3d3f45d739ec" containerID="1136ce451c0dddd2c9cbcfcb5f38288098a69c45635846edaf608507b1c4cff3" exitCode=0
Oct 31 01:30:46.634475 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:30:46.633986    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-e7eea9-openshift-default-76c54449f5-c6bvv" event={"ID":"eb13aca8-a24f-4fc8-a51f-3d3f45d739ec","Type":"ContainerDied","Data":"1136ce451c0dddd2c9cbcfcb5f38288098a69c45635846edaf608507b1c4cff3"}
Oct 31 01:30:46.634556 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:30:46.634511    4964 scope.go:122] "RemoveContainer" containerID="1136ce451c0dddd2c9cbcfcb5f38288098a69c45635846edaf608507b1c4cff3"
Oct 31 01:30:46.636238 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:30:46.636209    4964 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 01:30:46.636354 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:30:46.636276    4964 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 01:30:46.636354 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:30:46.636309    4964 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 01:30:46.730931 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:30:46.730896    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-e7eea9-openshift-default-76c54449f5-c6bvv"
Oct 31 01:30:47.645267 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:30:47.645229    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-e7eea9-openshift-default-76c54449f5-c6bvv" event={"ID":"eb13aca8-a24f-4fc8-a51f-3d3f45d739ec","Type":"ContainerStarted","Data":"6432c11433f0325c461454a8717d858534f8b25b0cf44ae9a07089dfb9c9877c"}
Oct 31 01:30:48.535210 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:30:48.535174    4964 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-e7eea9-openshift-default-76c54449f5-c6bvv"
Oct 31 01:30:48.536348 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:30:48.536302    4964 patch_prober.go:29] interesting pod/e2e-api-b-e7eea9-openshift-default-76c54449f5-c6bvv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" start-of-body=
Oct 31 01:30:48.536466 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:30:48.536386    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-e7eea9-openshift-default-76c54449f5-c6bvv" podUID="eb13aca8-a24f-4fc8-a51f-3d3f45d739ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused"
Oct 31 01:30:48.659410 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:30:48.659256    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:30:49.535980 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:30:49.535948    4964 patch_prober.go:29] interesting pod/e2e-api-b-e7eea9-openshift-default-76c54449f5-c6bvv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" start-of-body=
Oct 31 01:30:49.536155 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:30:49.536017    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-e7eea9-openshift-default-76c54449f5-c6bvv" podUID="eb13aca8-a24f-4fc8-a51f-3d3f45d739ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused"
Oct 31 01:30:50.536007 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:30:50.535972    4964 patch_prober.go:29] interesting pod/e2e-api-b-e7eea9-openshift-default-76c54449f5-c6bvv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" start-of-body=
Oct 31 01:30:50.536463 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:30:50.536026    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-e7eea9-openshift-default-76c54449f5-c6bvv" podUID="eb13aca8-a24f-4fc8-a51f-3d3f45d739ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused"
Oct 31 01:30:51.536058 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:30:51.536015    4964 patch_prober.go:29] interesting pod/e2e-api-b-e7eea9-openshift-default-76c54449f5-c6bvv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" start-of-body=
Oct 31 01:30:51.536473 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:30:51.536086    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-e7eea9-openshift-default-76c54449f5-c6bvv" podUID="eb13aca8-a24f-4fc8-a51f-3d3f45d739ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused"
Oct 31 01:30:52.535492 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:30:52.535392    4964 patch_prober.go:29] interesting pod/e2e-api-b-e7eea9-openshift-default-76c54449f5-c6bvv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" start-of-body=
Oct 31 01:30:52.535492 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:30:52.535471    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-e7eea9-openshift-default-76c54449f5-c6bvv" podUID="eb13aca8-a24f-4fc8-a51f-3d3f45d739ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused"
Oct 31 01:30:53.508931 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:30:53.508876    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:30:53.540393 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:30:53.540362    4964 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-e7eea9-openshift-default-76c54449f5-c6bvv"
Oct 31 01:30:53.540759 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:30:53.540731    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-e7eea9-openshift-default-76c54449f5-c6bvv"
Oct 31 01:30:53.541523 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:30:53.541504    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-e7eea9-openshift-default-76c54449f5-c6bvv"
Oct 31 01:30:57.362883 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:30:57.362848    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:30:57.581208 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:30:57.581170    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:30:58.323659 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:30:58.323621    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:30:58.710679 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:30:58.710591    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:30:58.900303 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:30:58.900266    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:30:59.599762 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:30:59.599726    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:31:00.215279 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:31:00.215247    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:31:00.436280 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:31:00.436242    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:31:03.013866 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:31:03.013826    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:31:03.217674 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:31:03.217638    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:31:03.318670 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:31:03.318589    4964 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-18bf762c-openshift-default-77b9fdd9b5-z7xxk"]
Oct 31 01:31:03.318840 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:31:03.318816    4964 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-18bf762c-openshift-default-77b9fdd9b5-z7xxk" podUID="7c25fc09-ee27-46c3-95fc-bfc6bacbc46f" containerName="istio-proxy" containerID="cri-o://c916abb8c68f7d0f205b477f82ba80edcb54699901ab78a106d6c0e02815c9fc" gracePeriod=30
Oct 31 01:31:03.432146 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:31:03.430942    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:31:03.745818 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:31:03.745784    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:31:05.488591 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:31:05.488556    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:31:05.684462 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:31:05.684427    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:31:08.658457 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:31:08.658423    4964 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-18bf762c-openshift-default-77b9fdd9b5-z7xxk"
Oct 31 01:31:08.713949 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:31:08.713919    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7c25fc09-ee27-46c3-95fc-bfc6bacbc46f-kube-api-access-dsdcq\" (UniqueName: \"kubernetes.io/projected/7c25fc09-ee27-46c3-95fc-bfc6bacbc46f-kube-api-access-dsdcq\") pod \"7c25fc09-ee27-46c3-95fc-bfc6bacbc46f\" (UID: \"7c25fc09-ee27-46c3-95fc-bfc6bacbc46f\") "
Oct 31 01:31:08.714120 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:31:08.713954    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7c25fc09-ee27-46c3-95fc-bfc6bacbc46f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7c25fc09-ee27-46c3-95fc-bfc6bacbc46f-istio-podinfo\") pod \"7c25fc09-ee27-46c3-95fc-bfc6bacbc46f\" (UID: \"7c25fc09-ee27-46c3-95fc-bfc6bacbc46f\") "
Oct 31 01:31:08.714120 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:31:08.714006    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7c25fc09-ee27-46c3-95fc-bfc6bacbc46f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7c25fc09-ee27-46c3-95fc-bfc6bacbc46f-istiod-ca-cert\") pod \"7c25fc09-ee27-46c3-95fc-bfc6bacbc46f\" (UID: \"7c25fc09-ee27-46c3-95fc-bfc6bacbc46f\") "
Oct 31 01:31:08.714120 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:31:08.714054    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7c25fc09-ee27-46c3-95fc-bfc6bacbc46f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c25fc09-ee27-46c3-95fc-bfc6bacbc46f-workload-certs\") pod \"7c25fc09-ee27-46c3-95fc-bfc6bacbc46f\" (UID: \"7c25fc09-ee27-46c3-95fc-bfc6bacbc46f\") "
Oct 31 01:31:08.714269 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:31:08.714145    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7c25fc09-ee27-46c3-95fc-bfc6bacbc46f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c25fc09-ee27-46c3-95fc-bfc6bacbc46f-workload-socket\") pod \"7c25fc09-ee27-46c3-95fc-bfc6bacbc46f\" (UID: \"7c25fc09-ee27-46c3-95fc-bfc6bacbc46f\") "
Oct 31 01:31:08.714269 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:31:08.714202    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7c25fc09-ee27-46c3-95fc-bfc6bacbc46f-istio-token\" (UniqueName: \"kubernetes.io/projected/7c25fc09-ee27-46c3-95fc-bfc6bacbc46f-istio-token\") pod \"7c25fc09-ee27-46c3-95fc-bfc6bacbc46f\" (UID: \"7c25fc09-ee27-46c3-95fc-bfc6bacbc46f\") "
Oct 31 01:31:08.714269 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:31:08.714254    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7c25fc09-ee27-46c3-95fc-bfc6bacbc46f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c25fc09-ee27-46c3-95fc-bfc6bacbc46f-credential-socket\") pod \"7c25fc09-ee27-46c3-95fc-bfc6bacbc46f\" (UID: \"7c25fc09-ee27-46c3-95fc-bfc6bacbc46f\") "
Oct 31 01:31:08.714508 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:31:08.714405    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7c25fc09-ee27-46c3-95fc-bfc6bacbc46f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7c25fc09-ee27-46c3-95fc-bfc6bacbc46f-istio-data\") pod \"7c25fc09-ee27-46c3-95fc-bfc6bacbc46f\" (UID: \"7c25fc09-ee27-46c3-95fc-bfc6bacbc46f\") "
Oct 31 01:31:08.714578 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:31:08.714515    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7c25fc09-ee27-46c3-95fc-bfc6bacbc46f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7c25fc09-ee27-46c3-95fc-bfc6bacbc46f-istio-envoy\") pod \"7c25fc09-ee27-46c3-95fc-bfc6bacbc46f\" (UID: \"7c25fc09-ee27-46c3-95fc-bfc6bacbc46f\") "
Oct 31 01:31:08.714578 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:31:08.714518    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c25fc09-ee27-46c3-95fc-bfc6bacbc46f-workload-certs" pod "7c25fc09-ee27-46c3-95fc-bfc6bacbc46f" (UID: "7c25fc09-ee27-46c3-95fc-bfc6bacbc46f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:31:08.714738 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:31:08.714715    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c25fc09-ee27-46c3-95fc-bfc6bacbc46f-credential-socket" pod "7c25fc09-ee27-46c3-95fc-bfc6bacbc46f" (UID: "7c25fc09-ee27-46c3-95fc-bfc6bacbc46f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:31:08.714925 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:31:08.714521    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7c25fc09-ee27-46c3-95fc-bfc6bacbc46f-istiod-ca-cert" pod "7c25fc09-ee27-46c3-95fc-bfc6bacbc46f" (UID: "7c25fc09-ee27-46c3-95fc-bfc6bacbc46f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:31:08.715016 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:31:08.714921    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c25fc09-ee27-46c3-95fc-bfc6bacbc46f-istio-data" pod "7c25fc09-ee27-46c3-95fc-bfc6bacbc46f" (UID: "7c25fc09-ee27-46c3-95fc-bfc6bacbc46f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:31:08.715185 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:31:08.715101    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c25fc09-ee27-46c3-95fc-bfc6bacbc46f-workload-socket" pod "7c25fc09-ee27-46c3-95fc-bfc6bacbc46f" (UID: "7c25fc09-ee27-46c3-95fc-bfc6bacbc46f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:31:08.715603 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:31:08.715580    4964 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7c25fc09-ee27-46c3-95fc-bfc6bacbc46f-istiod-ca-cert\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:31:08.715710 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:31:08.715609    4964 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c25fc09-ee27-46c3-95fc-bfc6bacbc46f-workload-certs\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:31:08.715710 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:31:08.715627    4964 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c25fc09-ee27-46c3-95fc-bfc6bacbc46f-workload-socket\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:31:08.715710 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:31:08.715643    4964 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c25fc09-ee27-46c3-95fc-bfc6bacbc46f-credential-socket\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:31:08.715710 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:31:08.715658    4964 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7c25fc09-ee27-46c3-95fc-bfc6bacbc46f-istio-data\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:31:08.716673 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:31:08.716646    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c25fc09-ee27-46c3-95fc-bfc6bacbc46f-kube-api-access-dsdcq" pod "7c25fc09-ee27-46c3-95fc-bfc6bacbc46f" (UID: "7c25fc09-ee27-46c3-95fc-bfc6bacbc46f"). InnerVolumeSpecName "kube-api-access-dsdcq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:31:08.716842 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:31:08.716817    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7c25fc09-ee27-46c3-95fc-bfc6bacbc46f-istio-podinfo" pod "7c25fc09-ee27-46c3-95fc-bfc6bacbc46f" (UID: "7c25fc09-ee27-46c3-95fc-bfc6bacbc46f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 01:31:08.716988 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:31:08.716971    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c25fc09-ee27-46c3-95fc-bfc6bacbc46f-istio-token" pod "7c25fc09-ee27-46c3-95fc-bfc6bacbc46f" (UID: "7c25fc09-ee27-46c3-95fc-bfc6bacbc46f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:31:08.717072 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:31:08.717033    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c25fc09-ee27-46c3-95fc-bfc6bacbc46f-istio-envoy" pod "7c25fc09-ee27-46c3-95fc-bfc6bacbc46f" (UID: "7c25fc09-ee27-46c3-95fc-bfc6bacbc46f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:31:08.732096 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:31:08.732064    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:31:08.817242 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:31:08.817204    4964 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7c25fc09-ee27-46c3-95fc-bfc6bacbc46f-istio-token\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:31:08.817242 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:31:08.817244    4964 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7c25fc09-ee27-46c3-95fc-bfc6bacbc46f-istio-envoy\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:31:08.817533 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:31:08.817261    4964 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dsdcq\" (UniqueName: \"kubernetes.io/projected/7c25fc09-ee27-46c3-95fc-bfc6bacbc46f-kube-api-access-dsdcq\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:31:08.817533 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:31:08.817278    4964 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7c25fc09-ee27-46c3-95fc-bfc6bacbc46f-istio-podinfo\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:31:08.871592 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:31:08.871489    4964 generic.go:356] "Generic (PLEG): container finished" podID="7c25fc09-ee27-46c3-95fc-bfc6bacbc46f" containerID="c916abb8c68f7d0f205b477f82ba80edcb54699901ab78a106d6c0e02815c9fc" exitCode=0
Oct 31 01:31:08.871592 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:31:08.871562    4964 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-18bf762c-openshift-default-77b9fdd9b5-z7xxk"
Oct 31 01:31:08.871592 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:31:08.871564    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-18bf762c-openshift-default-77b9fdd9b5-z7xxk" event={"ID":"7c25fc09-ee27-46c3-95fc-bfc6bacbc46f","Type":"ContainerDied","Data":"c916abb8c68f7d0f205b477f82ba80edcb54699901ab78a106d6c0e02815c9fc"}
Oct 31 01:31:08.871861 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:31:08.871640    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-18bf762c-openshift-default-77b9fdd9b5-z7xxk" event={"ID":"7c25fc09-ee27-46c3-95fc-bfc6bacbc46f","Type":"ContainerDied","Data":"67208aa252abb8d485b62d47a0847a996cc12bb19fa4fba6c730b39f68d88e00"}
Oct 31 01:31:08.871861 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:31:08.871671    4964 scope.go:122] "RemoveContainer" containerID="c916abb8c68f7d0f205b477f82ba80edcb54699901ab78a106d6c0e02815c9fc"
Oct 31 01:31:08.892033 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:31:08.891996    4964 scope.go:122] "RemoveContainer" containerID="c916abb8c68f7d0f205b477f82ba80edcb54699901ab78a106d6c0e02815c9fc"
Oct 31 01:31:08.892426 ip-10-0-133-197 kubenswrapper[4964]: E1031 01:31:08.892387    4964 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c916abb8c68f7d0f205b477f82ba80edcb54699901ab78a106d6c0e02815c9fc\": container with ID starting with c916abb8c68f7d0f205b477f82ba80edcb54699901ab78a106d6c0e02815c9fc not found: ID does not exist" containerID="c916abb8c68f7d0f205b477f82ba80edcb54699901ab78a106d6c0e02815c9fc"
Oct 31 01:31:08.892526 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:31:08.892436    4964 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c916abb8c68f7d0f205b477f82ba80edcb54699901ab78a106d6c0e02815c9fc"} err="failed to get container status \"c916abb8c68f7d0f205b477f82ba80edcb54699901ab78a106d6c0e02815c9fc\": rpc error: code = NotFound desc = could not find container \"c916abb8c68f7d0f205b477f82ba80edcb54699901ab78a106d6c0e02815c9fc\": container with ID starting with c916abb8c68f7d0f205b477f82ba80edcb54699901ab78a106d6c0e02815c9fc not found: ID does not exist"
Oct 31 01:31:08.915211 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:31:08.915148    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:31:09.042981 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:31:09.042948    4964 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-18bf762c-openshift-default-77b9fdd9b5-z7xxk"]
Oct 31 01:31:09.049244 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:31:09.049218    4964 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-18bf762c-openshift-default-77b9fdd9b5-z7xxk"]
Oct 31 01:31:09.136171 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:31:09.136084    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:31:09.261143 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:31:09.261112    4964 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-bc2046-579d7d879-mkdpx"]
Oct 31 01:31:09.262093 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:31:09.262064    4964 cpu_manager.go:414] "containerMap: removing container" podUID="479904d8-8280-451e-9182-e7d15c28c13d" containerName="istio-proxy"
Oct 31 01:31:09.262093 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:31:09.262090    4964 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="479904d8-8280-451e-9182-e7d15c28c13d" containerName="istio-proxy"
Oct 31 01:31:09.262289 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:31:09.262100    4964 cpu_manager.go:414] "containerMap: removing container" podUID="fb021400-cb4b-494d-8967-5d0a7b35a2a7" containerName="cleanup"
Oct 31 01:31:09.262289 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:31:09.262105    4964 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fb021400-cb4b-494d-8967-5d0a7b35a2a7" containerName="cleanup"
Oct 31 01:31:09.262289 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:31:09.262126    4964 cpu_manager.go:414] "containerMap: removing container" podUID="a2038dbc-765f-488b-9701-c4081448ac4a" containerName="istio-proxy"
Oct 31 01:31:09.262289 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:31:09.262131    4964 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a2038dbc-765f-488b-9701-c4081448ac4a" containerName="istio-proxy"
Oct 31 01:31:09.262289 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:31:09.262138    4964 cpu_manager.go:414] "containerMap: removing container" podUID="7c25fc09-ee27-46c3-95fc-bfc6bacbc46f" containerName="istio-proxy"
Oct 31 01:31:09.262289 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:31:09.262142    4964 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7c25fc09-ee27-46c3-95fc-bfc6bacbc46f" containerName="istio-proxy"
Oct 31 01:31:09.262289 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:31:09.262153    4964 cpu_manager.go:414] "containerMap: removing container" podUID="38e3a193-4f84-478a-910e-bed4fac148e2" containerName="istio-proxy"
Oct 31 01:31:09.262289 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:31:09.262157    4964 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="38e3a193-4f84-478a-910e-bed4fac148e2" containerName="istio-proxy"
Oct 31 01:31:09.262289 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:31:09.262170    4964 cpu_manager.go:414] "containerMap: removing container" podUID="e190985a-446c-409d-87bb-7926a945e43c" containerName="istio-proxy"
Oct 31 01:31:09.262289 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:31:09.262175    4964 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e190985a-446c-409d-87bb-7926a945e43c" containerName="istio-proxy"
Oct 31 01:31:09.262289 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:31:09.262194    4964 cpu_manager.go:414] "containerMap: removing container" podUID="247208c2-428e-4f58-8dff-55939802b345" containerName="istio-proxy"
Oct 31 01:31:09.262289 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:31:09.262198    4964 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="247208c2-428e-4f58-8dff-55939802b345" containerName="istio-proxy"
Oct 31 01:31:09.262289 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:31:09.262211    4964 cpu_manager.go:414] "containerMap: removing container" podUID="30c81ef3-5c85-411a-ac4e-524dbd5e3d3d" containerName="istio-proxy"
Oct 31 01:31:09.262289 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:31:09.262215    4964 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="30c81ef3-5c85-411a-ac4e-524dbd5e3d3d" containerName="istio-proxy"
Oct 31 01:31:09.262745 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:31:09.262308    4964 memory_manager.go:365] "RemoveStaleState removing state" podUID="30c81ef3-5c85-411a-ac4e-524dbd5e3d3d" containerName="istio-proxy"
Oct 31 01:31:09.262745 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:31:09.262334    4964 memory_manager.go:365] "RemoveStaleState removing state" podUID="fb021400-cb4b-494d-8967-5d0a7b35a2a7" containerName="cleanup"
Oct 31 01:31:09.262745 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:31:09.262342    4964 memory_manager.go:365] "RemoveStaleState removing state" podUID="479904d8-8280-451e-9182-e7d15c28c13d" containerName="istio-proxy"
Oct 31 01:31:09.262745 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:31:09.262348    4964 memory_manager.go:365] "RemoveStaleState removing state" podUID="a2038dbc-765f-488b-9701-c4081448ac4a" containerName="istio-proxy"
Oct 31 01:31:09.262745 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:31:09.262353    4964 memory_manager.go:365] "RemoveStaleState removing state" podUID="7c25fc09-ee27-46c3-95fc-bfc6bacbc46f" containerName="istio-proxy"
Oct 31 01:31:09.262745 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:31:09.262360    4964 memory_manager.go:365] "RemoveStaleState removing state" podUID="38e3a193-4f84-478a-910e-bed4fac148e2" containerName="istio-proxy"
Oct 31 01:31:09.262745 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:31:09.262367    4964 memory_manager.go:365] "RemoveStaleState removing state" podUID="e190985a-446c-409d-87bb-7926a945e43c" containerName="istio-proxy"
Oct 31 01:31:09.262745 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:31:09.262375    4964 memory_manager.go:365] "RemoveStaleState removing state" podUID="247208c2-428e-4f58-8dff-55939802b345" containerName="istio-proxy"
Oct 31 01:31:09.267069 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:31:09.267046    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-bc2046-579d7d879-mkdpx"
Oct 31 01:31:09.318922 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:31:09.318880    4964 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-bc2046\""
Oct 31 01:31:09.319148 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:31:09.319071    4964 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-bc2046-dockercfg-qk4d6\""
Oct 31 01:31:09.323611 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:31:09.323555    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-bc2046-579d7d879-mkdpx"]
Oct 31 01:31:09.328339 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:31:09.326093    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-df6qp\" (UniqueName: \"kubernetes.io/projected/3a504885-cccb-4330-a15e-2f121ad43896-kube-api-access-df6qp\") pod \"payload-processing-e2e-api-b-bc2046-579d7d879-mkdpx\" (UID: \"3a504885-cccb-4330-a15e-2f121ad43896\") " pod="openshift-ingress/payload-processing-e2e-api-b-bc2046-579d7d879-mkdpx"
Oct 31 01:31:09.328339 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:31:09.326396    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3a504885-cccb-4330-a15e-2f121ad43896-plugins-config-volume\") pod \"payload-processing-e2e-api-b-bc2046-579d7d879-mkdpx\" (UID: \"3a504885-cccb-4330-a15e-2f121ad43896\") " pod="openshift-ingress/payload-processing-e2e-api-b-bc2046-579d7d879-mkdpx"
Oct 31 01:31:09.427550 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:31:09.427459    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3a504885-cccb-4330-a15e-2f121ad43896-plugins-config-volume\") pod \"payload-processing-e2e-api-b-bc2046-579d7d879-mkdpx\" (UID: \"3a504885-cccb-4330-a15e-2f121ad43896\") " pod="openshift-ingress/payload-processing-e2e-api-b-bc2046-579d7d879-mkdpx"
Oct 31 01:31:09.427550 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:31:09.427527    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-df6qp\" (UniqueName: \"kubernetes.io/projected/3a504885-cccb-4330-a15e-2f121ad43896-kube-api-access-df6qp\") pod \"payload-processing-e2e-api-b-bc2046-579d7d879-mkdpx\" (UID: \"3a504885-cccb-4330-a15e-2f121ad43896\") " pod="openshift-ingress/payload-processing-e2e-api-b-bc2046-579d7d879-mkdpx"
Oct 31 01:31:09.428432 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:31:09.428407    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3a504885-cccb-4330-a15e-2f121ad43896-plugins-config-volume\") pod \"payload-processing-e2e-api-b-bc2046-579d7d879-mkdpx\" (UID: \"3a504885-cccb-4330-a15e-2f121ad43896\") " pod="openshift-ingress/payload-processing-e2e-api-b-bc2046-579d7d879-mkdpx"
Oct 31 01:31:09.516456 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:31:09.516396    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-df6qp\" (UniqueName: \"kubernetes.io/projected/3a504885-cccb-4330-a15e-2f121ad43896-kube-api-access-df6qp\") pod \"payload-processing-e2e-api-b-bc2046-579d7d879-mkdpx\" (UID: \"3a504885-cccb-4330-a15e-2f121ad43896\") " pod="openshift-ingress/payload-processing-e2e-api-b-bc2046-579d7d879-mkdpx"
Oct 31 01:31:09.587759 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:31:09.587720    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-bc2046-579d7d879-mkdpx"
Oct 31 01:31:09.782462 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:31:09.782424    4964 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7c25fc09-ee27-46c3-95fc-bfc6bacbc46f" path="/var/lib/kubelet/pods/7c25fc09-ee27-46c3-95fc-bfc6bacbc46f/volumes"
Oct 31 01:31:09.964185 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:31:09.964157    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:31:10.038040 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:31:10.038011    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-bc2046-579d7d879-mkdpx"]
Oct 31 01:31:10.038949 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:31:10.038923    4964 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a504885_cccb_4330_a15e_2f121ad43896.slice/crio-d0599c144e7e496f61eccd077feaa44ac9f019e0d78de9ecb93b82e228a3a67f WatchSource:0}: Error finding container d0599c144e7e496f61eccd077feaa44ac9f019e0d78de9ecb93b82e228a3a67f: Status 404 returned error can't find the container with id d0599c144e7e496f61eccd077feaa44ac9f019e0d78de9ecb93b82e228a3a67f
Oct 31 01:31:10.889923 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:31:10.889745    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-bc2046-579d7d879-mkdpx" event={"ID":"3a504885-cccb-4330-a15e-2f121ad43896","Type":"ContainerStarted","Data":"6cedfd16ebdd9d9788c36cedd84b6ae6d386ef24878991c62b3fccceed6377dd"}
Oct 31 01:31:10.889923 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:31:10.889801    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-bc2046-579d7d879-mkdpx" event={"ID":"3a504885-cccb-4330-a15e-2f121ad43896","Type":"ContainerStarted","Data":"d0599c144e7e496f61eccd077feaa44ac9f019e0d78de9ecb93b82e228a3a67f"}
Oct 31 01:31:10.889923 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:31:10.889845    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-bc2046-579d7d879-mkdpx"
Oct 31 01:31:10.891447 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:31:10.891419    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:31:11.005437 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:31:11.005383    4964 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-bc2046-579d7d879-mkdpx" podStartSLOduration=2.005368246 podStartE2EDuration="2.005368246s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:31:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:31:11.00438557 +0000 UTC m=+1639.480951951" watchObservedRunningTime="2026-10-31 01:31:11.005368246 +0000 UTC m=+1639.481934626"
Oct 31 01:31:11.241279 ip-10-0-133-197 kubenswrapper[4964]: E1031 01:31:11.241244    4964 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q56x5" containerName="extractor"
Oct 31 01:31:11.939296 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:31:11.939257    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:31:14.229279 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:31:14.229242    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:31:14.570604 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:31:14.570514    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:31:15.276713 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:31:15.276672    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:31:19.279986 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:31:19.279952    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:31:19.490975 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:31:19.490941    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:31:21.901556 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:31:21.901519    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-bc2046-579d7d879-mkdpx"
Oct 31 01:31:24.111733 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:31:24.111698    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:31:25.680864 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:31:25.680827    4964 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-7cec02-6d95f44b95-wgspn"]
Oct 31 01:31:25.839696 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:31:25.839660    4964 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-7cec02-64cb84f795-865w7"]
Oct 31 01:31:26.184901 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:31:26.184866    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:31:26.374602 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:31:26.374465    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:31:28.096227 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:31:28.096181    4964 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-7rdlf"]
Oct 31 01:31:28.096839 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:31:28.096472    4964 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-7rdlf" podUID="11f27521-38d0-47d3-8884-accc44d652cb" containerName="istio-proxy" containerID="cri-o://8db46917c1436ffe81adc65b48b30690d2948bb71f66b67021c430a74b6368e9" gracePeriod=30
Oct 31 01:31:28.207736 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:31:28.207688    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:31:29.003511 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:31:29.003467    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:31:29.523992 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:31:29.523956    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:31:29.702977 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:31:29.702928    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:31:29.962371 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:31:29.962246    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:31:30.681377 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:31:30.681312    4964 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-7cec02-6d95f44b95-wgspn" podUID="a26b5198-d0fe-403b-858f-0e6619c15c3b" containerName="payload-processing" containerID="cri-o://3af96a2b131aba33ff6fb070316294ff8effa58cff6e12c4335f18ac167a713a" gracePeriod=25
Oct 31 01:31:30.840778 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:31:30.840739    4964 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-7cec02-64cb84f795-865w7" podUID="b9a6e0ab-3025-4363-8ad7-4434733148b7" containerName="payload-pre-processing" containerID="cri-o://5e2a25f80a5a70f23e8bdbcf22ad6dcf857b166706697945deb90bf4f2a17ab6" gracePeriod=25
Oct 31 01:31:30.985675 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:31:30.985648    4964 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-7cec02-6d95f44b95-wgspn"
Oct 31 01:31:31.116259 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:31:31.116225    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a26b5198-d0fe-403b-858f-0e6619c15c3b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a26b5198-d0fe-403b-858f-0e6619c15c3b-plugins-config-volume\") pod \"a26b5198-d0fe-403b-858f-0e6619c15c3b\" (UID: \"a26b5198-d0fe-403b-858f-0e6619c15c3b\") "
Oct 31 01:31:31.116857 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:31:31.116503    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a26b5198-d0fe-403b-858f-0e6619c15c3b-kube-api-access-vd4rr\" (UniqueName: \"kubernetes.io/projected/a26b5198-d0fe-403b-858f-0e6619c15c3b-kube-api-access-vd4rr\") pod \"a26b5198-d0fe-403b-858f-0e6619c15c3b\" (UID: \"a26b5198-d0fe-403b-858f-0e6619c15c3b\") "
Oct 31 01:31:31.116857 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:31:31.116782    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a26b5198-d0fe-403b-858f-0e6619c15c3b-plugins-config-volume" pod "a26b5198-d0fe-403b-858f-0e6619c15c3b" (UID: "a26b5198-d0fe-403b-858f-0e6619c15c3b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:31:31.117156 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:31:31.117135    4964 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a26b5198-d0fe-403b-858f-0e6619c15c3b-plugins-config-volume\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:31:31.119576 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:31:31.119426    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a26b5198-d0fe-403b-858f-0e6619c15c3b-kube-api-access-vd4rr" pod "a26b5198-d0fe-403b-858f-0e6619c15c3b" (UID: "a26b5198-d0fe-403b-858f-0e6619c15c3b"). InnerVolumeSpecName "kube-api-access-vd4rr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:31:31.134365 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:31:31.134305    4964 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-7cec02-64cb84f795-865w7"
Oct 31 01:31:31.218975 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:31:31.218872    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b9a6e0ab-3025-4363-8ad7-4434733148b7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b9a6e0ab-3025-4363-8ad7-4434733148b7-plugins-config-volume\") pod \"b9a6e0ab-3025-4363-8ad7-4434733148b7\" (UID: \"b9a6e0ab-3025-4363-8ad7-4434733148b7\") "
Oct 31 01:31:31.218975 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:31:31.218970    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b9a6e0ab-3025-4363-8ad7-4434733148b7-kube-api-access-8mjd4\" (UniqueName: \"kubernetes.io/projected/b9a6e0ab-3025-4363-8ad7-4434733148b7-kube-api-access-8mjd4\") pod \"b9a6e0ab-3025-4363-8ad7-4434733148b7\" (UID: \"b9a6e0ab-3025-4363-8ad7-4434733148b7\") "
Oct 31 01:31:31.219752 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:31:31.219726    4964 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vd4rr\" (UniqueName: \"kubernetes.io/projected/a26b5198-d0fe-403b-858f-0e6619c15c3b-kube-api-access-vd4rr\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:31:31.221650 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:31:31.221601    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b9a6e0ab-3025-4363-8ad7-4434733148b7-plugins-config-volume" pod "b9a6e0ab-3025-4363-8ad7-4434733148b7" (UID: "b9a6e0ab-3025-4363-8ad7-4434733148b7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:31:31.222884 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:31:31.222855    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b9a6e0ab-3025-4363-8ad7-4434733148b7-kube-api-access-8mjd4" pod "b9a6e0ab-3025-4363-8ad7-4434733148b7" (UID: "b9a6e0ab-3025-4363-8ad7-4434733148b7"). InnerVolumeSpecName "kube-api-access-8mjd4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:31:31.228351 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:31:31.228191    4964 generic.go:356] "Generic (PLEG): container finished" podID="a26b5198-d0fe-403b-858f-0e6619c15c3b" containerID="3af96a2b131aba33ff6fb070316294ff8effa58cff6e12c4335f18ac167a713a" exitCode=0
Oct 31 01:31:31.228351 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:31:31.228247    4964 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-7cec02-6d95f44b95-wgspn"
Oct 31 01:31:31.228351 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:31:31.228254    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-7cec02-6d95f44b95-wgspn" event={"ID":"a26b5198-d0fe-403b-858f-0e6619c15c3b","Type":"ContainerDied","Data":"3af96a2b131aba33ff6fb070316294ff8effa58cff6e12c4335f18ac167a713a"}
Oct 31 01:31:31.228663 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:31:31.228360    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-7cec02-6d95f44b95-wgspn" event={"ID":"a26b5198-d0fe-403b-858f-0e6619c15c3b","Type":"ContainerDied","Data":"b54f447f16e6e7a0412cfb17317821f3695da7df9106e943b7c7289f1d0bc84a"}
Oct 31 01:31:31.228663 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:31:31.228393    4964 scope.go:122] "RemoveContainer" containerID="3af96a2b131aba33ff6fb070316294ff8effa58cff6e12c4335f18ac167a713a"
Oct 31 01:31:31.231733 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:31:31.231699    4964 generic.go:356] "Generic (PLEG): container finished" podID="b9a6e0ab-3025-4363-8ad7-4434733148b7" containerID="5e2a25f80a5a70f23e8bdbcf22ad6dcf857b166706697945deb90bf4f2a17ab6" exitCode=0
Oct 31 01:31:31.231965 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:31:31.231942    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-7cec02-64cb84f795-865w7" event={"ID":"b9a6e0ab-3025-4363-8ad7-4434733148b7","Type":"ContainerDied","Data":"5e2a25f80a5a70f23e8bdbcf22ad6dcf857b166706697945deb90bf4f2a17ab6"}
Oct 31 01:31:31.232039 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:31:31.231995    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-7cec02-64cb84f795-865w7" event={"ID":"b9a6e0ab-3025-4363-8ad7-4434733148b7","Type":"ContainerDied","Data":"32e7522530bc35d81fa846c35c91964406257cf3b63d8151b9a8b3dbe64eb20f"}
Oct 31 01:31:31.232096 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:31:31.232072    4964 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-7cec02-64cb84f795-865w7"
Oct 31 01:31:31.243877 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:31:31.243855    4964 scope.go:122] "RemoveContainer" containerID="3af96a2b131aba33ff6fb070316294ff8effa58cff6e12c4335f18ac167a713a"
Oct 31 01:31:31.244162 ip-10-0-133-197 kubenswrapper[4964]: E1031 01:31:31.244141    4964 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3af96a2b131aba33ff6fb070316294ff8effa58cff6e12c4335f18ac167a713a\": container with ID starting with 3af96a2b131aba33ff6fb070316294ff8effa58cff6e12c4335f18ac167a713a not found: ID does not exist" containerID="3af96a2b131aba33ff6fb070316294ff8effa58cff6e12c4335f18ac167a713a"
Oct 31 01:31:31.244219 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:31:31.244169    4964 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3af96a2b131aba33ff6fb070316294ff8effa58cff6e12c4335f18ac167a713a"} err="failed to get container status \"3af96a2b131aba33ff6fb070316294ff8effa58cff6e12c4335f18ac167a713a\": rpc error: code = NotFound desc = could not find container \"3af96a2b131aba33ff6fb070316294ff8effa58cff6e12c4335f18ac167a713a\": container with ID starting with 3af96a2b131aba33ff6fb070316294ff8effa58cff6e12c4335f18ac167a713a not found: ID does not exist"
Oct 31 01:31:31.244219 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:31:31.244186    4964 scope.go:122] "RemoveContainer" containerID="5e2a25f80a5a70f23e8bdbcf22ad6dcf857b166706697945deb90bf4f2a17ab6"
Oct 31 01:31:31.258600 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:31:31.257305    4964 scope.go:122] "RemoveContainer" containerID="5e2a25f80a5a70f23e8bdbcf22ad6dcf857b166706697945deb90bf4f2a17ab6"
Oct 31 01:31:31.258600 ip-10-0-133-197 kubenswrapper[4964]: E1031 01:31:31.257800    4964 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5e2a25f80a5a70f23e8bdbcf22ad6dcf857b166706697945deb90bf4f2a17ab6\": container with ID starting with 5e2a25f80a5a70f23e8bdbcf22ad6dcf857b166706697945deb90bf4f2a17ab6 not found: ID does not exist" containerID="5e2a25f80a5a70f23e8bdbcf22ad6dcf857b166706697945deb90bf4f2a17ab6"
Oct 31 01:31:31.258600 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:31:31.257837    4964 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5e2a25f80a5a70f23e8bdbcf22ad6dcf857b166706697945deb90bf4f2a17ab6"} err="failed to get container status \"5e2a25f80a5a70f23e8bdbcf22ad6dcf857b166706697945deb90bf4f2a17ab6\": rpc error: code = NotFound desc = could not find container \"5e2a25f80a5a70f23e8bdbcf22ad6dcf857b166706697945deb90bf4f2a17ab6\": container with ID starting with 5e2a25f80a5a70f23e8bdbcf22ad6dcf857b166706697945deb90bf4f2a17ab6 not found: ID does not exist"
Oct 31 01:31:31.320760 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:31:31.320724    4964 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b9a6e0ab-3025-4363-8ad7-4434733148b7-plugins-config-volume\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:31:31.320760 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:31:31.320754    4964 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8mjd4\" (UniqueName: \"kubernetes.io/projected/b9a6e0ab-3025-4363-8ad7-4434733148b7-kube-api-access-8mjd4\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:31:31.358939 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:31:31.358909    4964 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-7cec02-6d95f44b95-wgspn"]
Oct 31 01:31:31.362767 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:31:31.362738    4964 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-7cec02-6d95f44b95-wgspn"]
Oct 31 01:31:31.410940 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:31:31.410902    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:31:31.520947 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:31:31.520873    4964 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-7cec02-64cb84f795-865w7"]
Oct 31 01:31:31.524497 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:31:31.524477    4964 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-7cec02-64cb84f795-865w7"]
Oct 31 01:31:31.778358 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:31:31.778253    4964 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a26b5198-d0fe-403b-858f-0e6619c15c3b" path="/var/lib/kubelet/pods/a26b5198-d0fe-403b-858f-0e6619c15c3b/volumes"
Oct 31 01:31:31.778761 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:31:31.778741    4964 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b9a6e0ab-3025-4363-8ad7-4434733148b7" path="/var/lib/kubelet/pods/b9a6e0ab-3025-4363-8ad7-4434733148b7/volumes"
Oct 31 01:31:33.264502 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:31:33.264466    4964 generic.go:356] "Generic (PLEG): container finished" podID="11f27521-38d0-47d3-8884-accc44d652cb" containerID="8db46917c1436ffe81adc65b48b30690d2948bb71f66b67021c430a74b6368e9" exitCode=0
Oct 31 01:31:33.264967 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:31:33.264571    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-7rdlf" event={"ID":"11f27521-38d0-47d3-8884-accc44d652cb","Type":"ContainerDied","Data":"8db46917c1436ffe81adc65b48b30690d2948bb71f66b67021c430a74b6368e9"}
Oct 31 01:31:33.372992 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:31:33.372930    4964 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-7rdlf"
Oct 31 01:31:33.453686 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:31:33.453655    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/11f27521-38d0-47d3-8884-accc44d652cb-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/11f27521-38d0-47d3-8884-accc44d652cb-workload-socket\") pod \"11f27521-38d0-47d3-8884-accc44d652cb\" (UID: \"11f27521-38d0-47d3-8884-accc44d652cb\") "
Oct 31 01:31:33.453827 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:31:33.453719    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/11f27521-38d0-47d3-8884-accc44d652cb-kube-api-access-8fh6j\" (UniqueName: \"kubernetes.io/projected/11f27521-38d0-47d3-8884-accc44d652cb-kube-api-access-8fh6j\") pod \"11f27521-38d0-47d3-8884-accc44d652cb\" (UID: \"11f27521-38d0-47d3-8884-accc44d652cb\") "
Oct 31 01:31:33.453909 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:31:33.453835    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/11f27521-38d0-47d3-8884-accc44d652cb-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/11f27521-38d0-47d3-8884-accc44d652cb-istio-envoy\") pod \"11f27521-38d0-47d3-8884-accc44d652cb\" (UID: \"11f27521-38d0-47d3-8884-accc44d652cb\") "
Oct 31 01:31:33.453909 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:31:33.453883    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/11f27521-38d0-47d3-8884-accc44d652cb-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/11f27521-38d0-47d3-8884-accc44d652cb-credential-socket\") pod \"11f27521-38d0-47d3-8884-accc44d652cb\" (UID: \"11f27521-38d0-47d3-8884-accc44d652cb\") "
Oct 31 01:31:33.453909 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:31:33.453892    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11f27521-38d0-47d3-8884-accc44d652cb-workload-socket" pod "11f27521-38d0-47d3-8884-accc44d652cb" (UID: "11f27521-38d0-47d3-8884-accc44d652cb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:31:33.454058 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:31:33.453948    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/11f27521-38d0-47d3-8884-accc44d652cb-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/11f27521-38d0-47d3-8884-accc44d652cb-istio-data\") pod \"11f27521-38d0-47d3-8884-accc44d652cb\" (UID: \"11f27521-38d0-47d3-8884-accc44d652cb\") "
Oct 31 01:31:33.454058 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:31:33.454015    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/11f27521-38d0-47d3-8884-accc44d652cb-istio-token\" (UniqueName: \"kubernetes.io/projected/11f27521-38d0-47d3-8884-accc44d652cb-istio-token\") pod \"11f27521-38d0-47d3-8884-accc44d652cb\" (UID: \"11f27521-38d0-47d3-8884-accc44d652cb\") "
Oct 31 01:31:33.454163 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:31:33.454103    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/11f27521-38d0-47d3-8884-accc44d652cb-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/11f27521-38d0-47d3-8884-accc44d652cb-istio-podinfo\") pod \"11f27521-38d0-47d3-8884-accc44d652cb\" (UID: \"11f27521-38d0-47d3-8884-accc44d652cb\") "
Oct 31 01:31:33.454163 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:31:33.454132    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/11f27521-38d0-47d3-8884-accc44d652cb-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/11f27521-38d0-47d3-8884-accc44d652cb-workload-certs\") pod \"11f27521-38d0-47d3-8884-accc44d652cb\" (UID: \"11f27521-38d0-47d3-8884-accc44d652cb\") "
Oct 31 01:31:33.454259 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:31:33.454176    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11f27521-38d0-47d3-8884-accc44d652cb-credential-socket" pod "11f27521-38d0-47d3-8884-accc44d652cb" (UID: "11f27521-38d0-47d3-8884-accc44d652cb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:31:33.454259 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:31:33.454204    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11f27521-38d0-47d3-8884-accc44d652cb-istio-data" pod "11f27521-38d0-47d3-8884-accc44d652cb" (UID: "11f27521-38d0-47d3-8884-accc44d652cb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:31:33.454259 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:31:33.454209    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/11f27521-38d0-47d3-8884-accc44d652cb-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/11f27521-38d0-47d3-8884-accc44d652cb-istiod-ca-cert\") pod \"11f27521-38d0-47d3-8884-accc44d652cb\" (UID: \"11f27521-38d0-47d3-8884-accc44d652cb\") "
Oct 31 01:31:33.454610 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:31:33.454586    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/11f27521-38d0-47d3-8884-accc44d652cb-istiod-ca-cert" pod "11f27521-38d0-47d3-8884-accc44d652cb" (UID: "11f27521-38d0-47d3-8884-accc44d652cb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:31:33.454928 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:31:33.454906    4964 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/11f27521-38d0-47d3-8884-accc44d652cb-istiod-ca-cert\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:31:33.455019 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:31:33.454937    4964 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/11f27521-38d0-47d3-8884-accc44d652cb-workload-socket\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:31:33.455019 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:31:33.454952    4964 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/11f27521-38d0-47d3-8884-accc44d652cb-credential-socket\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:31:33.455019 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:31:33.454967    4964 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/11f27521-38d0-47d3-8884-accc44d652cb-istio-data\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:31:33.456357 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:31:33.456309    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11f27521-38d0-47d3-8884-accc44d652cb-workload-certs" pod "11f27521-38d0-47d3-8884-accc44d652cb" (UID: "11f27521-38d0-47d3-8884-accc44d652cb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:31:33.456699 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:31:33.456673    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11f27521-38d0-47d3-8884-accc44d652cb-istio-token" pod "11f27521-38d0-47d3-8884-accc44d652cb" (UID: "11f27521-38d0-47d3-8884-accc44d652cb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:31:33.456817 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:31:33.456793    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11f27521-38d0-47d3-8884-accc44d652cb-kube-api-access-8fh6j" pod "11f27521-38d0-47d3-8884-accc44d652cb" (UID: "11f27521-38d0-47d3-8884-accc44d652cb"). InnerVolumeSpecName "kube-api-access-8fh6j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:31:33.456892 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:31:33.456848    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11f27521-38d0-47d3-8884-accc44d652cb-istio-envoy" pod "11f27521-38d0-47d3-8884-accc44d652cb" (UID: "11f27521-38d0-47d3-8884-accc44d652cb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:31:33.456970 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:31:33.456954    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/11f27521-38d0-47d3-8884-accc44d652cb-istio-podinfo" pod "11f27521-38d0-47d3-8884-accc44d652cb" (UID: "11f27521-38d0-47d3-8884-accc44d652cb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 01:31:33.555797 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:31:33.555766    4964 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8fh6j\" (UniqueName: \"kubernetes.io/projected/11f27521-38d0-47d3-8884-accc44d652cb-kube-api-access-8fh6j\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:31:33.555797 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:31:33.555797    4964 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/11f27521-38d0-47d3-8884-accc44d652cb-istio-envoy\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:31:33.555978 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:31:33.555811    4964 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/11f27521-38d0-47d3-8884-accc44d652cb-istio-token\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:31:33.555978 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:31:33.555823    4964 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/11f27521-38d0-47d3-8884-accc44d652cb-istio-podinfo\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:31:33.555978 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:31:33.555835    4964 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/11f27521-38d0-47d3-8884-accc44d652cb-workload-certs\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:31:33.924649 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:31:33.924613    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:31:34.162770 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:31:34.162689    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:31:34.272796 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:31:34.272764    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-7rdlf" event={"ID":"11f27521-38d0-47d3-8884-accc44d652cb","Type":"ContainerDied","Data":"7e569dfa1a72d8facf5ef11d7513542ee8a36f6b0013a2bddeac16d79fd32d7d"}
Oct 31 01:31:34.272796 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:31:34.272795    4964 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-7rdlf"
Oct 31 01:31:34.273308 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:31:34.272835    4964 scope.go:122] "RemoveContainer" containerID="8db46917c1436ffe81adc65b48b30690d2948bb71f66b67021c430a74b6368e9"
Oct 31 01:31:34.489096 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:31:34.489065    4964 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-7rdlf"]
Oct 31 01:31:34.496735 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:31:34.496705    4964 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-7rdlf"]
Oct 31 01:31:34.641712 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:31:34.641668    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:31:35.778379 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:31:35.778347    4964 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="11f27521-38d0-47d3-8884-accc44d652cb" path="/var/lib/kubelet/pods/11f27521-38d0-47d3-8884-accc44d652cb/volumes"
Oct 31 01:31:37.709626 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:31:37.709589    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:31:37.916077 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:31:37.916044    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:31:38.096087 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:31:38.096050    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:31:39.124444 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:31:39.124396    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:31:39.752743 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:31:39.752701    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:31:39.952017 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:31:39.951985    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:31:41.239377 ip-10-0-133-197 kubenswrapper[4964]: E1031 01:31:41.239344    4964 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q56x5" containerName="extractor"
Oct 31 01:31:44.916770 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:31:44.916739    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:31:50.046659 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:31:50.046620    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:31:51.351920 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:31:51.351888    4964 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-01cbd8-75c567cb95-tpmh5"]
Oct 31 01:31:51.465254 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:31:51.465215    4964 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-01cbd8-f74599447-dqk2f"]
Oct 31 01:31:51.751288 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:31:51.751251    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:31:51.927880 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:31:51.927841    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:31:55.388498 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:31:55.388459    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:31:56.434208 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:31:56.434162    4964 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-01cbd8-75c567cb95-tpmh5" podUID="086815fd-df1b-4b33-a457-5f67321d8224" containerName="payload-processing" containerID="cri-o://95e1130d0c5f85f21f8b944a35613eb4bcf22513fd52227884b8ff22e332a0f2" gracePeriod=25
Oct 31 01:31:56.465566 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:31:56.465519    4964 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-01cbd8-f74599447-dqk2f" podUID="68178bce-da5e-4260-bdee-291a5d7bd028" containerName="payload-pre-processing" containerID="cri-o://b033908913c9b6f9d5a64503b7f06256b112e8742c763f61262410896d216321" gracePeriod=25
Oct 31 01:31:56.779897 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:31:56.779866    4964 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-01cbd8-75c567cb95-tpmh5"
Oct 31 01:31:56.810198 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:31:56.810169    4964 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-01cbd8-f74599447-dqk2f"
Oct 31 01:31:56.874695 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:31:56.874666    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/086815fd-df1b-4b33-a457-5f67321d8224-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/086815fd-df1b-4b33-a457-5f67321d8224-plugins-config-volume\") pod \"086815fd-df1b-4b33-a457-5f67321d8224\" (UID: \"086815fd-df1b-4b33-a457-5f67321d8224\") "
Oct 31 01:31:56.874822 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:31:56.874814    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/086815fd-df1b-4b33-a457-5f67321d8224-kube-api-access-pdbjt\" (UniqueName: \"kubernetes.io/projected/086815fd-df1b-4b33-a457-5f67321d8224-kube-api-access-pdbjt\") pod \"086815fd-df1b-4b33-a457-5f67321d8224\" (UID: \"086815fd-df1b-4b33-a457-5f67321d8224\") "
Oct 31 01:31:56.875081 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:31:56.875053    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/086815fd-df1b-4b33-a457-5f67321d8224-plugins-config-volume" pod "086815fd-df1b-4b33-a457-5f67321d8224" (UID: "086815fd-df1b-4b33-a457-5f67321d8224"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:31:56.875537 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:31:56.875448    4964 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/086815fd-df1b-4b33-a457-5f67321d8224-plugins-config-volume\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:31:56.877034 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:31:56.877014    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/086815fd-df1b-4b33-a457-5f67321d8224-kube-api-access-pdbjt" pod "086815fd-df1b-4b33-a457-5f67321d8224" (UID: "086815fd-df1b-4b33-a457-5f67321d8224"). InnerVolumeSpecName "kube-api-access-pdbjt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:31:56.976085 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:31:56.976065    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/68178bce-da5e-4260-bdee-291a5d7bd028-kube-api-access-8225k\" (UniqueName: \"kubernetes.io/projected/68178bce-da5e-4260-bdee-291a5d7bd028-kube-api-access-8225k\") pod \"68178bce-da5e-4260-bdee-291a5d7bd028\" (UID: \"68178bce-da5e-4260-bdee-291a5d7bd028\") "
Oct 31 01:31:56.976254 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:31:56.976135    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/68178bce-da5e-4260-bdee-291a5d7bd028-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/68178bce-da5e-4260-bdee-291a5d7bd028-plugins-config-volume\") pod \"68178bce-da5e-4260-bdee-291a5d7bd028\" (UID: \"68178bce-da5e-4260-bdee-291a5d7bd028\") "
Oct 31 01:31:56.976628 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:31:56.976599    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/68178bce-da5e-4260-bdee-291a5d7bd028-plugins-config-volume" pod "68178bce-da5e-4260-bdee-291a5d7bd028" (UID: "68178bce-da5e-4260-bdee-291a5d7bd028"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:31:56.976831 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:31:56.976780    4964 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pdbjt\" (UniqueName: \"kubernetes.io/projected/086815fd-df1b-4b33-a457-5f67321d8224-kube-api-access-pdbjt\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:31:56.976831 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:31:56.976803    4964 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/68178bce-da5e-4260-bdee-291a5d7bd028-plugins-config-volume\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:31:56.978409 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:31:56.978386    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/68178bce-da5e-4260-bdee-291a5d7bd028-kube-api-access-8225k" pod "68178bce-da5e-4260-bdee-291a5d7bd028" (UID: "68178bce-da5e-4260-bdee-291a5d7bd028"). InnerVolumeSpecName "kube-api-access-8225k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:31:57.078150 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:31:57.078117    4964 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8225k\" (UniqueName: \"kubernetes.io/projected/68178bce-da5e-4260-bdee-291a5d7bd028-kube-api-access-8225k\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:31:57.477466 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:31:57.477437    4964 generic.go:356] "Generic (PLEG): container finished" podID="086815fd-df1b-4b33-a457-5f67321d8224" containerID="95e1130d0c5f85f21f8b944a35613eb4bcf22513fd52227884b8ff22e332a0f2" exitCode=0
Oct 31 01:31:57.477888 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:31:57.477482    4964 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-01cbd8-75c567cb95-tpmh5"
Oct 31 01:31:57.477888 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:31:57.477532    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-01cbd8-75c567cb95-tpmh5" event={"ID":"086815fd-df1b-4b33-a457-5f67321d8224","Type":"ContainerDied","Data":"95e1130d0c5f85f21f8b944a35613eb4bcf22513fd52227884b8ff22e332a0f2"}
Oct 31 01:31:57.477888 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:31:57.477595    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-01cbd8-75c567cb95-tpmh5" event={"ID":"086815fd-df1b-4b33-a457-5f67321d8224","Type":"ContainerDied","Data":"0ced1e5730d293c3c71f1c6b1ffa376661ecafa13ad3ec402d0426db6f67a0b6"}
Oct 31 01:31:57.477888 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:31:57.477628    4964 scope.go:122] "RemoveContainer" containerID="95e1130d0c5f85f21f8b944a35613eb4bcf22513fd52227884b8ff22e332a0f2"
Oct 31 01:31:57.478936 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:31:57.478919    4964 generic.go:356] "Generic (PLEG): container finished" podID="68178bce-da5e-4260-bdee-291a5d7bd028" containerID="b033908913c9b6f9d5a64503b7f06256b112e8742c763f61262410896d216321" exitCode=0
Oct 31 01:31:57.478997 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:31:57.478949    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-01cbd8-f74599447-dqk2f" event={"ID":"68178bce-da5e-4260-bdee-291a5d7bd028","Type":"ContainerDied","Data":"b033908913c9b6f9d5a64503b7f06256b112e8742c763f61262410896d216321"}
Oct 31 01:31:57.478997 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:31:57.478973    4964 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-01cbd8-f74599447-dqk2f"
Oct 31 01:31:57.479068 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:31:57.478997    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-01cbd8-f74599447-dqk2f" event={"ID":"68178bce-da5e-4260-bdee-291a5d7bd028","Type":"ContainerDied","Data":"e15c5b94ba8f79fabb65ec125568045bc9aee8a6e1088c812b965e981a664ada"}
Oct 31 01:31:57.490811 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:31:57.490797    4964 scope.go:122] "RemoveContainer" containerID="95e1130d0c5f85f21f8b944a35613eb4bcf22513fd52227884b8ff22e332a0f2"
Oct 31 01:31:57.491095 ip-10-0-133-197 kubenswrapper[4964]: E1031 01:31:57.491077    4964 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"95e1130d0c5f85f21f8b944a35613eb4bcf22513fd52227884b8ff22e332a0f2\": container with ID starting with 95e1130d0c5f85f21f8b944a35613eb4bcf22513fd52227884b8ff22e332a0f2 not found: ID does not exist" containerID="95e1130d0c5f85f21f8b944a35613eb4bcf22513fd52227884b8ff22e332a0f2"
Oct 31 01:31:57.491173 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:31:57.491100    4964 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"95e1130d0c5f85f21f8b944a35613eb4bcf22513fd52227884b8ff22e332a0f2"} err="failed to get container status \"95e1130d0c5f85f21f8b944a35613eb4bcf22513fd52227884b8ff22e332a0f2\": rpc error: code = NotFound desc = could not find container \"95e1130d0c5f85f21f8b944a35613eb4bcf22513fd52227884b8ff22e332a0f2\": container with ID starting with 95e1130d0c5f85f21f8b944a35613eb4bcf22513fd52227884b8ff22e332a0f2 not found: ID does not exist"
Oct 31 01:31:57.491173 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:31:57.491116    4964 scope.go:122] "RemoveContainer" containerID="b033908913c9b6f9d5a64503b7f06256b112e8742c763f61262410896d216321"
Oct 31 01:31:57.500399 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:31:57.500381    4964 scope.go:122] "RemoveContainer" containerID="b033908913c9b6f9d5a64503b7f06256b112e8742c763f61262410896d216321"
Oct 31 01:31:57.500687 ip-10-0-133-197 kubenswrapper[4964]: E1031 01:31:57.500665    4964 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b033908913c9b6f9d5a64503b7f06256b112e8742c763f61262410896d216321\": container with ID starting with b033908913c9b6f9d5a64503b7f06256b112e8742c763f61262410896d216321 not found: ID does not exist" containerID="b033908913c9b6f9d5a64503b7f06256b112e8742c763f61262410896d216321"
Oct 31 01:31:57.500792 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:31:57.500692    4964 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b033908913c9b6f9d5a64503b7f06256b112e8742c763f61262410896d216321"} err="failed to get container status \"b033908913c9b6f9d5a64503b7f06256b112e8742c763f61262410896d216321\": rpc error: code = NotFound desc = could not find container \"b033908913c9b6f9d5a64503b7f06256b112e8742c763f61262410896d216321\": container with ID starting with b033908913c9b6f9d5a64503b7f06256b112e8742c763f61262410896d216321 not found: ID does not exist"
Oct 31 01:31:57.642379 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:31:57.642342    4964 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-01cbd8-75c567cb95-tpmh5"]
Oct 31 01:31:57.691709 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:31:57.691675    4964 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-01cbd8-75c567cb95-tpmh5"]
Oct 31 01:31:57.752420 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:31:57.752389    4964 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-01cbd8-f74599447-dqk2f"]
Oct 31 01:31:57.756303 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:31:57.756281    4964 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-01cbd8-f74599447-dqk2f"]
Oct 31 01:31:57.778232 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:31:57.778206    4964 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="086815fd-df1b-4b33-a457-5f67321d8224" path="/var/lib/kubelet/pods/086815fd-df1b-4b33-a457-5f67321d8224/volumes"
Oct 31 01:31:57.778633 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:31:57.778617    4964 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="68178bce-da5e-4260-bdee-291a5d7bd028" path="/var/lib/kubelet/pods/68178bce-da5e-4260-bdee-291a5d7bd028/volumes"
Oct 31 01:31:58.667544 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:31:58.667509    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:31:59.999613 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:31:59.999580    4964 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-bc2046-7988f9b774-pz82x"]
Oct 31 01:32:00.000236 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:32:00.000221    4964 cpu_manager.go:414] "containerMap: removing container" podUID="b9a6e0ab-3025-4363-8ad7-4434733148b7" containerName="payload-pre-processing"
Oct 31 01:32:00.000280 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:32:00.000237    4964 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b9a6e0ab-3025-4363-8ad7-4434733148b7" containerName="payload-pre-processing"
Oct 31 01:32:00.000280 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:32:00.000250    4964 cpu_manager.go:414] "containerMap: removing container" podUID="68178bce-da5e-4260-bdee-291a5d7bd028" containerName="payload-pre-processing"
Oct 31 01:32:00.000280 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:32:00.000254    4964 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="68178bce-da5e-4260-bdee-291a5d7bd028" containerName="payload-pre-processing"
Oct 31 01:32:00.000280 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:32:00.000268    4964 cpu_manager.go:414] "containerMap: removing container" podUID="086815fd-df1b-4b33-a457-5f67321d8224" containerName="payload-processing"
Oct 31 01:32:00.000280 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:32:00.000273    4964 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="086815fd-df1b-4b33-a457-5f67321d8224" containerName="payload-processing"
Oct 31 01:32:00.000280 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:32:00.000283    4964 cpu_manager.go:414] "containerMap: removing container" podUID="11f27521-38d0-47d3-8884-accc44d652cb" containerName="istio-proxy"
Oct 31 01:32:00.000500 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:32:00.000288    4964 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="11f27521-38d0-47d3-8884-accc44d652cb" containerName="istio-proxy"
Oct 31 01:32:00.000500 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:32:00.000299    4964 cpu_manager.go:414] "containerMap: removing container" podUID="a26b5198-d0fe-403b-858f-0e6619c15c3b" containerName="payload-processing"
Oct 31 01:32:00.000500 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:32:00.000303    4964 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a26b5198-d0fe-403b-858f-0e6619c15c3b" containerName="payload-processing"
Oct 31 01:32:00.000500 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:32:00.000469    4964 memory_manager.go:365] "RemoveStaleState removing state" podUID="b9a6e0ab-3025-4363-8ad7-4434733148b7" containerName="payload-pre-processing"
Oct 31 01:32:00.000500 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:32:00.000480    4964 memory_manager.go:365] "RemoveStaleState removing state" podUID="68178bce-da5e-4260-bdee-291a5d7bd028" containerName="payload-pre-processing"
Oct 31 01:32:00.000500 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:32:00.000487    4964 memory_manager.go:365] "RemoveStaleState removing state" podUID="086815fd-df1b-4b33-a457-5f67321d8224" containerName="payload-processing"
Oct 31 01:32:00.000500 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:32:00.000494    4964 memory_manager.go:365] "RemoveStaleState removing state" podUID="11f27521-38d0-47d3-8884-accc44d652cb" containerName="istio-proxy"
Oct 31 01:32:00.000500 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:32:00.000500    4964 memory_manager.go:365] "RemoveStaleState removing state" podUID="a26b5198-d0fe-403b-858f-0e6619c15c3b" containerName="payload-processing"
Oct 31 01:32:00.005514 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:32:00.005498    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bc2046-7988f9b774-pz82x"
Oct 31 01:32:00.060825 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:32:00.060795    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-bc2046-7988f9b774-pz82x"]
Oct 31 01:32:00.104958 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:32:00.104923    4964 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-bc2046-serving-cert\""
Oct 31 01:32:00.115477 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:32:00.115445    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nr2bv\" (UniqueName: \"kubernetes.io/projected/65cf4fa6-618d-44d2-91a6-2b8d7b1dcde2-kube-api-access-nr2bv\") pod \"maas-api-e2e-api-b-bc2046-7988f9b774-pz82x\" (UID: \"65cf4fa6-618d-44d2-91a6-2b8d7b1dcde2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bc2046-7988f9b774-pz82x"
Oct 31 01:32:00.115614 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:32:00.115603    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/65cf4fa6-618d-44d2-91a6-2b8d7b1dcde2-maas-api-tls\") pod \"maas-api-e2e-api-b-bc2046-7988f9b774-pz82x\" (UID: \"65cf4fa6-618d-44d2-91a6-2b8d7b1dcde2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bc2046-7988f9b774-pz82x"
Oct 31 01:32:00.115663 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:32:00.115651    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/65cf4fa6-618d-44d2-91a6-2b8d7b1dcde2-openshift-service-ca\") pod \"maas-api-e2e-api-b-bc2046-7988f9b774-pz82x\" (UID: \"65cf4fa6-618d-44d2-91a6-2b8d7b1dcde2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bc2046-7988f9b774-pz82x"
Oct 31 01:32:00.115811 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:32:00.115795    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/65cf4fa6-618d-44d2-91a6-2b8d7b1dcde2-metrics-certs\") pod \"maas-api-e2e-api-b-bc2046-7988f9b774-pz82x\" (UID: \"65cf4fa6-618d-44d2-91a6-2b8d7b1dcde2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bc2046-7988f9b774-pz82x"
Oct 31 01:32:00.216665 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:32:00.216634    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/65cf4fa6-618d-44d2-91a6-2b8d7b1dcde2-metrics-certs\") pod \"maas-api-e2e-api-b-bc2046-7988f9b774-pz82x\" (UID: \"65cf4fa6-618d-44d2-91a6-2b8d7b1dcde2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bc2046-7988f9b774-pz82x"
Oct 31 01:32:00.216881 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:32:00.216686    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nr2bv\" (UniqueName: \"kubernetes.io/projected/65cf4fa6-618d-44d2-91a6-2b8d7b1dcde2-kube-api-access-nr2bv\") pod \"maas-api-e2e-api-b-bc2046-7988f9b774-pz82x\" (UID: \"65cf4fa6-618d-44d2-91a6-2b8d7b1dcde2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bc2046-7988f9b774-pz82x"
Oct 31 01:32:00.216881 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:32:00.216798    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/65cf4fa6-618d-44d2-91a6-2b8d7b1dcde2-maas-api-tls\") pod \"maas-api-e2e-api-b-bc2046-7988f9b774-pz82x\" (UID: \"65cf4fa6-618d-44d2-91a6-2b8d7b1dcde2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bc2046-7988f9b774-pz82x"
Oct 31 01:32:00.216881 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:32:00.216867    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/65cf4fa6-618d-44d2-91a6-2b8d7b1dcde2-openshift-service-ca\") pod \"maas-api-e2e-api-b-bc2046-7988f9b774-pz82x\" (UID: \"65cf4fa6-618d-44d2-91a6-2b8d7b1dcde2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bc2046-7988f9b774-pz82x"
Oct 31 01:32:00.217067 ip-10-0-133-197 kubenswrapper[4964]: E1031 01:32:00.216958    4964 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-bc2046-serving-cert: secret "maas-api-e2e-api-b-bc2046-serving-cert" not found
Oct 31 01:32:00.217067 ip-10-0-133-197 kubenswrapper[4964]: E1031 01:32:00.217027    4964 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/65cf4fa6-618d-44d2-91a6-2b8d7b1dcde2-maas-api-tls podName:65cf4fa6-618d-44d2-91a6-2b8d7b1dcde2 nodeName:}" failed. No retries permitted until 2026-10-31 01:32:00.717007526 +0000 UTC m=+1689.193573898 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/65cf4fa6-618d-44d2-91a6-2b8d7b1dcde2-maas-api-tls") pod "maas-api-e2e-api-b-bc2046-7988f9b774-pz82x" (UID: "65cf4fa6-618d-44d2-91a6-2b8d7b1dcde2") : secret "maas-api-e2e-api-b-bc2046-serving-cert" not found
Oct 31 01:32:00.217685 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:32:00.217661    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/65cf4fa6-618d-44d2-91a6-2b8d7b1dcde2-openshift-service-ca\") pod \"maas-api-e2e-api-b-bc2046-7988f9b774-pz82x\" (UID: \"65cf4fa6-618d-44d2-91a6-2b8d7b1dcde2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bc2046-7988f9b774-pz82x"
Oct 31 01:32:00.219494 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:32:00.219471    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/65cf4fa6-618d-44d2-91a6-2b8d7b1dcde2-metrics-certs\") pod \"maas-api-e2e-api-b-bc2046-7988f9b774-pz82x\" (UID: \"65cf4fa6-618d-44d2-91a6-2b8d7b1dcde2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bc2046-7988f9b774-pz82x"
Oct 31 01:32:00.237311 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:32:00.237278    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:32:00.276294 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:32:00.276226    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nr2bv\" (UniqueName: \"kubernetes.io/projected/65cf4fa6-618d-44d2-91a6-2b8d7b1dcde2-kube-api-access-nr2bv\") pod \"maas-api-e2e-api-b-bc2046-7988f9b774-pz82x\" (UID: \"65cf4fa6-618d-44d2-91a6-2b8d7b1dcde2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bc2046-7988f9b774-pz82x"
Oct 31 01:32:00.412390 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:32:00.412253    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:32:00.612932 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:32:00.612851    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:32:00.723968 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:32:00.723932    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/65cf4fa6-618d-44d2-91a6-2b8d7b1dcde2-maas-api-tls\") pod \"maas-api-e2e-api-b-bc2046-7988f9b774-pz82x\" (UID: \"65cf4fa6-618d-44d2-91a6-2b8d7b1dcde2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bc2046-7988f9b774-pz82x"
Oct 31 01:32:00.726732 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:32:00.726702    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/65cf4fa6-618d-44d2-91a6-2b8d7b1dcde2-maas-api-tls\") pod \"maas-api-e2e-api-b-bc2046-7988f9b774-pz82x\" (UID: \"65cf4fa6-618d-44d2-91a6-2b8d7b1dcde2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bc2046-7988f9b774-pz82x"
Oct 31 01:32:00.922350 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:32:00.922245    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bc2046-7988f9b774-pz82x"
Oct 31 01:32:01.335484 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:32:01.335456    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-bc2046-7988f9b774-pz82x"]
Oct 31 01:32:01.337196 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:32:01.337170    4964 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod65cf4fa6_618d_44d2_91a6_2b8d7b1dcde2.slice/crio-a2e756ab327ef7ff406a4df2ea329002c8632b5f6975f7bd05ae414e83dfbd61 WatchSource:0}: Error finding container a2e756ab327ef7ff406a4df2ea329002c8632b5f6975f7bd05ae414e83dfbd61: Status 404 returned error can't find the container with id a2e756ab327ef7ff406a4df2ea329002c8632b5f6975f7bd05ae414e83dfbd61
Oct 31 01:32:01.524155 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:32:01.524123    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bc2046-7988f9b774-pz82x" event={"ID":"65cf4fa6-618d-44d2-91a6-2b8d7b1dcde2","Type":"ContainerStarted","Data":"a2e756ab327ef7ff406a4df2ea329002c8632b5f6975f7bd05ae414e83dfbd61"}
Oct 31 01:32:01.927096 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:32:01.927061    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:32:02.099311 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:32:02.099272    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:32:02.532847 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:32:02.532770    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bc2046-7988f9b774-pz82x" event={"ID":"65cf4fa6-618d-44d2-91a6-2b8d7b1dcde2","Type":"ContainerStarted","Data":"fdcdb07f25434852f6a2fccd7cff41e1655d9fa1884d5084b9d0694840dd4f53"}
Oct 31 01:32:02.533200 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:32:02.532987    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bc2046-7988f9b774-pz82x"
Oct 31 01:32:05.191697 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:32:05.191658    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:32:08.543875 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:32:08.543833    4964 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bc2046-7988f9b774-pz82x" podUID="65cf4fa6-618d-44d2-91a6-2b8d7b1dcde2" containerName="maas-api" probeResult="failure" output="Get \"https://10.132.0.119:8443/health\": dial tcp 10.132.0.119:8443: connect: connection refused"
Oct 31 01:32:10.095827 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:32:10.095784    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:32:10.307511 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:32:10.307468    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:32:11.241650 ip-10-0-133-197 kubenswrapper[4964]: E1031 01:32:11.241618    4964 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q56x5" containerName="extractor"
Oct 31 01:32:12.614772 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:32:12.614738    4964 generic.go:356] "Generic (PLEG): container finished" podID="65cf4fa6-618d-44d2-91a6-2b8d7b1dcde2" containerID="fdcdb07f25434852f6a2fccd7cff41e1655d9fa1884d5084b9d0694840dd4f53" exitCode=1
Oct 31 01:32:12.615223 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:32:12.614802    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bc2046-7988f9b774-pz82x" event={"ID":"65cf4fa6-618d-44d2-91a6-2b8d7b1dcde2","Type":"ContainerDied","Data":"fdcdb07f25434852f6a2fccd7cff41e1655d9fa1884d5084b9d0694840dd4f53"}
Oct 31 01:32:12.615308 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:32:12.615286    4964 scope.go:122] "RemoveContainer" containerID="fdcdb07f25434852f6a2fccd7cff41e1655d9fa1884d5084b9d0694840dd4f53"
Oct 31 01:32:13.624600 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:32:13.624563    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bc2046-7988f9b774-pz82x" event={"ID":"65cf4fa6-618d-44d2-91a6-2b8d7b1dcde2","Type":"ContainerStarted","Data":"c7a8511df73e48da71ad92a53f2441f1b3ac4d0d8bfd14baa046a5f3e8ac3604"}
Oct 31 01:32:13.625045 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:32:13.624792    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bc2046-7988f9b774-pz82x"
Oct 31 01:32:13.744618 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:32:13.744572    4964 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bc2046-7988f9b774-pz82x" podStartSLOduration=12.938921964 podStartE2EDuration="14.74455934s" totalImagesPullingTime="1.805637376s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:31:59 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:32:02.677505691 +0000 UTC m=+1691.154072080" watchObservedRunningTime="2026-10-31 01:32:13.74455934 +0000 UTC m=+1702.221125719"
Oct 31 01:32:14.900092 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:32:14.900056    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:32:19.640181 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:32:19.640152    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bc2046-7988f9b774-pz82x"
Oct 31 01:32:19.686163 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:32:19.686129    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:32:19.904560 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:32:19.904476    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:32:24.593103 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:32:24.593064    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:32:29.009943 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:32:29.009912    4964 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-57ea84-bd5bb4cd7-vkghl"]
Oct 31 01:32:29.014852 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:32:29.014833    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-57ea84-bd5bb4cd7-vkghl"
Oct 31 01:32:29.118537 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:32:29.118499    4964 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-57ea84-serving-cert\""
Oct 31 01:32:29.119256 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:32:29.119222    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/afa6c9e7-5acc-4894-ab75-38c00dc49ae2-openshift-service-ca\") pod \"maas-api-e2e-api-b-57ea84-bd5bb4cd7-vkghl\" (UID: \"afa6c9e7-5acc-4894-ab75-38c00dc49ae2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-57ea84-bd5bb4cd7-vkghl"
Oct 31 01:32:29.119468 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:32:29.119448    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/afa6c9e7-5acc-4894-ab75-38c00dc49ae2-metrics-certs\") pod \"maas-api-e2e-api-b-57ea84-bd5bb4cd7-vkghl\" (UID: \"afa6c9e7-5acc-4894-ab75-38c00dc49ae2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-57ea84-bd5bb4cd7-vkghl"
Oct 31 01:32:29.119613 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:32:29.119591    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/afa6c9e7-5acc-4894-ab75-38c00dc49ae2-maas-api-tls\") pod \"maas-api-e2e-api-b-57ea84-bd5bb4cd7-vkghl\" (UID: \"afa6c9e7-5acc-4894-ab75-38c00dc49ae2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-57ea84-bd5bb4cd7-vkghl"
Oct 31 01:32:29.119813 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:32:29.119790    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dzq5w\" (UniqueName: \"kubernetes.io/projected/afa6c9e7-5acc-4894-ab75-38c00dc49ae2-kube-api-access-dzq5w\") pod \"maas-api-e2e-api-b-57ea84-bd5bb4cd7-vkghl\" (UID: \"afa6c9e7-5acc-4894-ab75-38c00dc49ae2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-57ea84-bd5bb4cd7-vkghl"
Oct 31 01:32:29.122711 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:32:29.122686    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-57ea84-bd5bb4cd7-vkghl"]
Oct 31 01:32:29.221218 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:32:29.221183    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/afa6c9e7-5acc-4894-ab75-38c00dc49ae2-maas-api-tls\") pod \"maas-api-e2e-api-b-57ea84-bd5bb4cd7-vkghl\" (UID: \"afa6c9e7-5acc-4894-ab75-38c00dc49ae2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-57ea84-bd5bb4cd7-vkghl"
Oct 31 01:32:29.221445 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:32:29.221287    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dzq5w\" (UniqueName: \"kubernetes.io/projected/afa6c9e7-5acc-4894-ab75-38c00dc49ae2-kube-api-access-dzq5w\") pod \"maas-api-e2e-api-b-57ea84-bd5bb4cd7-vkghl\" (UID: \"afa6c9e7-5acc-4894-ab75-38c00dc49ae2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-57ea84-bd5bb4cd7-vkghl"
Oct 31 01:32:29.221445 ip-10-0-133-197 kubenswrapper[4964]: E1031 01:32:29.221363    4964 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-57ea84-serving-cert: secret "maas-api-e2e-api-b-57ea84-serving-cert" not found
Oct 31 01:32:29.221535 ip-10-0-133-197 kubenswrapper[4964]: E1031 01:32:29.221458    4964 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/afa6c9e7-5acc-4894-ab75-38c00dc49ae2-maas-api-tls podName:afa6c9e7-5acc-4894-ab75-38c00dc49ae2 nodeName:}" failed. No retries permitted until 2026-10-31 01:32:29.721440682 +0000 UTC m=+1718.198007060 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/afa6c9e7-5acc-4894-ab75-38c00dc49ae2-maas-api-tls") pod "maas-api-e2e-api-b-57ea84-bd5bb4cd7-vkghl" (UID: "afa6c9e7-5acc-4894-ab75-38c00dc49ae2") : secret "maas-api-e2e-api-b-57ea84-serving-cert" not found
Oct 31 01:32:29.221535 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:32:29.221369    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/afa6c9e7-5acc-4894-ab75-38c00dc49ae2-openshift-service-ca\") pod \"maas-api-e2e-api-b-57ea84-bd5bb4cd7-vkghl\" (UID: \"afa6c9e7-5acc-4894-ab75-38c00dc49ae2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-57ea84-bd5bb4cd7-vkghl"
Oct 31 01:32:29.221629 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:32:29.221613    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/afa6c9e7-5acc-4894-ab75-38c00dc49ae2-metrics-certs\") pod \"maas-api-e2e-api-b-57ea84-bd5bb4cd7-vkghl\" (UID: \"afa6c9e7-5acc-4894-ab75-38c00dc49ae2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-57ea84-bd5bb4cd7-vkghl"
Oct 31 01:32:29.221933 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:32:29.221917    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/afa6c9e7-5acc-4894-ab75-38c00dc49ae2-openshift-service-ca\") pod \"maas-api-e2e-api-b-57ea84-bd5bb4cd7-vkghl\" (UID: \"afa6c9e7-5acc-4894-ab75-38c00dc49ae2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-57ea84-bd5bb4cd7-vkghl"
Oct 31 01:32:29.224211 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:32:29.224184    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/afa6c9e7-5acc-4894-ab75-38c00dc49ae2-metrics-certs\") pod \"maas-api-e2e-api-b-57ea84-bd5bb4cd7-vkghl\" (UID: \"afa6c9e7-5acc-4894-ab75-38c00dc49ae2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-57ea84-bd5bb4cd7-vkghl"
Oct 31 01:32:29.253557 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:32:29.253522    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:32:29.279680 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:32:29.279603    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dzq5w\" (UniqueName: \"kubernetes.io/projected/afa6c9e7-5acc-4894-ab75-38c00dc49ae2-kube-api-access-dzq5w\") pod \"maas-api-e2e-api-b-57ea84-bd5bb4cd7-vkghl\" (UID: \"afa6c9e7-5acc-4894-ab75-38c00dc49ae2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-57ea84-bd5bb4cd7-vkghl"
Oct 31 01:32:29.428899 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:32:29.428866    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:32:29.600948 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:32:29.600833    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:32:29.728436 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:32:29.728404    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/afa6c9e7-5acc-4894-ab75-38c00dc49ae2-maas-api-tls\") pod \"maas-api-e2e-api-b-57ea84-bd5bb4cd7-vkghl\" (UID: \"afa6c9e7-5acc-4894-ab75-38c00dc49ae2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-57ea84-bd5bb4cd7-vkghl"
Oct 31 01:32:29.731054 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:32:29.731030    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/afa6c9e7-5acc-4894-ab75-38c00dc49ae2-maas-api-tls\") pod \"maas-api-e2e-api-b-57ea84-bd5bb4cd7-vkghl\" (UID: \"afa6c9e7-5acc-4894-ab75-38c00dc49ae2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-57ea84-bd5bb4cd7-vkghl"
Oct 31 01:32:29.928572 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:32:29.928485    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-57ea84-bd5bb4cd7-vkghl"
Oct 31 01:32:30.311042 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:32:30.311010    4964 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podafa6c9e7_5acc_4894_ab75_38c00dc49ae2.slice/crio-daaff8873aad701d00b643b32a704d38b3687e3328fbdd97138d79e17497b369 WatchSource:0}: Error finding container daaff8873aad701d00b643b32a704d38b3687e3328fbdd97138d79e17497b369: Status 404 returned error can't find the container with id daaff8873aad701d00b643b32a704d38b3687e3328fbdd97138d79e17497b369
Oct 31 01:32:30.311438 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:32:30.311258    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-57ea84-bd5bb4cd7-vkghl"]
Oct 31 01:32:30.770748 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:32:30.770703    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-57ea84-bd5bb4cd7-vkghl" event={"ID":"afa6c9e7-5acc-4894-ab75-38c00dc49ae2","Type":"ContainerStarted","Data":"daaff8873aad701d00b643b32a704d38b3687e3328fbdd97138d79e17497b369"}
Oct 31 01:32:31.780204 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:32:31.780158    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-57ea84-bd5bb4cd7-vkghl" event={"ID":"afa6c9e7-5acc-4894-ab75-38c00dc49ae2","Type":"ContainerStarted","Data":"8018a3510633d3e7a49693b3367564bae9905e35511caf06884f1b84756b0126"}
Oct 31 01:32:31.780659 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:32:31.780246    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-57ea84-bd5bb4cd7-vkghl"
Oct 31 01:32:31.897995 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:32:31.897948    4964 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-57ea84-bd5bb4cd7-vkghl" podStartSLOduration=3.023337303 podStartE2EDuration="3.897934293s" totalImagesPullingTime="874.59699ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:32:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:32:31.895165054 +0000 UTC m=+1720.371731435" watchObservedRunningTime="2026-10-31 01:32:31.897934293 +0000 UTC m=+1720.374500672"
Oct 31 01:32:33.470700 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:32:33.470661    4964 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-7cec02-56fb787486-ms9st"]
Oct 31 01:32:33.471117 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:32:33.470904    4964 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-7cec02-56fb787486-ms9st" podUID="c8cba8e6-afc6-459a-89a5-615f812b930c" containerName="maas-api" containerID="cri-o://6cf56edb7185751ae6e9c849d46c2c29a2f0ebbde3be50c5fccb8d36c77ab8f9" gracePeriod=30
Oct 31 01:32:33.606372 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:32:33.606336    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:32:33.798364 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:32:33.798336    4964 generic.go:356] "Generic (PLEG): container finished" podID="c8cba8e6-afc6-459a-89a5-615f812b930c" containerID="6cf56edb7185751ae6e9c849d46c2c29a2f0ebbde3be50c5fccb8d36c77ab8f9" exitCode=0
Oct 31 01:32:33.798364 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:32:33.798351    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-7cec02-56fb787486-ms9st" event={"ID":"c8cba8e6-afc6-459a-89a5-615f812b930c","Type":"ContainerDied","Data":"6cf56edb7185751ae6e9c849d46c2c29a2f0ebbde3be50c5fccb8d36c77ab8f9"}
Oct 31 01:32:33.798558 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:32:33.798412    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-7cec02-56fb787486-ms9st" event={"ID":"c8cba8e6-afc6-459a-89a5-615f812b930c","Type":"ContainerDied","Data":"1da8202cb714f5ccc3465c25ff579ad5e85dbb804e0bc48508f0b319b12bb1c5"}
Oct 31 01:32:33.798558 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:32:33.798438    4964 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1da8202cb714f5ccc3465c25ff579ad5e85dbb804e0bc48508f0b319b12bb1c5"
Oct 31 01:32:33.806789 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:32:33.806767    4964 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-7cec02-56fb787486-ms9st"
Oct 31 01:32:33.889933 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:32:33.889890    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c8cba8e6-afc6-459a-89a5-615f812b930c-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c8cba8e6-afc6-459a-89a5-615f812b930c-openshift-service-ca\") pod \"c8cba8e6-afc6-459a-89a5-615f812b930c\" (UID: \"c8cba8e6-afc6-459a-89a5-615f812b930c\") "
Oct 31 01:32:33.890139 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:32:33.889943    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c8cba8e6-afc6-459a-89a5-615f812b930c-kube-api-access-vc6fs\" (UniqueName: \"kubernetes.io/projected/c8cba8e6-afc6-459a-89a5-615f812b930c-kube-api-access-vc6fs\") pod \"c8cba8e6-afc6-459a-89a5-615f812b930c\" (UID: \"c8cba8e6-afc6-459a-89a5-615f812b930c\") "
Oct 31 01:32:33.890139 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:32:33.889995    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c8cba8e6-afc6-459a-89a5-615f812b930c-metrics-certs\" (UniqueName: \"kubernetes.io/secret/c8cba8e6-afc6-459a-89a5-615f812b930c-metrics-certs\") pod \"c8cba8e6-afc6-459a-89a5-615f812b930c\" (UID: \"c8cba8e6-afc6-459a-89a5-615f812b930c\") "
Oct 31 01:32:33.890259 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:32:33.890232    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c8cba8e6-afc6-459a-89a5-615f812b930c-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c8cba8e6-afc6-459a-89a5-615f812b930c-maas-api-tls\") pod \"c8cba8e6-afc6-459a-89a5-615f812b930c\" (UID: \"c8cba8e6-afc6-459a-89a5-615f812b930c\") "
Oct 31 01:32:33.891572 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:32:33.891347    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c8cba8e6-afc6-459a-89a5-615f812b930c-openshift-service-ca" pod "c8cba8e6-afc6-459a-89a5-615f812b930c" (UID: "c8cba8e6-afc6-459a-89a5-615f812b930c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:32:33.895364 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:32:33.895307    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c8cba8e6-afc6-459a-89a5-615f812b930c-metrics-certs" pod "c8cba8e6-afc6-459a-89a5-615f812b930c" (UID: "c8cba8e6-afc6-459a-89a5-615f812b930c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:32:33.911375 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:32:33.909543    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c8cba8e6-afc6-459a-89a5-615f812b930c-kube-api-access-vc6fs" pod "c8cba8e6-afc6-459a-89a5-615f812b930c" (UID: "c8cba8e6-afc6-459a-89a5-615f812b930c"). InnerVolumeSpecName "kube-api-access-vc6fs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:32:33.920505 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:32:33.920472    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c8cba8e6-afc6-459a-89a5-615f812b930c-maas-api-tls" pod "c8cba8e6-afc6-459a-89a5-615f812b930c" (UID: "c8cba8e6-afc6-459a-89a5-615f812b930c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:32:33.992242 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:32:33.992197    4964 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c8cba8e6-afc6-459a-89a5-615f812b930c-openshift-service-ca\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:32:33.992242 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:32:33.992229    4964 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vc6fs\" (UniqueName: \"kubernetes.io/projected/c8cba8e6-afc6-459a-89a5-615f812b930c-kube-api-access-vc6fs\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:32:33.992242 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:32:33.992245    4964 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c8cba8e6-afc6-459a-89a5-615f812b930c-metrics-certs\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:32:33.992515 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:32:33.992257    4964 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c8cba8e6-afc6-459a-89a5-615f812b930c-maas-api-tls\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:32:34.813211 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:32:34.813173    4964 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-7cec02-56fb787486-ms9st"
Oct 31 01:32:34.977706 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:32:34.977677    4964 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-7cec02-56fb787486-ms9st"]
Oct 31 01:32:35.026506 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:32:35.026479    4964 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-7cec02-56fb787486-ms9st"]
Oct 31 01:32:35.167925 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:32:35.167834    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:32:35.400436 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:32:35.400385    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:32:35.576188 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:32:35.576154    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:32:35.779004 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:32:35.778969    4964 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c8cba8e6-afc6-459a-89a5-615f812b930c" path="/var/lib/kubelet/pods/c8cba8e6-afc6-459a-89a5-615f812b930c/volumes"
Oct 31 01:32:36.117074 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:32:36.117034    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:32:37.794023 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:32:37.793998    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-57ea84-bd5bb4cd7-vkghl"
Oct 31 01:32:40.494007 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:32:40.493961    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:32:41.239419 ip-10-0-133-197 kubenswrapper[4964]: E1031 01:32:41.239379    4964 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q56x5" containerName="extractor"
Oct 31 01:32:41.784062 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:32:41.784033    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:32:41.974330 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:32:41.974275    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:32:42.099459 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:32:42.099370    4964 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-7cec02-openshift-default-6fc456b49-jn9wm"]
Oct 31 01:32:42.099649 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:32:42.099623    4964 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-7cec02-openshift-default-6fc456b49-jn9wm" podUID="45450e7a-2b62-4417-acca-d370251f6d54" containerName="istio-proxy" containerID="cri-o://8c9a9da3abc3f800d88d817b91d0d19a6e8c67d550b7bf80bc28279b921e16f0" gracePeriod=30
Oct 31 01:32:42.202435 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:32:42.196870    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:32:44.661413 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:32:44.661350    4964 patch_prober.go:29] interesting pod/e2e-api-a-7cec02-openshift-default-6fc456b49-jn9wm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 01:32:44.661904 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:32:44.661444    4964 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-7cec02-openshift-default-6fc456b49-jn9wm" podUID="45450e7a-2b62-4417-acca-d370251f6d54" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 01:32:45.557253 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:32:45.556952    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:32:47.379039 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:32:47.378974    4964 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-7cec02-openshift-default-6fc456b49-jn9wm"
Oct 31 01:32:47.463576 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:32:47.463552    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/45450e7a-2b62-4417-acca-d370251f6d54-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/45450e7a-2b62-4417-acca-d370251f6d54-credential-socket\") pod \"45450e7a-2b62-4417-acca-d370251f6d54\" (UID: \"45450e7a-2b62-4417-acca-d370251f6d54\") "
Oct 31 01:32:47.463732 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:32:47.463581    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/45450e7a-2b62-4417-acca-d370251f6d54-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/45450e7a-2b62-4417-acca-d370251f6d54-workload-socket\") pod \"45450e7a-2b62-4417-acca-d370251f6d54\" (UID: \"45450e7a-2b62-4417-acca-d370251f6d54\") "
Oct 31 01:32:47.463732 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:32:47.463609    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/45450e7a-2b62-4417-acca-d370251f6d54-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/45450e7a-2b62-4417-acca-d370251f6d54-istio-envoy\") pod \"45450e7a-2b62-4417-acca-d370251f6d54\" (UID: \"45450e7a-2b62-4417-acca-d370251f6d54\") "
Oct 31 01:32:47.463732 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:32:47.463641    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/45450e7a-2b62-4417-acca-d370251f6d54-istio-token\" (UniqueName: \"kubernetes.io/projected/45450e7a-2b62-4417-acca-d370251f6d54-istio-token\") pod \"45450e7a-2b62-4417-acca-d370251f6d54\" (UID: \"45450e7a-2b62-4417-acca-d370251f6d54\") "
Oct 31 01:32:47.463732 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:32:47.463687    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/45450e7a-2b62-4417-acca-d370251f6d54-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/45450e7a-2b62-4417-acca-d370251f6d54-istiod-ca-cert\") pod \"45450e7a-2b62-4417-acca-d370251f6d54\" (UID: \"45450e7a-2b62-4417-acca-d370251f6d54\") "
Oct 31 01:32:47.463965 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:32:47.463833    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45450e7a-2b62-4417-acca-d370251f6d54-credential-socket" pod "45450e7a-2b62-4417-acca-d370251f6d54" (UID: "45450e7a-2b62-4417-acca-d370251f6d54"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:32:47.463965 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:32:47.463854    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45450e7a-2b62-4417-acca-d370251f6d54-workload-socket" pod "45450e7a-2b62-4417-acca-d370251f6d54" (UID: "45450e7a-2b62-4417-acca-d370251f6d54"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:32:47.463965 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:32:47.463861    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/45450e7a-2b62-4417-acca-d370251f6d54-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/45450e7a-2b62-4417-acca-d370251f6d54-istio-data\") pod \"45450e7a-2b62-4417-acca-d370251f6d54\" (UID: \"45450e7a-2b62-4417-acca-d370251f6d54\") "
Oct 31 01:32:47.463965 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:32:47.463930    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/45450e7a-2b62-4417-acca-d370251f6d54-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/45450e7a-2b62-4417-acca-d370251f6d54-workload-certs\") pod \"45450e7a-2b62-4417-acca-d370251f6d54\" (UID: \"45450e7a-2b62-4417-acca-d370251f6d54\") "
Oct 31 01:32:47.464175 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:32:47.464035    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/45450e7a-2b62-4417-acca-d370251f6d54-kube-api-access-lpn8j\" (UniqueName: \"kubernetes.io/projected/45450e7a-2b62-4417-acca-d370251f6d54-kube-api-access-lpn8j\") pod \"45450e7a-2b62-4417-acca-d370251f6d54\" (UID: \"45450e7a-2b62-4417-acca-d370251f6d54\") "
Oct 31 01:32:47.464175 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:32:47.464045    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45450e7a-2b62-4417-acca-d370251f6d54-istio-data" pod "45450e7a-2b62-4417-acca-d370251f6d54" (UID: "45450e7a-2b62-4417-acca-d370251f6d54"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:32:47.464175 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:32:47.464081    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/45450e7a-2b62-4417-acca-d370251f6d54-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/45450e7a-2b62-4417-acca-d370251f6d54-istio-podinfo\") pod \"45450e7a-2b62-4417-acca-d370251f6d54\" (UID: \"45450e7a-2b62-4417-acca-d370251f6d54\") "
Oct 31 01:32:47.464359 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:32:47.464181    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45450e7a-2b62-4417-acca-d370251f6d54-workload-certs" pod "45450e7a-2b62-4417-acca-d370251f6d54" (UID: "45450e7a-2b62-4417-acca-d370251f6d54"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:32:47.464359 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:32:47.464194    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/45450e7a-2b62-4417-acca-d370251f6d54-istiod-ca-cert" pod "45450e7a-2b62-4417-acca-d370251f6d54" (UID: "45450e7a-2b62-4417-acca-d370251f6d54"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:32:47.464726 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:32:47.464701    4964 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/45450e7a-2b62-4417-acca-d370251f6d54-istiod-ca-cert\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:32:47.464865 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:32:47.464729    4964 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/45450e7a-2b62-4417-acca-d370251f6d54-istio-data\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:32:47.464865 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:32:47.464745    4964 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/45450e7a-2b62-4417-acca-d370251f6d54-workload-certs\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:32:47.464865 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:32:47.464759    4964 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/45450e7a-2b62-4417-acca-d370251f6d54-credential-socket\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:32:47.464865 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:32:47.464772    4964 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/45450e7a-2b62-4417-acca-d370251f6d54-workload-socket\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:32:47.466195 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:32:47.466172    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/45450e7a-2b62-4417-acca-d370251f6d54-istio-token" pod "45450e7a-2b62-4417-acca-d370251f6d54" (UID: "45450e7a-2b62-4417-acca-d370251f6d54"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:32:47.466293 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:32:47.466273    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/45450e7a-2b62-4417-acca-d370251f6d54-kube-api-access-lpn8j" pod "45450e7a-2b62-4417-acca-d370251f6d54" (UID: "45450e7a-2b62-4417-acca-d370251f6d54"). InnerVolumeSpecName "kube-api-access-lpn8j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:32:47.466646 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:32:47.466629    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45450e7a-2b62-4417-acca-d370251f6d54-istio-envoy" pod "45450e7a-2b62-4417-acca-d370251f6d54" (UID: "45450e7a-2b62-4417-acca-d370251f6d54"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:32:47.466724 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:32:47.466706    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/45450e7a-2b62-4417-acca-d370251f6d54-istio-podinfo" pod "45450e7a-2b62-4417-acca-d370251f6d54" (UID: "45450e7a-2b62-4417-acca-d370251f6d54"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 01:32:47.566890 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:32:47.566857    4964 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/45450e7a-2b62-4417-acca-d370251f6d54-istio-envoy\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:32:47.566890 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:32:47.566894    4964 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/45450e7a-2b62-4417-acca-d370251f6d54-istio-token\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:32:47.567079 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:32:47.566912    4964 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lpn8j\" (UniqueName: \"kubernetes.io/projected/45450e7a-2b62-4417-acca-d370251f6d54-kube-api-access-lpn8j\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:32:47.567079 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:32:47.566927    4964 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/45450e7a-2b62-4417-acca-d370251f6d54-istio-podinfo\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:32:47.948959 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:32:47.948926    4964 generic.go:356] "Generic (PLEG): container finished" podID="45450e7a-2b62-4417-acca-d370251f6d54" containerID="8c9a9da3abc3f800d88d817b91d0d19a6e8c67d550b7bf80bc28279b921e16f0" exitCode=0
Oct 31 01:32:47.949153 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:32:47.948969    4964 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-7cec02-openshift-default-6fc456b49-jn9wm"
Oct 31 01:32:47.949153 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:32:47.949019    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-7cec02-openshift-default-6fc456b49-jn9wm" event={"ID":"45450e7a-2b62-4417-acca-d370251f6d54","Type":"ContainerDied","Data":"8c9a9da3abc3f800d88d817b91d0d19a6e8c67d550b7bf80bc28279b921e16f0"}
Oct 31 01:32:47.949153 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:32:47.949075    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-7cec02-openshift-default-6fc456b49-jn9wm" event={"ID":"45450e7a-2b62-4417-acca-d370251f6d54","Type":"ContainerDied","Data":"2675968707a4e50660ef1f8ced15cb1dd0dbfdd6f20dc619fa426b24dd4a1b53"}
Oct 31 01:32:47.949153 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:32:47.949103    4964 scope.go:122] "RemoveContainer" containerID="8c9a9da3abc3f800d88d817b91d0d19a6e8c67d550b7bf80bc28279b921e16f0"
Oct 31 01:32:47.960190 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:32:47.960173    4964 scope.go:122] "RemoveContainer" containerID="362bac2a8bd4ba033fd5150f6a3973611b86e493d849686055f98cf44a5b996a"
Oct 31 01:32:47.969714 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:32:47.969685    4964 scope.go:122] "RemoveContainer" containerID="8c9a9da3abc3f800d88d817b91d0d19a6e8c67d550b7bf80bc28279b921e16f0"
Oct 31 01:32:47.970015 ip-10-0-133-197 kubenswrapper[4964]: E1031 01:32:47.969991    4964 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8c9a9da3abc3f800d88d817b91d0d19a6e8c67d550b7bf80bc28279b921e16f0\": container with ID starting with 8c9a9da3abc3f800d88d817b91d0d19a6e8c67d550b7bf80bc28279b921e16f0 not found: ID does not exist" containerID="8c9a9da3abc3f800d88d817b91d0d19a6e8c67d550b7bf80bc28279b921e16f0"
Oct 31 01:32:47.970153 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:32:47.970023    4964 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8c9a9da3abc3f800d88d817b91d0d19a6e8c67d550b7bf80bc28279b921e16f0"} err="failed to get container status \"8c9a9da3abc3f800d88d817b91d0d19a6e8c67d550b7bf80bc28279b921e16f0\": rpc error: code = NotFound desc = could not find container \"8c9a9da3abc3f800d88d817b91d0d19a6e8c67d550b7bf80bc28279b921e16f0\": container with ID starting with 8c9a9da3abc3f800d88d817b91d0d19a6e8c67d550b7bf80bc28279b921e16f0 not found: ID does not exist"
Oct 31 01:32:47.970153 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:32:47.970044    4964 scope.go:122] "RemoveContainer" containerID="362bac2a8bd4ba033fd5150f6a3973611b86e493d849686055f98cf44a5b996a"
Oct 31 01:32:47.970392 ip-10-0-133-197 kubenswrapper[4964]: E1031 01:32:47.970301    4964 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"362bac2a8bd4ba033fd5150f6a3973611b86e493d849686055f98cf44a5b996a\": container with ID starting with 362bac2a8bd4ba033fd5150f6a3973611b86e493d849686055f98cf44a5b996a not found: ID does not exist" containerID="362bac2a8bd4ba033fd5150f6a3973611b86e493d849686055f98cf44a5b996a"
Oct 31 01:32:47.970392 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:32:47.970361    4964 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"362bac2a8bd4ba033fd5150f6a3973611b86e493d849686055f98cf44a5b996a"} err="failed to get container status \"362bac2a8bd4ba033fd5150f6a3973611b86e493d849686055f98cf44a5b996a\": rpc error: code = NotFound desc = could not find container \"362bac2a8bd4ba033fd5150f6a3973611b86e493d849686055f98cf44a5b996a\": container with ID starting with 362bac2a8bd4ba033fd5150f6a3973611b86e493d849686055f98cf44a5b996a not found: ID does not exist"
Oct 31 01:32:48.115297 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:32:48.115209    4964 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-7cec02-openshift-default-6fc456b49-jn9wm"]
Oct 31 01:32:48.162951 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:32:48.162919    4964 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-7cec02-openshift-default-6fc456b49-jn9wm"]
Oct 31 01:32:49.777275 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:32:49.777230    4964 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="45450e7a-2b62-4417-acca-d370251f6d54" path="/var/lib/kubelet/pods/45450e7a-2b62-4417-acca-d370251f6d54/volumes"
Oct 31 01:32:50.576768 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:32:50.576731    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:32:54.461928 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:32:54.461897    4964 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-e7eea9-7699bc58bf-b8snr"]
Oct 31 01:32:54.573199 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:32:54.573164    4964 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-e7eea9-6464f8bcd7-wdd67"]
Oct 31 01:32:54.883620 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:32:54.883548    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:32:55.125812 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:32:55.125775    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:32:55.510506 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:32:55.510453    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:32:59.462435 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:32:59.462367    4964 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-e7eea9-7699bc58bf-b8snr" podUID="fbf50441-0853-4ecf-99ea-38a47e9d5677" containerName="payload-processing" containerID="cri-o://790cd26a5733b2b9f30b0b221059c2786bd948b7717a861fc45ec55b164ffee7" gracePeriod=25
Oct 31 01:32:59.496844 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:32:59.496814    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:32:59.574135 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:32:59.574080    4964 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-e7eea9-6464f8bcd7-wdd67" podUID="f1770ebf-036d-4645-b3f2-c1f14574438f" containerName="payload-pre-processing" containerID="cri-o://75b89ffa93193cdda1bfd503acfef5a3c5dba50f75e0a7b9a6c9b36f6a0d05ca" gracePeriod=25
Oct 31 01:32:59.765833 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:32:59.765804    4964 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-e7eea9-7699bc58bf-b8snr"
Oct 31 01:32:59.831636 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:32:59.831594    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fbf50441-0853-4ecf-99ea-38a47e9d5677-kube-api-access-55gqg\" (UniqueName: \"kubernetes.io/projected/fbf50441-0853-4ecf-99ea-38a47e9d5677-kube-api-access-55gqg\") pod \"fbf50441-0853-4ecf-99ea-38a47e9d5677\" (UID: \"fbf50441-0853-4ecf-99ea-38a47e9d5677\") "
Oct 31 01:32:59.831826 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:32:59.831756    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fbf50441-0853-4ecf-99ea-38a47e9d5677-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fbf50441-0853-4ecf-99ea-38a47e9d5677-plugins-config-volume\") pod \"fbf50441-0853-4ecf-99ea-38a47e9d5677\" (UID: \"fbf50441-0853-4ecf-99ea-38a47e9d5677\") "
Oct 31 01:32:59.832115 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:32:59.832093    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fbf50441-0853-4ecf-99ea-38a47e9d5677-plugins-config-volume" pod "fbf50441-0853-4ecf-99ea-38a47e9d5677" (UID: "fbf50441-0853-4ecf-99ea-38a47e9d5677"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:32:59.832613 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:32:59.832477    4964 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fbf50441-0853-4ecf-99ea-38a47e9d5677-plugins-config-volume\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:32:59.848851 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:32:59.848809    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fbf50441-0853-4ecf-99ea-38a47e9d5677-kube-api-access-55gqg" pod "fbf50441-0853-4ecf-99ea-38a47e9d5677" (UID: "fbf50441-0853-4ecf-99ea-38a47e9d5677"). InnerVolumeSpecName "kube-api-access-55gqg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:32:59.883401 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:32:59.883378    4964 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-e7eea9-6464f8bcd7-wdd67"
Oct 31 01:32:59.934442 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:32:59.934408    4964 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-55gqg\" (UniqueName: \"kubernetes.io/projected/fbf50441-0853-4ecf-99ea-38a47e9d5677-kube-api-access-55gqg\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:33:00.035057 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:33:00.034985    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f1770ebf-036d-4645-b3f2-c1f14574438f-kube-api-access-2ds2m\" (UniqueName: \"kubernetes.io/projected/f1770ebf-036d-4645-b3f2-c1f14574438f-kube-api-access-2ds2m\") pod \"f1770ebf-036d-4645-b3f2-c1f14574438f\" (UID: \"f1770ebf-036d-4645-b3f2-c1f14574438f\") "
Oct 31 01:33:00.035199 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:33:00.035188    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f1770ebf-036d-4645-b3f2-c1f14574438f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f1770ebf-036d-4645-b3f2-c1f14574438f-plugins-config-volume\") pod \"f1770ebf-036d-4645-b3f2-c1f14574438f\" (UID: \"f1770ebf-036d-4645-b3f2-c1f14574438f\") "
Oct 31 01:33:00.036286 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:33:00.036249    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f1770ebf-036d-4645-b3f2-c1f14574438f-plugins-config-volume" pod "f1770ebf-036d-4645-b3f2-c1f14574438f" (UID: "f1770ebf-036d-4645-b3f2-c1f14574438f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:33:00.038764 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:33:00.038737    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f1770ebf-036d-4645-b3f2-c1f14574438f-kube-api-access-2ds2m" pod "f1770ebf-036d-4645-b3f2-c1f14574438f" (UID: "f1770ebf-036d-4645-b3f2-c1f14574438f"). InnerVolumeSpecName "kube-api-access-2ds2m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:33:00.044595 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:33:00.044571    4964 generic.go:356] "Generic (PLEG): container finished" podID="fbf50441-0853-4ecf-99ea-38a47e9d5677" containerID="790cd26a5733b2b9f30b0b221059c2786bd948b7717a861fc45ec55b164ffee7" exitCode=0
Oct 31 01:33:00.044710 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:33:00.044696    4964 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-e7eea9-7699bc58bf-b8snr"
Oct 31 01:33:00.045220 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:33:00.045199    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-e7eea9-7699bc58bf-b8snr" event={"ID":"fbf50441-0853-4ecf-99ea-38a47e9d5677","Type":"ContainerDied","Data":"790cd26a5733b2b9f30b0b221059c2786bd948b7717a861fc45ec55b164ffee7"}
Oct 31 01:33:00.045508 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:33:00.045488    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-e7eea9-7699bc58bf-b8snr" event={"ID":"fbf50441-0853-4ecf-99ea-38a47e9d5677","Type":"ContainerDied","Data":"7f872630e42b957495e3769566b657beb4e3c54106318da91b4cfaf33efd5bf5"}
Oct 31 01:33:00.045648 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:33:00.045634    4964 scope.go:122] "RemoveContainer" containerID="790cd26a5733b2b9f30b0b221059c2786bd948b7717a861fc45ec55b164ffee7"
Oct 31 01:33:00.048390 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:33:00.047601    4964 generic.go:356] "Generic (PLEG): container finished" podID="f1770ebf-036d-4645-b3f2-c1f14574438f" containerID="75b89ffa93193cdda1bfd503acfef5a3c5dba50f75e0a7b9a6c9b36f6a0d05ca" exitCode=0
Oct 31 01:33:00.048390 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:33:00.047854    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-e7eea9-6464f8bcd7-wdd67" event={"ID":"f1770ebf-036d-4645-b3f2-c1f14574438f","Type":"ContainerDied","Data":"75b89ffa93193cdda1bfd503acfef5a3c5dba50f75e0a7b9a6c9b36f6a0d05ca"}
Oct 31 01:33:00.048390 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:33:00.047957    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-e7eea9-6464f8bcd7-wdd67" event={"ID":"f1770ebf-036d-4645-b3f2-c1f14574438f","Type":"ContainerDied","Data":"d4d10aa2ba02745117656a513ec3558c8a9701a63c73df6a80fe036193e84baa"}
Oct 31 01:33:00.048390 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:33:00.048028    4964 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-e7eea9-6464f8bcd7-wdd67"
Oct 31 01:33:00.063678 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:33:00.063655    4964 scope.go:122] "RemoveContainer" containerID="790cd26a5733b2b9f30b0b221059c2786bd948b7717a861fc45ec55b164ffee7"
Oct 31 01:33:00.063993 ip-10-0-133-197 kubenswrapper[4964]: E1031 01:33:00.063977    4964 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"790cd26a5733b2b9f30b0b221059c2786bd948b7717a861fc45ec55b164ffee7\": container with ID starting with 790cd26a5733b2b9f30b0b221059c2786bd948b7717a861fc45ec55b164ffee7 not found: ID does not exist" containerID="790cd26a5733b2b9f30b0b221059c2786bd948b7717a861fc45ec55b164ffee7"
Oct 31 01:33:00.064056 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:33:00.064001    4964 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"790cd26a5733b2b9f30b0b221059c2786bd948b7717a861fc45ec55b164ffee7"} err="failed to get container status \"790cd26a5733b2b9f30b0b221059c2786bd948b7717a861fc45ec55b164ffee7\": rpc error: code = NotFound desc = could not find container \"790cd26a5733b2b9f30b0b221059c2786bd948b7717a861fc45ec55b164ffee7\": container with ID starting with 790cd26a5733b2b9f30b0b221059c2786bd948b7717a861fc45ec55b164ffee7 not found: ID does not exist"
Oct 31 01:33:00.064056 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:33:00.064017    4964 scope.go:122] "RemoveContainer" containerID="75b89ffa93193cdda1bfd503acfef5a3c5dba50f75e0a7b9a6c9b36f6a0d05ca"
Oct 31 01:33:00.073306 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:33:00.073279    4964 scope.go:122] "RemoveContainer" containerID="75b89ffa93193cdda1bfd503acfef5a3c5dba50f75e0a7b9a6c9b36f6a0d05ca"
Oct 31 01:33:00.073650 ip-10-0-133-197 kubenswrapper[4964]: E1031 01:33:00.073630    4964 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"75b89ffa93193cdda1bfd503acfef5a3c5dba50f75e0a7b9a6c9b36f6a0d05ca\": container with ID starting with 75b89ffa93193cdda1bfd503acfef5a3c5dba50f75e0a7b9a6c9b36f6a0d05ca not found: ID does not exist" containerID="75b89ffa93193cdda1bfd503acfef5a3c5dba50f75e0a7b9a6c9b36f6a0d05ca"
Oct 31 01:33:00.073728 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:33:00.073655    4964 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"75b89ffa93193cdda1bfd503acfef5a3c5dba50f75e0a7b9a6c9b36f6a0d05ca"} err="failed to get container status \"75b89ffa93193cdda1bfd503acfef5a3c5dba50f75e0a7b9a6c9b36f6a0d05ca\": rpc error: code = NotFound desc = could not find container \"75b89ffa93193cdda1bfd503acfef5a3c5dba50f75e0a7b9a6c9b36f6a0d05ca\": container with ID starting with 75b89ffa93193cdda1bfd503acfef5a3c5dba50f75e0a7b9a6c9b36f6a0d05ca not found: ID does not exist"
Oct 31 01:33:00.137288 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:33:00.137256    4964 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f1770ebf-036d-4645-b3f2-c1f14574438f-plugins-config-volume\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:33:00.137288 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:33:00.137289    4964 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2ds2m\" (UniqueName: \"kubernetes.io/projected/f1770ebf-036d-4645-b3f2-c1f14574438f-kube-api-access-2ds2m\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:33:00.209480 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:33:00.209385    4964 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-e7eea9-7699bc58bf-b8snr"]
Oct 31 01:33:00.212274 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:33:00.212252    4964 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-e7eea9-7699bc58bf-b8snr"]
Oct 31 01:33:00.311992 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:33:00.311900    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:33:00.319198 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:33:00.319170    4964 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-e7eea9-6464f8bcd7-wdd67"]
Oct 31 01:33:00.323801 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:33:00.323770    4964 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-e7eea9-6464f8bcd7-wdd67"]
Oct 31 01:33:01.777839 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:33:01.777809    4964 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f1770ebf-036d-4645-b3f2-c1f14574438f" path="/var/lib/kubelet/pods/f1770ebf-036d-4645-b3f2-c1f14574438f/volumes"
Oct 31 01:33:01.778239 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:33:01.778223    4964 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fbf50441-0853-4ecf-99ea-38a47e9d5677" path="/var/lib/kubelet/pods/fbf50441-0853-4ecf-99ea-38a47e9d5677/volumes"
Oct 31 01:33:05.199775 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:33:05.199737    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:33:07.884466 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:33:07.884428    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:33:09.759889 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:33:09.759856    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:33:10.666045 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:33:10.666006    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:33:11.241445 ip-10-0-133-197 kubenswrapper[4964]: E1031 01:33:11.241409    4964 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q56x5" containerName="extractor"
Oct 31 01:33:11.327352 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:33:11.327305    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:33:11.591193 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:33:11.591113    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:33:11.776962 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:33:11.776931    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:33:12.266747 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:33:12.266714    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:33:16.699364 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:33:16.699239    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:33:18.276916 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:33:18.276885    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:33:18.508164 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:33:18.508128    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:33:18.696811 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:33:18.696722    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:33:20.014353 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:33:20.014272    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:33:20.529526 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:33:20.529495    4964 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-42c96d-65764fbd8c-84plq"]
Oct 31 01:33:20.819435 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:33:20.819349    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:33:21.046670 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:33:21.046632    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:33:21.374077 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:33:21.374039    4964 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-tt9c2"]
Oct 31 01:33:21.374304 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:33:21.374280    4964 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-tt9c2" podUID="2a53cf91-69f9-4c47-8271-79736ce35af9" containerName="istio-proxy" containerID="cri-o://d43ea5f023a2f0d664642c01246e01a25f303a5ef1d04b70f6c9e573d24c1320" gracePeriod=30
Oct 31 01:33:21.377715 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:33:21.377690    4964 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-flwkb"]
Oct 31 01:33:21.377913 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:33:21.377891    4964 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-flwkb" podUID="da53c8d7-cc60-45cd-ab19-365d486484b1" containerName="istio-proxy" containerID="cri-o://bdcf04a8c954cdaf6e316d4dc62721e3b63f9762d805650794f18632b675f279" gracePeriod=30
Oct 31 01:33:21.700422 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:33:21.698678    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:33:25.535190 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:33:25.535124    4964 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-42c96d-65764fbd8c-84plq" podUID="065217f5-7fda-424b-a2e7-fed363588c32" containerName="payload-pre-processing" containerID="cri-o://426c3d564ffe278eee6386373d24d17b3d168df9b492f545e353a1d68a07d042" gracePeriod=25
Oct 31 01:33:25.827479 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:33:25.827450    4964 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-42c96d-65764fbd8c-84plq"
Oct 31 01:33:26.004781 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:33:26.004752    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/065217f5-7fda-424b-a2e7-fed363588c32-kube-api-access-4zj4c\" (UniqueName: \"kubernetes.io/projected/065217f5-7fda-424b-a2e7-fed363588c32-kube-api-access-4zj4c\") pod \"065217f5-7fda-424b-a2e7-fed363588c32\" (UID: \"065217f5-7fda-424b-a2e7-fed363588c32\") "
Oct 31 01:33:26.004976 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:33:26.004886    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/065217f5-7fda-424b-a2e7-fed363588c32-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/065217f5-7fda-424b-a2e7-fed363588c32-plugins-config-volume\") pod \"065217f5-7fda-424b-a2e7-fed363588c32\" (UID: \"065217f5-7fda-424b-a2e7-fed363588c32\") "
Oct 31 01:33:26.005242 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:33:26.005219    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/065217f5-7fda-424b-a2e7-fed363588c32-plugins-config-volume" pod "065217f5-7fda-424b-a2e7-fed363588c32" (UID: "065217f5-7fda-424b-a2e7-fed363588c32"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:33:26.005406 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:33:26.005392    4964 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/065217f5-7fda-424b-a2e7-fed363588c32-plugins-config-volume\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:33:26.007028 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:33:26.007008    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/065217f5-7fda-424b-a2e7-fed363588c32-kube-api-access-4zj4c" pod "065217f5-7fda-424b-a2e7-fed363588c32" (UID: "065217f5-7fda-424b-a2e7-fed363588c32"). InnerVolumeSpecName "kube-api-access-4zj4c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:33:26.106666 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:33:26.106599    4964 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4zj4c\" (UniqueName: \"kubernetes.io/projected/065217f5-7fda-424b-a2e7-fed363588c32-kube-api-access-4zj4c\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:33:26.253906 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:33:26.253876    4964 generic.go:356] "Generic (PLEG): container finished" podID="065217f5-7fda-424b-a2e7-fed363588c32" containerID="426c3d564ffe278eee6386373d24d17b3d168df9b492f545e353a1d68a07d042" exitCode=0
Oct 31 01:33:26.254073 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:33:26.253931    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-42c96d-65764fbd8c-84plq" event={"ID":"065217f5-7fda-424b-a2e7-fed363588c32","Type":"ContainerDied","Data":"426c3d564ffe278eee6386373d24d17b3d168df9b492f545e353a1d68a07d042"}
Oct 31 01:33:26.254073 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:33:26.253940    4964 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-42c96d-65764fbd8c-84plq"
Oct 31 01:33:26.254073 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:33:26.253971    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-42c96d-65764fbd8c-84plq" event={"ID":"065217f5-7fda-424b-a2e7-fed363588c32","Type":"ContainerDied","Data":"5a6d7191b21c05ffab402d134bcaa5fef64d724daad9a1e0399c552c0a63ab76"}
Oct 31 01:33:26.254073 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:33:26.253994    4964 scope.go:122] "RemoveContainer" containerID="426c3d564ffe278eee6386373d24d17b3d168df9b492f545e353a1d68a07d042"
Oct 31 01:33:26.264403 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:33:26.264373    4964 scope.go:122] "RemoveContainer" containerID="426c3d564ffe278eee6386373d24d17b3d168df9b492f545e353a1d68a07d042"
Oct 31 01:33:26.264799 ip-10-0-133-197 kubenswrapper[4964]: E1031 01:33:26.264689    4964 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"426c3d564ffe278eee6386373d24d17b3d168df9b492f545e353a1d68a07d042\": container with ID starting with 426c3d564ffe278eee6386373d24d17b3d168df9b492f545e353a1d68a07d042 not found: ID does not exist" containerID="426c3d564ffe278eee6386373d24d17b3d168df9b492f545e353a1d68a07d042"
Oct 31 01:33:26.264799 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:33:26.264727    4964 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"426c3d564ffe278eee6386373d24d17b3d168df9b492f545e353a1d68a07d042"} err="failed to get container status \"426c3d564ffe278eee6386373d24d17b3d168df9b492f545e353a1d68a07d042\": rpc error: code = NotFound desc = could not find container \"426c3d564ffe278eee6386373d24d17b3d168df9b492f545e353a1d68a07d042\": container with ID starting with 426c3d564ffe278eee6386373d24d17b3d168df9b492f545e353a1d68a07d042 not found: ID does not exist"
Oct 31 01:33:26.419528 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:33:26.419496    4964 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-42c96d-65764fbd8c-84plq"]
Oct 31 01:33:26.466928 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:33:26.466900    4964 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-42c96d-65764fbd8c-84plq"]
Oct 31 01:33:26.697132 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:33:26.697105    4964 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-tt9c2"
Oct 31 01:33:26.726432 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:33:26.726409    4964 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-flwkb"
Oct 31 01:33:26.767410 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:33:26.767373    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:33:26.818011 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:33:26.817977    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2a53cf91-69f9-4c47-8271-79736ce35af9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2a53cf91-69f9-4c47-8271-79736ce35af9-istio-envoy\") pod \"2a53cf91-69f9-4c47-8271-79736ce35af9\" (UID: \"2a53cf91-69f9-4c47-8271-79736ce35af9\") "
Oct 31 01:33:26.818212 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:33:26.818119    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2a53cf91-69f9-4c47-8271-79736ce35af9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2a53cf91-69f9-4c47-8271-79736ce35af9-istio-podinfo\") pod \"2a53cf91-69f9-4c47-8271-79736ce35af9\" (UID: \"2a53cf91-69f9-4c47-8271-79736ce35af9\") "
Oct 31 01:33:26.818212 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:33:26.818162    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2a53cf91-69f9-4c47-8271-79736ce35af9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2a53cf91-69f9-4c47-8271-79736ce35af9-istiod-ca-cert\") pod \"2a53cf91-69f9-4c47-8271-79736ce35af9\" (UID: \"2a53cf91-69f9-4c47-8271-79736ce35af9\") "
Oct 31 01:33:26.818376 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:33:26.818216    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2a53cf91-69f9-4c47-8271-79736ce35af9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a53cf91-69f9-4c47-8271-79736ce35af9-workload-socket\") pod \"2a53cf91-69f9-4c47-8271-79736ce35af9\" (UID: \"2a53cf91-69f9-4c47-8271-79736ce35af9\") "
Oct 31 01:33:26.818376 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:33:26.818339    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2a53cf91-69f9-4c47-8271-79736ce35af9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2a53cf91-69f9-4c47-8271-79736ce35af9-workload-certs\") pod \"2a53cf91-69f9-4c47-8271-79736ce35af9\" (UID: \"2a53cf91-69f9-4c47-8271-79736ce35af9\") "
Oct 31 01:33:26.818496 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:33:26.818409    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2a53cf91-69f9-4c47-8271-79736ce35af9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a53cf91-69f9-4c47-8271-79736ce35af9-credential-socket\") pod \"2a53cf91-69f9-4c47-8271-79736ce35af9\" (UID: \"2a53cf91-69f9-4c47-8271-79736ce35af9\") "
Oct 31 01:33:26.818496 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:33:26.818448    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2a53cf91-69f9-4c47-8271-79736ce35af9-kube-api-access-mqg5w\" (UniqueName: \"kubernetes.io/projected/2a53cf91-69f9-4c47-8271-79736ce35af9-kube-api-access-mqg5w\") pod \"2a53cf91-69f9-4c47-8271-79736ce35af9\" (UID: \"2a53cf91-69f9-4c47-8271-79736ce35af9\") "
Oct 31 01:33:26.818599 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:33:26.818579    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2a53cf91-69f9-4c47-8271-79736ce35af9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2a53cf91-69f9-4c47-8271-79736ce35af9-istio-data\") pod \"2a53cf91-69f9-4c47-8271-79736ce35af9\" (UID: \"2a53cf91-69f9-4c47-8271-79736ce35af9\") "
Oct 31 01:33:26.818650 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:33:26.818614    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2a53cf91-69f9-4c47-8271-79736ce35af9-istio-token\" (UniqueName: \"kubernetes.io/projected/2a53cf91-69f9-4c47-8271-79736ce35af9-istio-token\") pod \"2a53cf91-69f9-4c47-8271-79736ce35af9\" (UID: \"2a53cf91-69f9-4c47-8271-79736ce35af9\") "
Oct 31 01:33:26.818726 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:33:26.818694    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a53cf91-69f9-4c47-8271-79736ce35af9-credential-socket" pod "2a53cf91-69f9-4c47-8271-79736ce35af9" (UID: "2a53cf91-69f9-4c47-8271-79736ce35af9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:33:26.818781 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:33:26.818737    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a53cf91-69f9-4c47-8271-79736ce35af9-workload-certs" pod "2a53cf91-69f9-4c47-8271-79736ce35af9" (UID: "2a53cf91-69f9-4c47-8271-79736ce35af9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:33:26.819032 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:33:26.819010    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a53cf91-69f9-4c47-8271-79736ce35af9-workload-socket" pod "2a53cf91-69f9-4c47-8271-79736ce35af9" (UID: "2a53cf91-69f9-4c47-8271-79736ce35af9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:33:26.819263 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:33:26.819239    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a53cf91-69f9-4c47-8271-79736ce35af9-istio-data" pod "2a53cf91-69f9-4c47-8271-79736ce35af9" (UID: "2a53cf91-69f9-4c47-8271-79736ce35af9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:33:26.821708 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:33:26.821662    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2a53cf91-69f9-4c47-8271-79736ce35af9-istiod-ca-cert" pod "2a53cf91-69f9-4c47-8271-79736ce35af9" (UID: "2a53cf91-69f9-4c47-8271-79736ce35af9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:33:26.823160 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:33:26.823132    4964 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2a53cf91-69f9-4c47-8271-79736ce35af9-istiod-ca-cert\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:33:26.823272 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:33:26.823164    4964 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a53cf91-69f9-4c47-8271-79736ce35af9-workload-socket\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:33:26.823272 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:33:26.823179    4964 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2a53cf91-69f9-4c47-8271-79736ce35af9-workload-certs\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:33:26.823272 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:33:26.823193    4964 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a53cf91-69f9-4c47-8271-79736ce35af9-credential-socket\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:33:26.823272 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:33:26.823206    4964 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2a53cf91-69f9-4c47-8271-79736ce35af9-istio-data\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:33:26.823621 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:33:26.823501    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a53cf91-69f9-4c47-8271-79736ce35af9-kube-api-access-mqg5w" pod "2a53cf91-69f9-4c47-8271-79736ce35af9" (UID: "2a53cf91-69f9-4c47-8271-79736ce35af9"). InnerVolumeSpecName "kube-api-access-mqg5w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:33:26.826531 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:33:26.826500    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a53cf91-69f9-4c47-8271-79736ce35af9-istio-token" pod "2a53cf91-69f9-4c47-8271-79736ce35af9" (UID: "2a53cf91-69f9-4c47-8271-79736ce35af9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:33:26.826647 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:33:26.826581    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2a53cf91-69f9-4c47-8271-79736ce35af9-istio-podinfo" pod "2a53cf91-69f9-4c47-8271-79736ce35af9" (UID: "2a53cf91-69f9-4c47-8271-79736ce35af9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 01:33:26.826757 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:33:26.826736    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a53cf91-69f9-4c47-8271-79736ce35af9-istio-envoy" pod "2a53cf91-69f9-4c47-8271-79736ce35af9" (UID: "2a53cf91-69f9-4c47-8271-79736ce35af9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:33:26.923837 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:33:26.923758    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/da53c8d7-cc60-45cd-ab19-365d486484b1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/da53c8d7-cc60-45cd-ab19-365d486484b1-istio-envoy\") pod \"da53c8d7-cc60-45cd-ab19-365d486484b1\" (UID: \"da53c8d7-cc60-45cd-ab19-365d486484b1\") "
Oct 31 01:33:26.923837 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:33:26.923812    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/da53c8d7-cc60-45cd-ab19-365d486484b1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/da53c8d7-cc60-45cd-ab19-365d486484b1-credential-socket\") pod \"da53c8d7-cc60-45cd-ab19-365d486484b1\" (UID: \"da53c8d7-cc60-45cd-ab19-365d486484b1\") "
Oct 31 01:33:26.924056 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:33:26.923848    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/da53c8d7-cc60-45cd-ab19-365d486484b1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/da53c8d7-cc60-45cd-ab19-365d486484b1-istiod-ca-cert\") pod \"da53c8d7-cc60-45cd-ab19-365d486484b1\" (UID: \"da53c8d7-cc60-45cd-ab19-365d486484b1\") "
Oct 31 01:33:26.924056 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:33:26.923899    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/da53c8d7-cc60-45cd-ab19-365d486484b1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/da53c8d7-cc60-45cd-ab19-365d486484b1-workload-socket\") pod \"da53c8d7-cc60-45cd-ab19-365d486484b1\" (UID: \"da53c8d7-cc60-45cd-ab19-365d486484b1\") "
Oct 31 01:33:26.924056 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:33:26.923994    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/da53c8d7-cc60-45cd-ab19-365d486484b1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/da53c8d7-cc60-45cd-ab19-365d486484b1-workload-certs\") pod \"da53c8d7-cc60-45cd-ab19-365d486484b1\" (UID: \"da53c8d7-cc60-45cd-ab19-365d486484b1\") "
Oct 31 01:33:26.924056 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:33:26.924030    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/da53c8d7-cc60-45cd-ab19-365d486484b1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/da53c8d7-cc60-45cd-ab19-365d486484b1-istio-data\") pod \"da53c8d7-cc60-45cd-ab19-365d486484b1\" (UID: \"da53c8d7-cc60-45cd-ab19-365d486484b1\") "
Oct 31 01:33:26.924271 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:33:26.924065    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/da53c8d7-cc60-45cd-ab19-365d486484b1-istio-token\" (UniqueName: \"kubernetes.io/projected/da53c8d7-cc60-45cd-ab19-365d486484b1-istio-token\") pod \"da53c8d7-cc60-45cd-ab19-365d486484b1\" (UID: \"da53c8d7-cc60-45cd-ab19-365d486484b1\") "
Oct 31 01:33:26.924271 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:33:26.924127    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/da53c8d7-cc60-45cd-ab19-365d486484b1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/da53c8d7-cc60-45cd-ab19-365d486484b1-istio-podinfo\") pod \"da53c8d7-cc60-45cd-ab19-365d486484b1\" (UID: \"da53c8d7-cc60-45cd-ab19-365d486484b1\") "
Oct 31 01:33:26.924271 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:33:26.924175    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da53c8d7-cc60-45cd-ab19-365d486484b1-credential-socket" pod "da53c8d7-cc60-45cd-ab19-365d486484b1" (UID: "da53c8d7-cc60-45cd-ab19-365d486484b1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:33:26.924271 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:33:26.924188    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/da53c8d7-cc60-45cd-ab19-365d486484b1-kube-api-access-nhlrq\" (UniqueName: \"kubernetes.io/projected/da53c8d7-cc60-45cd-ab19-365d486484b1-kube-api-access-nhlrq\") pod \"da53c8d7-cc60-45cd-ab19-365d486484b1\" (UID: \"da53c8d7-cc60-45cd-ab19-365d486484b1\") "
Oct 31 01:33:26.924271 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:33:26.924225    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da53c8d7-cc60-45cd-ab19-365d486484b1-workload-socket" pod "da53c8d7-cc60-45cd-ab19-365d486484b1" (UID: "da53c8d7-cc60-45cd-ab19-365d486484b1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:33:26.924565 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:33:26.924311    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/da53c8d7-cc60-45cd-ab19-365d486484b1-istiod-ca-cert" pod "da53c8d7-cc60-45cd-ab19-365d486484b1" (UID: "da53c8d7-cc60-45cd-ab19-365d486484b1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:33:26.924565 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:33:26.924356    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da53c8d7-cc60-45cd-ab19-365d486484b1-workload-certs" pod "da53c8d7-cc60-45cd-ab19-365d486484b1" (UID: "da53c8d7-cc60-45cd-ab19-365d486484b1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:33:26.924833 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:33:26.924768    4964 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2a53cf91-69f9-4c47-8271-79736ce35af9-istio-podinfo\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:33:26.924833 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:33:26.924799    4964 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/da53c8d7-cc60-45cd-ab19-365d486484b1-credential-socket\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:33:26.924833 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:33:26.924815    4964 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/da53c8d7-cc60-45cd-ab19-365d486484b1-istiod-ca-cert\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:33:26.924833 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:33:26.924829    4964 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/da53c8d7-cc60-45cd-ab19-365d486484b1-workload-socket\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:33:26.925266 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:33:26.924842    4964 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/da53c8d7-cc60-45cd-ab19-365d486484b1-workload-certs\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:33:26.925266 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:33:26.924856    4964 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mqg5w\" (UniqueName: \"kubernetes.io/projected/2a53cf91-69f9-4c47-8271-79736ce35af9-kube-api-access-mqg5w\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:33:26.925266 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:33:26.924870    4964 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2a53cf91-69f9-4c47-8271-79736ce35af9-istio-token\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:33:26.925266 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:33:26.924884    4964 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2a53cf91-69f9-4c47-8271-79736ce35af9-istio-envoy\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:33:26.926386 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:33:26.926360    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da53c8d7-cc60-45cd-ab19-365d486484b1-istio-data" pod "da53c8d7-cc60-45cd-ab19-365d486484b1" (UID: "da53c8d7-cc60-45cd-ab19-365d486484b1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:33:26.926648 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:33:26.926631    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da53c8d7-cc60-45cd-ab19-365d486484b1-istio-token" pod "da53c8d7-cc60-45cd-ab19-365d486484b1" (UID: "da53c8d7-cc60-45cd-ab19-365d486484b1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:33:26.927170 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:33:26.926908    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da53c8d7-cc60-45cd-ab19-365d486484b1-istio-envoy" pod "da53c8d7-cc60-45cd-ab19-365d486484b1" (UID: "da53c8d7-cc60-45cd-ab19-365d486484b1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:33:26.927170 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:33:26.926970    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da53c8d7-cc60-45cd-ab19-365d486484b1-kube-api-access-nhlrq" pod "da53c8d7-cc60-45cd-ab19-365d486484b1" (UID: "da53c8d7-cc60-45cd-ab19-365d486484b1"). InnerVolumeSpecName "kube-api-access-nhlrq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:33:26.927394 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:33:26.927020    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/da53c8d7-cc60-45cd-ab19-365d486484b1-istio-podinfo" pod "da53c8d7-cc60-45cd-ab19-365d486484b1" (UID: "da53c8d7-cc60-45cd-ab19-365d486484b1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 01:33:27.025967 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:33:27.025934    4964 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/da53c8d7-cc60-45cd-ab19-365d486484b1-istio-envoy\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:33:27.025967 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:33:27.025961    4964 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/da53c8d7-cc60-45cd-ab19-365d486484b1-istio-data\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:33:27.026168 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:33:27.025973    4964 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/da53c8d7-cc60-45cd-ab19-365d486484b1-istio-token\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:33:27.026168 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:33:27.025988    4964 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/da53c8d7-cc60-45cd-ab19-365d486484b1-istio-podinfo\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:33:27.026168 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:33:27.025999    4964 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nhlrq\" (UniqueName: \"kubernetes.io/projected/da53c8d7-cc60-45cd-ab19-365d486484b1-kube-api-access-nhlrq\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:33:27.261976 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:33:27.261940    4964 generic.go:356] "Generic (PLEG): container finished" podID="2a53cf91-69f9-4c47-8271-79736ce35af9" containerID="d43ea5f023a2f0d664642c01246e01a25f303a5ef1d04b70f6c9e573d24c1320" exitCode=0
Oct 31 01:33:27.262152 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:33:27.261989    4964 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-tt9c2"
Oct 31 01:33:27.262152 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:33:27.262026    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-tt9c2" event={"ID":"2a53cf91-69f9-4c47-8271-79736ce35af9","Type":"ContainerDied","Data":"d43ea5f023a2f0d664642c01246e01a25f303a5ef1d04b70f6c9e573d24c1320"}
Oct 31 01:33:27.262152 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:33:27.262074    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-tt9c2" event={"ID":"2a53cf91-69f9-4c47-8271-79736ce35af9","Type":"ContainerDied","Data":"2ceab8619f6f8e164630736e01b9e2348332566d903aa4d45fb00af7acc1ab90"}
Oct 31 01:33:27.262152 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:33:27.262101    4964 scope.go:122] "RemoveContainer" containerID="d43ea5f023a2f0d664642c01246e01a25f303a5ef1d04b70f6c9e573d24c1320"
Oct 31 01:33:27.263801 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:33:27.263774    4964 generic.go:356] "Generic (PLEG): container finished" podID="da53c8d7-cc60-45cd-ab19-365d486484b1" containerID="bdcf04a8c954cdaf6e316d4dc62721e3b63f9762d805650794f18632b675f279" exitCode=0
Oct 31 01:33:27.263918 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:33:27.263812    4964 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-flwkb"
Oct 31 01:33:27.263918 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:33:27.263867    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-flwkb" event={"ID":"da53c8d7-cc60-45cd-ab19-365d486484b1","Type":"ContainerDied","Data":"bdcf04a8c954cdaf6e316d4dc62721e3b63f9762d805650794f18632b675f279"}
Oct 31 01:33:27.263918 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:33:27.263906    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-flwkb" event={"ID":"da53c8d7-cc60-45cd-ab19-365d486484b1","Type":"ContainerDied","Data":"a6b38a7354dbf52e201f388f9aee18d1b1cfb98ed7600edff83b36d677c4f484"}
Oct 31 01:33:27.273811 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:33:27.273757    4964 scope.go:122] "RemoveContainer" containerID="d43ea5f023a2f0d664642c01246e01a25f303a5ef1d04b70f6c9e573d24c1320"
Oct 31 01:33:27.274092 ip-10-0-133-197 kubenswrapper[4964]: E1031 01:33:27.274068    4964 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d43ea5f023a2f0d664642c01246e01a25f303a5ef1d04b70f6c9e573d24c1320\": container with ID starting with d43ea5f023a2f0d664642c01246e01a25f303a5ef1d04b70f6c9e573d24c1320 not found: ID does not exist" containerID="d43ea5f023a2f0d664642c01246e01a25f303a5ef1d04b70f6c9e573d24c1320"
Oct 31 01:33:27.274180 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:33:27.274103    4964 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d43ea5f023a2f0d664642c01246e01a25f303a5ef1d04b70f6c9e573d24c1320"} err="failed to get container status \"d43ea5f023a2f0d664642c01246e01a25f303a5ef1d04b70f6c9e573d24c1320\": rpc error: code = NotFound desc = could not find container \"d43ea5f023a2f0d664642c01246e01a25f303a5ef1d04b70f6c9e573d24c1320\": container with ID starting with d43ea5f023a2f0d664642c01246e01a25f303a5ef1d04b70f6c9e573d24c1320 not found: ID does not exist"
Oct 31 01:33:27.274180 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:33:27.274127    4964 scope.go:122] "RemoveContainer" containerID="bdcf04a8c954cdaf6e316d4dc62721e3b63f9762d805650794f18632b675f279"
Oct 31 01:33:27.283602 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:33:27.283579    4964 scope.go:122] "RemoveContainer" containerID="bdcf04a8c954cdaf6e316d4dc62721e3b63f9762d805650794f18632b675f279"
Oct 31 01:33:27.283867 ip-10-0-133-197 kubenswrapper[4964]: E1031 01:33:27.283850    4964 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bdcf04a8c954cdaf6e316d4dc62721e3b63f9762d805650794f18632b675f279\": container with ID starting with bdcf04a8c954cdaf6e316d4dc62721e3b63f9762d805650794f18632b675f279 not found: ID does not exist" containerID="bdcf04a8c954cdaf6e316d4dc62721e3b63f9762d805650794f18632b675f279"
Oct 31 01:33:27.283912 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:33:27.283876    4964 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bdcf04a8c954cdaf6e316d4dc62721e3b63f9762d805650794f18632b675f279"} err="failed to get container status \"bdcf04a8c954cdaf6e316d4dc62721e3b63f9762d805650794f18632b675f279\": rpc error: code = NotFound desc = could not find container \"bdcf04a8c954cdaf6e316d4dc62721e3b63f9762d805650794f18632b675f279\": container with ID starting with bdcf04a8c954cdaf6e316d4dc62721e3b63f9762d805650794f18632b675f279 not found: ID does not exist"
Oct 31 01:33:27.427248 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:33:27.427217    4964 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-flwkb"]
Oct 31 01:33:27.477025 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:33:27.476998    4964 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-flwkb"]
Oct 31 01:33:27.587370 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:33:27.586269    4964 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-tt9c2"]
Oct 31 01:33:27.590946 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:33:27.590923    4964 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-tt9c2"]
Oct 31 01:33:27.783629 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:33:27.783597    4964 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="065217f5-7fda-424b-a2e7-fed363588c32" path="/var/lib/kubelet/pods/065217f5-7fda-424b-a2e7-fed363588c32/volumes"
Oct 31 01:33:27.784098 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:33:27.784084    4964 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2a53cf91-69f9-4c47-8271-79736ce35af9" path="/var/lib/kubelet/pods/2a53cf91-69f9-4c47-8271-79736ce35af9/volumes"
Oct 31 01:33:27.784544 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:33:27.784530    4964 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="da53c8d7-cc60-45cd-ab19-365d486484b1" path="/var/lib/kubelet/pods/da53c8d7-cc60-45cd-ab19-365d486484b1/volumes"
Oct 31 01:33:30.864786 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:33:30.864750    4964 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-14ca09-59f575c7d9-ttflk"]
Oct 31 01:33:31.215658 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:33:31.215572    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:33:31.401385 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:33:31.399174    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:33:32.150741 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:33:32.150708    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:33:34.848606 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:33:34.848574    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:33:35.865956 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:33:35.865900    4964 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-14ca09-59f575c7d9-ttflk" podUID="d8871960-8a84-44b1-b24a-a6447283f329" containerName="payload-processing" containerID="cri-o://5bf89c629df41376d9ed36fef057240cf348680d0ed8a93f6d86556fd2b22006" gracePeriod=25
Oct 31 01:33:36.152148 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:33:36.152123    4964 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-14ca09-59f575c7d9-ttflk"
Oct 31 01:33:36.259276 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:33:36.259240    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d8871960-8a84-44b1-b24a-a6447283f329-kube-api-access-ttgbh\" (UniqueName: \"kubernetes.io/projected/d8871960-8a84-44b1-b24a-a6447283f329-kube-api-access-ttgbh\") pod \"d8871960-8a84-44b1-b24a-a6447283f329\" (UID: \"d8871960-8a84-44b1-b24a-a6447283f329\") "
Oct 31 01:33:36.259508 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:33:36.259403    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d8871960-8a84-44b1-b24a-a6447283f329-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d8871960-8a84-44b1-b24a-a6447283f329-plugins-config-volume\") pod \"d8871960-8a84-44b1-b24a-a6447283f329\" (UID: \"d8871960-8a84-44b1-b24a-a6447283f329\") "
Oct 31 01:33:36.259789 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:33:36.259766    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d8871960-8a84-44b1-b24a-a6447283f329-plugins-config-volume" pod "d8871960-8a84-44b1-b24a-a6447283f329" (UID: "d8871960-8a84-44b1-b24a-a6447283f329"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:33:36.260020 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:33:36.260003    4964 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d8871960-8a84-44b1-b24a-a6447283f329-plugins-config-volume\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:33:36.261693 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:33:36.261671    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d8871960-8a84-44b1-b24a-a6447283f329-kube-api-access-ttgbh" pod "d8871960-8a84-44b1-b24a-a6447283f329" (UID: "d8871960-8a84-44b1-b24a-a6447283f329"). InnerVolumeSpecName "kube-api-access-ttgbh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:33:36.361037 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:33:36.361006    4964 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ttgbh\" (UniqueName: \"kubernetes.io/projected/d8871960-8a84-44b1-b24a-a6447283f329-kube-api-access-ttgbh\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:33:36.406036 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:33:36.405948    4964 generic.go:356] "Generic (PLEG): container finished" podID="d8871960-8a84-44b1-b24a-a6447283f329" containerID="5bf89c629df41376d9ed36fef057240cf348680d0ed8a93f6d86556fd2b22006" exitCode=0
Oct 31 01:33:36.406036 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:33:36.406008    4964 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-14ca09-59f575c7d9-ttflk"
Oct 31 01:33:36.406268 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:33:36.406037    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-14ca09-59f575c7d9-ttflk" event={"ID":"d8871960-8a84-44b1-b24a-a6447283f329","Type":"ContainerDied","Data":"5bf89c629df41376d9ed36fef057240cf348680d0ed8a93f6d86556fd2b22006"}
Oct 31 01:33:36.406268 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:33:36.406102    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-14ca09-59f575c7d9-ttflk" event={"ID":"d8871960-8a84-44b1-b24a-a6447283f329","Type":"ContainerDied","Data":"5922efb67f2965c56d392e0e328ec2c787c7dceb2c5bb21e83ff65c494acca13"}
Oct 31 01:33:36.406268 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:33:36.406132    4964 scope.go:122] "RemoveContainer" containerID="5bf89c629df41376d9ed36fef057240cf348680d0ed8a93f6d86556fd2b22006"
Oct 31 01:33:36.419261 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:33:36.419240    4964 scope.go:122] "RemoveContainer" containerID="5bf89c629df41376d9ed36fef057240cf348680d0ed8a93f6d86556fd2b22006"
Oct 31 01:33:36.419642 ip-10-0-133-197 kubenswrapper[4964]: E1031 01:33:36.419620    4964 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5bf89c629df41376d9ed36fef057240cf348680d0ed8a93f6d86556fd2b22006\": container with ID starting with 5bf89c629df41376d9ed36fef057240cf348680d0ed8a93f6d86556fd2b22006 not found: ID does not exist" containerID="5bf89c629df41376d9ed36fef057240cf348680d0ed8a93f6d86556fd2b22006"
Oct 31 01:33:36.419717 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:33:36.419650    4964 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5bf89c629df41376d9ed36fef057240cf348680d0ed8a93f6d86556fd2b22006"} err="failed to get container status \"5bf89c629df41376d9ed36fef057240cf348680d0ed8a93f6d86556fd2b22006\": rpc error: code = NotFound desc = could not find container \"5bf89c629df41376d9ed36fef057240cf348680d0ed8a93f6d86556fd2b22006\": container with ID starting with 5bf89c629df41376d9ed36fef057240cf348680d0ed8a93f6d86556fd2b22006 not found: ID does not exist"
Oct 31 01:33:36.569706 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:33:36.569674    4964 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-14ca09-59f575c7d9-ttflk"]
Oct 31 01:33:36.572907 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:33:36.572886    4964 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-14ca09-59f575c7d9-ttflk"]
Oct 31 01:33:36.808595 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:33:36.808561    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:33:37.301133 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:33:37.301099    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:33:37.778219 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:33:37.778191    4964 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d8871960-8a84-44b1-b24a-a6447283f329" path="/var/lib/kubelet/pods/d8871960-8a84-44b1-b24a-a6447283f329/volumes"
Oct 31 01:33:41.239888 ip-10-0-133-197 kubenswrapper[4964]: E1031 01:33:41.239845    4964 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q56x5" containerName="extractor"
Oct 31 01:33:42.504811 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:33:42.504770    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:33:44.103862 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:33:44.103825    4964 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-w9jpq"]
Oct 31 01:33:44.104399 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:33:44.104040    4964 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-w9jpq" podUID="145637b0-60a1-4990-8dfe-dd073d9b3929" containerName="istio-proxy" containerID="cri-o://d1f57631c27c364c9c7350fa54c860a7cf74b49618a770c325a6ae1754a9c97d" gracePeriod=30
Oct 31 01:33:45.530351 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:33:45.525893    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:33:45.701364 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:33:45.701277    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:33:47.579892 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:33:47.579861    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:33:49.372563 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:33:49.372501    4964 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-w9jpq"
Oct 31 01:33:49.532037 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:33:49.532004    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/145637b0-60a1-4990-8dfe-dd073d9b3929-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/145637b0-60a1-4990-8dfe-dd073d9b3929-istio-podinfo\") pod \"145637b0-60a1-4990-8dfe-dd073d9b3929\" (UID: \"145637b0-60a1-4990-8dfe-dd073d9b3929\") "
Oct 31 01:33:49.532186 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:33:49.532066    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/145637b0-60a1-4990-8dfe-dd073d9b3929-kube-api-access-hkf2b\" (UniqueName: \"kubernetes.io/projected/145637b0-60a1-4990-8dfe-dd073d9b3929-kube-api-access-hkf2b\") pod \"145637b0-60a1-4990-8dfe-dd073d9b3929\" (UID: \"145637b0-60a1-4990-8dfe-dd073d9b3929\") "
Oct 31 01:33:49.532186 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:33:49.532095    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/145637b0-60a1-4990-8dfe-dd073d9b3929-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/145637b0-60a1-4990-8dfe-dd073d9b3929-credential-socket\") pod \"145637b0-60a1-4990-8dfe-dd073d9b3929\" (UID: \"145637b0-60a1-4990-8dfe-dd073d9b3929\") "
Oct 31 01:33:49.532186 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:33:49.532145    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/145637b0-60a1-4990-8dfe-dd073d9b3929-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/145637b0-60a1-4990-8dfe-dd073d9b3929-istio-data\") pod \"145637b0-60a1-4990-8dfe-dd073d9b3929\" (UID: \"145637b0-60a1-4990-8dfe-dd073d9b3929\") "
Oct 31 01:33:49.532377 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:33:49.532201    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/145637b0-60a1-4990-8dfe-dd073d9b3929-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/145637b0-60a1-4990-8dfe-dd073d9b3929-workload-socket\") pod \"145637b0-60a1-4990-8dfe-dd073d9b3929\" (UID: \"145637b0-60a1-4990-8dfe-dd073d9b3929\") "
Oct 31 01:33:49.532377 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:33:49.532278    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/145637b0-60a1-4990-8dfe-dd073d9b3929-istio-token\" (UniqueName: \"kubernetes.io/projected/145637b0-60a1-4990-8dfe-dd073d9b3929-istio-token\") pod \"145637b0-60a1-4990-8dfe-dd073d9b3929\" (UID: \"145637b0-60a1-4990-8dfe-dd073d9b3929\") "
Oct 31 01:33:49.532506 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:33:49.532381    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/145637b0-60a1-4990-8dfe-dd073d9b3929-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/145637b0-60a1-4990-8dfe-dd073d9b3929-istio-envoy\") pod \"145637b0-60a1-4990-8dfe-dd073d9b3929\" (UID: \"145637b0-60a1-4990-8dfe-dd073d9b3929\") "
Oct 31 01:33:49.532506 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:33:49.532439    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/145637b0-60a1-4990-8dfe-dd073d9b3929-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/145637b0-60a1-4990-8dfe-dd073d9b3929-workload-certs\") pod \"145637b0-60a1-4990-8dfe-dd073d9b3929\" (UID: \"145637b0-60a1-4990-8dfe-dd073d9b3929\") "
Oct 31 01:33:49.532506 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:33:49.532481    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/145637b0-60a1-4990-8dfe-dd073d9b3929-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/145637b0-60a1-4990-8dfe-dd073d9b3929-istiod-ca-cert\") pod \"145637b0-60a1-4990-8dfe-dd073d9b3929\" (UID: \"145637b0-60a1-4990-8dfe-dd073d9b3929\") "
Oct 31 01:33:49.532667 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:33:49.532517    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/145637b0-60a1-4990-8dfe-dd073d9b3929-credential-socket" pod "145637b0-60a1-4990-8dfe-dd073d9b3929" (UID: "145637b0-60a1-4990-8dfe-dd073d9b3929"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:33:49.532667 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:33:49.532529    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/145637b0-60a1-4990-8dfe-dd073d9b3929-workload-socket" pod "145637b0-60a1-4990-8dfe-dd073d9b3929" (UID: "145637b0-60a1-4990-8dfe-dd073d9b3929"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:33:49.532667 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:33:49.532517    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/145637b0-60a1-4990-8dfe-dd073d9b3929-istio-data" pod "145637b0-60a1-4990-8dfe-dd073d9b3929" (UID: "145637b0-60a1-4990-8dfe-dd073d9b3929"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:33:49.532841 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:33:49.532705    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/145637b0-60a1-4990-8dfe-dd073d9b3929-workload-certs" pod "145637b0-60a1-4990-8dfe-dd073d9b3929" (UID: "145637b0-60a1-4990-8dfe-dd073d9b3929"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:33:49.532939 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:33:49.532916    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/145637b0-60a1-4990-8dfe-dd073d9b3929-istiod-ca-cert" pod "145637b0-60a1-4990-8dfe-dd073d9b3929" (UID: "145637b0-60a1-4990-8dfe-dd073d9b3929"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:33:49.533162 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:33:49.533140    4964 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/145637b0-60a1-4990-8dfe-dd073d9b3929-workload-certs\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:33:49.533162 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:33:49.533164    4964 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/145637b0-60a1-4990-8dfe-dd073d9b3929-istiod-ca-cert\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:33:49.533394 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:33:49.533174    4964 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/145637b0-60a1-4990-8dfe-dd073d9b3929-credential-socket\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:33:49.533394 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:33:49.533187    4964 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/145637b0-60a1-4990-8dfe-dd073d9b3929-istio-data\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:33:49.533394 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:33:49.533200    4964 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/145637b0-60a1-4990-8dfe-dd073d9b3929-workload-socket\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:33:49.534869 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:33:49.534844    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/145637b0-60a1-4990-8dfe-dd073d9b3929-kube-api-access-hkf2b" pod "145637b0-60a1-4990-8dfe-dd073d9b3929" (UID: "145637b0-60a1-4990-8dfe-dd073d9b3929"). InnerVolumeSpecName "kube-api-access-hkf2b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:33:49.535047 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:33:49.535013    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/145637b0-60a1-4990-8dfe-dd073d9b3929-istio-podinfo" pod "145637b0-60a1-4990-8dfe-dd073d9b3929" (UID: "145637b0-60a1-4990-8dfe-dd073d9b3929"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 01:33:49.535160 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:33:49.535093    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/145637b0-60a1-4990-8dfe-dd073d9b3929-istio-envoy" pod "145637b0-60a1-4990-8dfe-dd073d9b3929" (UID: "145637b0-60a1-4990-8dfe-dd073d9b3929"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:33:49.535160 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:33:49.535118    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/145637b0-60a1-4990-8dfe-dd073d9b3929-istio-token" pod "145637b0-60a1-4990-8dfe-dd073d9b3929" (UID: "145637b0-60a1-4990-8dfe-dd073d9b3929"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:33:49.562030 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:33:49.561994    4964 generic.go:356] "Generic (PLEG): container finished" podID="145637b0-60a1-4990-8dfe-dd073d9b3929" containerID="d1f57631c27c364c9c7350fa54c860a7cf74b49618a770c325a6ae1754a9c97d" exitCode=0
Oct 31 01:33:49.562181 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:33:49.562067    4964 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-w9jpq"
Oct 31 01:33:49.562181 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:33:49.562088    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-w9jpq" event={"ID":"145637b0-60a1-4990-8dfe-dd073d9b3929","Type":"ContainerDied","Data":"d1f57631c27c364c9c7350fa54c860a7cf74b49618a770c325a6ae1754a9c97d"}
Oct 31 01:33:49.562181 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:33:49.562165    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-w9jpq" event={"ID":"145637b0-60a1-4990-8dfe-dd073d9b3929","Type":"ContainerDied","Data":"a17c59f02a0fc62bd8d613147f8ee112f6210d0c2bf8ce4f1e075b11964fedae"}
Oct 31 01:33:49.562376 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:33:49.562198    4964 scope.go:122] "RemoveContainer" containerID="d1f57631c27c364c9c7350fa54c860a7cf74b49618a770c325a6ae1754a9c97d"
Oct 31 01:33:49.574525 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:33:49.574503    4964 scope.go:122] "RemoveContainer" containerID="d1f57631c27c364c9c7350fa54c860a7cf74b49618a770c325a6ae1754a9c97d"
Oct 31 01:33:49.574849 ip-10-0-133-197 kubenswrapper[4964]: E1031 01:33:49.574831    4964 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d1f57631c27c364c9c7350fa54c860a7cf74b49618a770c325a6ae1754a9c97d\": container with ID starting with d1f57631c27c364c9c7350fa54c860a7cf74b49618a770c325a6ae1754a9c97d not found: ID does not exist" containerID="d1f57631c27c364c9c7350fa54c860a7cf74b49618a770c325a6ae1754a9c97d"
Oct 31 01:33:49.574947 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:33:49.574857    4964 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d1f57631c27c364c9c7350fa54c860a7cf74b49618a770c325a6ae1754a9c97d"} err="failed to get container status \"d1f57631c27c364c9c7350fa54c860a7cf74b49618a770c325a6ae1754a9c97d\": rpc error: code = NotFound desc = could not find container \"d1f57631c27c364c9c7350fa54c860a7cf74b49618a770c325a6ae1754a9c97d\": container with ID starting with d1f57631c27c364c9c7350fa54c860a7cf74b49618a770c325a6ae1754a9c97d not found: ID does not exist"
Oct 31 01:33:49.634819 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:33:49.634750    4964 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/145637b0-60a1-4990-8dfe-dd073d9b3929-istio-podinfo\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:33:49.634819 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:33:49.634780    4964 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hkf2b\" (UniqueName: \"kubernetes.io/projected/145637b0-60a1-4990-8dfe-dd073d9b3929-kube-api-access-hkf2b\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:33:49.634819 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:33:49.634794    4964 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/145637b0-60a1-4990-8dfe-dd073d9b3929-istio-token\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:33:49.634819 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:33:49.634806    4964 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/145637b0-60a1-4990-8dfe-dd073d9b3929-istio-envoy\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:33:49.725777 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:33:49.725707    4964 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-w9jpq"]
Oct 31 01:33:49.728862 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:33:49.728840    4964 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-w9jpq"]
Oct 31 01:33:49.742875 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:33:49.742852    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:33:49.779691 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:33:49.779661    4964 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="145637b0-60a1-4990-8dfe-dd073d9b3929" path="/var/lib/kubelet/pods/145637b0-60a1-4990-8dfe-dd073d9b3929/volumes"
Oct 31 01:33:49.986894 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:33:49.986861    4964 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-2c70e7-5d586ccdb7-l82mc"]
Oct 31 01:33:50.052115 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:33:50.052081    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:33:50.338644 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:33:50.338566    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:33:50.525910 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:33:50.525872    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:33:52.024194 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:33:52.024168    4964 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-9257n_f0c0ada7-7688-4dc2-97fc-813e4d0c4ef3/console-operator/2.log"
Oct 31 01:33:52.030691 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:33:52.030670    4964 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-9257n_f0c0ada7-7688-4dc2-97fc-813e4d0c4ef3/console-operator/2.log"
Oct 31 01:33:52.855064 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:33:52.855024    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:33:53.773235 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:33:53.773188    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:33:54.023436 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:33:54.023328    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:33:54.987642 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:33:54.987580    4964 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-2c70e7-5d586ccdb7-l82mc" podUID="2708a82e-32b4-443c-8cfe-97815ae5f85b" containerName="payload-pre-processing" containerID="cri-o://c4b8f4383237145d9dc69e41d9af5c1cedeea41c16fc57d57dfec38c5cafa94b" gracePeriod=25
Oct 31 01:33:55.284924 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:33:55.284898    4964 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-2c70e7-5d586ccdb7-l82mc"
Oct 31 01:33:55.301831 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:33:55.301806    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2708a82e-32b4-443c-8cfe-97815ae5f85b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2708a82e-32b4-443c-8cfe-97815ae5f85b-plugins-config-volume\") pod \"2708a82e-32b4-443c-8cfe-97815ae5f85b\" (UID: \"2708a82e-32b4-443c-8cfe-97815ae5f85b\") "
Oct 31 01:33:55.301950 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:33:55.301897    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2708a82e-32b4-443c-8cfe-97815ae5f85b-kube-api-access-fncqm\" (UniqueName: \"kubernetes.io/projected/2708a82e-32b4-443c-8cfe-97815ae5f85b-kube-api-access-fncqm\") pod \"2708a82e-32b4-443c-8cfe-97815ae5f85b\" (UID: \"2708a82e-32b4-443c-8cfe-97815ae5f85b\") "
Oct 31 01:33:55.302199 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:33:55.302174    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2708a82e-32b4-443c-8cfe-97815ae5f85b-plugins-config-volume" pod "2708a82e-32b4-443c-8cfe-97815ae5f85b" (UID: "2708a82e-32b4-443c-8cfe-97815ae5f85b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:33:55.302495 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:33:55.302477    4964 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2708a82e-32b4-443c-8cfe-97815ae5f85b-plugins-config-volume\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:33:55.304233 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:33:55.304205    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2708a82e-32b4-443c-8cfe-97815ae5f85b-kube-api-access-fncqm" pod "2708a82e-32b4-443c-8cfe-97815ae5f85b" (UID: "2708a82e-32b4-443c-8cfe-97815ae5f85b"). InnerVolumeSpecName "kube-api-access-fncqm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:33:55.403276 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:33:55.403239    4964 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fncqm\" (UniqueName: \"kubernetes.io/projected/2708a82e-32b4-443c-8cfe-97815ae5f85b-kube-api-access-fncqm\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:33:55.609234 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:33:55.609138    4964 generic.go:356] "Generic (PLEG): container finished" podID="2708a82e-32b4-443c-8cfe-97815ae5f85b" containerID="c4b8f4383237145d9dc69e41d9af5c1cedeea41c16fc57d57dfec38c5cafa94b" exitCode=0
Oct 31 01:33:55.609234 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:33:55.609185    4964 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-2c70e7-5d586ccdb7-l82mc"
Oct 31 01:33:55.609234 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:33:55.609221    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-2c70e7-5d586ccdb7-l82mc" event={"ID":"2708a82e-32b4-443c-8cfe-97815ae5f85b","Type":"ContainerDied","Data":"c4b8f4383237145d9dc69e41d9af5c1cedeea41c16fc57d57dfec38c5cafa94b"}
Oct 31 01:33:55.609548 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:33:55.609299    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-2c70e7-5d586ccdb7-l82mc" event={"ID":"2708a82e-32b4-443c-8cfe-97815ae5f85b","Type":"ContainerDied","Data":"c203321453896b258b0e2cf120ef4ddb92b8e28ee63b31ebf818a2a23e1b5184"}
Oct 31 01:33:55.609548 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:33:55.609367    4964 scope.go:122] "RemoveContainer" containerID="c4b8f4383237145d9dc69e41d9af5c1cedeea41c16fc57d57dfec38c5cafa94b"
Oct 31 01:33:55.620685 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:33:55.620668    4964 scope.go:122] "RemoveContainer" containerID="c4b8f4383237145d9dc69e41d9af5c1cedeea41c16fc57d57dfec38c5cafa94b"
Oct 31 01:33:55.620986 ip-10-0-133-197 kubenswrapper[4964]: E1031 01:33:55.620966    4964 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c4b8f4383237145d9dc69e41d9af5c1cedeea41c16fc57d57dfec38c5cafa94b\": container with ID starting with c4b8f4383237145d9dc69e41d9af5c1cedeea41c16fc57d57dfec38c5cafa94b not found: ID does not exist" containerID="c4b8f4383237145d9dc69e41d9af5c1cedeea41c16fc57d57dfec38c5cafa94b"
Oct 31 01:33:55.621067 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:33:55.620994    4964 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c4b8f4383237145d9dc69e41d9af5c1cedeea41c16fc57d57dfec38c5cafa94b"} err="failed to get container status \"c4b8f4383237145d9dc69e41d9af5c1cedeea41c16fc57d57dfec38c5cafa94b\": rpc error: code = NotFound desc = could not find container \"c4b8f4383237145d9dc69e41d9af5c1cedeea41c16fc57d57dfec38c5cafa94b\": container with ID starting with c4b8f4383237145d9dc69e41d9af5c1cedeea41c16fc57d57dfec38c5cafa94b not found: ID does not exist"
Oct 31 01:33:55.790062 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:33:55.790031    4964 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-2c70e7-5d586ccdb7-l82mc"]
Oct 31 01:33:55.790062 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:33:55.790077    4964 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-2c70e7-5d586ccdb7-l82mc"]
Oct 31 01:33:57.134595 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:33:57.134563    4964 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-e7eea9-74d6c5586f-hp7tc"]
Oct 31 01:33:57.135011 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:33:57.134837    4964 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e7eea9-74d6c5586f-hp7tc" podUID="a4153b03-5882-419a-bb5c-f1c9592b23fa" containerName="maas-api" containerID="cri-o://058a52c23059c902e390ae4d0854ae7c11f69aaa3322ded8bc91de289aef211f" gracePeriod=30
Oct 31 01:33:57.229243 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:33:57.228596    4964 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e7eea9-74d6c5586f-hp7tc" podUID="a4153b03-5882-419a-bb5c-f1c9592b23fa" containerName="maas-api" probeResult="failure" output="Get \"https://10.132.0.116:8443/health\": dial tcp 10.132.0.116:8443: connect: connection refused"
Oct 31 01:33:57.281856 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:33:57.281820    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:33:57.434951 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:33:57.434925    4964 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e7eea9-74d6c5586f-hp7tc"
Oct 31 01:33:57.535910 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:33:57.535872    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a4153b03-5882-419a-bb5c-f1c9592b23fa-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a4153b03-5882-419a-bb5c-f1c9592b23fa-maas-api-tls\") pod \"a4153b03-5882-419a-bb5c-f1c9592b23fa\" (UID: \"a4153b03-5882-419a-bb5c-f1c9592b23fa\") "
Oct 31 01:33:57.536135 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:33:57.535996    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a4153b03-5882-419a-bb5c-f1c9592b23fa-kube-api-access-jp2r8\" (UniqueName: \"kubernetes.io/projected/a4153b03-5882-419a-bb5c-f1c9592b23fa-kube-api-access-jp2r8\") pod \"a4153b03-5882-419a-bb5c-f1c9592b23fa\" (UID: \"a4153b03-5882-419a-bb5c-f1c9592b23fa\") "
Oct 31 01:33:57.536135 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:33:57.536020    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a4153b03-5882-419a-bb5c-f1c9592b23fa-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a4153b03-5882-419a-bb5c-f1c9592b23fa-openshift-service-ca\") pod \"a4153b03-5882-419a-bb5c-f1c9592b23fa\" (UID: \"a4153b03-5882-419a-bb5c-f1c9592b23fa\") "
Oct 31 01:33:57.536245 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:33:57.536151    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a4153b03-5882-419a-bb5c-f1c9592b23fa-metrics-certs\" (UniqueName: \"kubernetes.io/secret/a4153b03-5882-419a-bb5c-f1c9592b23fa-metrics-certs\") pod \"a4153b03-5882-419a-bb5c-f1c9592b23fa\" (UID: \"a4153b03-5882-419a-bb5c-f1c9592b23fa\") "
Oct 31 01:33:57.537025 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:33:57.536991    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a4153b03-5882-419a-bb5c-f1c9592b23fa-openshift-service-ca" pod "a4153b03-5882-419a-bb5c-f1c9592b23fa" (UID: "a4153b03-5882-419a-bb5c-f1c9592b23fa"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:33:57.539353 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:33:57.539293    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a4153b03-5882-419a-bb5c-f1c9592b23fa-kube-api-access-jp2r8" pod "a4153b03-5882-419a-bb5c-f1c9592b23fa" (UID: "a4153b03-5882-419a-bb5c-f1c9592b23fa"). InnerVolumeSpecName "kube-api-access-jp2r8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:33:57.540634 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:33:57.540607    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a4153b03-5882-419a-bb5c-f1c9592b23fa-metrics-certs" pod "a4153b03-5882-419a-bb5c-f1c9592b23fa" (UID: "a4153b03-5882-419a-bb5c-f1c9592b23fa"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:33:57.541167 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:33:57.541144    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a4153b03-5882-419a-bb5c-f1c9592b23fa-maas-api-tls" pod "a4153b03-5882-419a-bb5c-f1c9592b23fa" (UID: "a4153b03-5882-419a-bb5c-f1c9592b23fa"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:33:57.629212 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:33:57.629179    4964 generic.go:356] "Generic (PLEG): container finished" podID="a4153b03-5882-419a-bb5c-f1c9592b23fa" containerID="058a52c23059c902e390ae4d0854ae7c11f69aaa3322ded8bc91de289aef211f" exitCode=0
Oct 31 01:33:57.629417 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:33:57.629297    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e7eea9-74d6c5586f-hp7tc" event={"ID":"a4153b03-5882-419a-bb5c-f1c9592b23fa","Type":"ContainerDied","Data":"058a52c23059c902e390ae4d0854ae7c11f69aaa3322ded8bc91de289aef211f"}
Oct 31 01:33:57.629417 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:33:57.629392    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e7eea9-74d6c5586f-hp7tc" event={"ID":"a4153b03-5882-419a-bb5c-f1c9592b23fa","Type":"ContainerDied","Data":"f2a84fa368f19fcc2914b0238f38730e9a5af216fdd4d60b682b74f1055aaed8"}
Oct 31 01:33:57.629586 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:33:57.629421    4964 scope.go:122] "RemoveContainer" containerID="058a52c23059c902e390ae4d0854ae7c11f69aaa3322ded8bc91de289aef211f"
Oct 31 01:33:57.629586 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:33:57.629557    4964 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e7eea9-74d6c5586f-hp7tc"
Oct 31 01:33:57.637936 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:33:57.637913    4964 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a4153b03-5882-419a-bb5c-f1c9592b23fa-metrics-certs\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:33:57.638083 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:33:57.637939    4964 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a4153b03-5882-419a-bb5c-f1c9592b23fa-maas-api-tls\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:33:57.638083 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:33:57.637954    4964 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jp2r8\" (UniqueName: \"kubernetes.io/projected/a4153b03-5882-419a-bb5c-f1c9592b23fa-kube-api-access-jp2r8\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:33:57.638083 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:33:57.637969    4964 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a4153b03-5882-419a-bb5c-f1c9592b23fa-openshift-service-ca\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:33:57.644609 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:33:57.644461    4964 scope.go:122] "RemoveContainer" containerID="058a52c23059c902e390ae4d0854ae7c11f69aaa3322ded8bc91de289aef211f"
Oct 31 01:33:57.644856 ip-10-0-133-197 kubenswrapper[4964]: E1031 01:33:57.644822    4964 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"058a52c23059c902e390ae4d0854ae7c11f69aaa3322ded8bc91de289aef211f\": container with ID starting with 058a52c23059c902e390ae4d0854ae7c11f69aaa3322ded8bc91de289aef211f not found: ID does not exist" containerID="058a52c23059c902e390ae4d0854ae7c11f69aaa3322ded8bc91de289aef211f"
Oct 31 01:33:57.644983 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:33:57.644864    4964 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"058a52c23059c902e390ae4d0854ae7c11f69aaa3322ded8bc91de289aef211f"} err="failed to get container status \"058a52c23059c902e390ae4d0854ae7c11f69aaa3322ded8bc91de289aef211f\": rpc error: code = NotFound desc = could not find container \"058a52c23059c902e390ae4d0854ae7c11f69aaa3322ded8bc91de289aef211f\": container with ID starting with 058a52c23059c902e390ae4d0854ae7c11f69aaa3322ded8bc91de289aef211f not found: ID does not exist"
Oct 31 01:33:57.779245 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:33:57.779212    4964 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2708a82e-32b4-443c-8cfe-97815ae5f85b" path="/var/lib/kubelet/pods/2708a82e-32b4-443c-8cfe-97815ae5f85b/volumes"
Oct 31 01:33:57.795127 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:33:57.795098    4964 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-e7eea9-74d6c5586f-hp7tc"]
Oct 31 01:33:57.798374 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:33:57.798346    4964 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-e7eea9-74d6c5586f-hp7tc"]
Oct 31 01:33:58.863365 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:33:58.863295    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:33:59.093352 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:33:59.093296    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:33:59.274332 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:33:59.274274    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:33:59.816790 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:33:59.816745    4964 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a4153b03-5882-419a-bb5c-f1c9592b23fa" path="/var/lib/kubelet/pods/a4153b03-5882-419a-bb5c-f1c9592b23fa/volumes"
Oct 31 01:33:59.866373 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:33:59.866333    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:34:04.369542 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:34:04.369510    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:34:05.899853 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:34:05.899806    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:34:06.085835 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:34:06.085796    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:34:06.207983 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:34:06.207906    4964 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-e7eea9-openshift-default-76c54449f5-c6bvv"]
Oct 31 01:34:06.208147 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:34:06.208121    4964 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-e7eea9-openshift-default-76c54449f5-c6bvv" podUID="eb13aca8-a24f-4fc8-a51f-3d3f45d739ec" containerName="istio-proxy" containerID="cri-o://6432c11433f0325c461454a8717d858534f8b25b0cf44ae9a07089dfb9c9877c" gracePeriod=30
Oct 31 01:34:06.287848 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:34:06.287813    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:34:07.231019 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:34:07.230981    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:34:08.542920 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:34:08.542883    4964 patch_prober.go:29] interesting pod/e2e-api-b-e7eea9-openshift-default-76c54449f5-c6bvv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 01:34:08.543372 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:34:08.542951    4964 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-e7eea9-openshift-default-76c54449f5-c6bvv" podUID="eb13aca8-a24f-4fc8-a51f-3d3f45d739ec" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 01:34:09.304198 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:34:09.304161    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:34:10.983854 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:34:10.983807    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:34:11.241456 ip-10-0-133-197 kubenswrapper[4964]: E1031 01:34:11.241422    4964 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q56x5" containerName="extractor"
Oct 31 01:34:11.476624 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:34:11.476601    4964 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-e7eea9-openshift-default-76c54449f5-c6bvv"
Oct 31 01:34:11.617229 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:34:11.617156    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/eb13aca8-a24f-4fc8-a51f-3d3f45d739ec-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eb13aca8-a24f-4fc8-a51f-3d3f45d739ec-istio-podinfo\") pod \"eb13aca8-a24f-4fc8-a51f-3d3f45d739ec\" (UID: \"eb13aca8-a24f-4fc8-a51f-3d3f45d739ec\") "
Oct 31 01:34:11.617229 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:34:11.617211    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eb13aca8-a24f-4fc8-a51f-3d3f45d739ec-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eb13aca8-a24f-4fc8-a51f-3d3f45d739ec-workload-certs\") pod \"eb13aca8-a24f-4fc8-a51f-3d3f45d739ec\" (UID: \"eb13aca8-a24f-4fc8-a51f-3d3f45d739ec\") "
Oct 31 01:34:11.617496 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:34:11.617256    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eb13aca8-a24f-4fc8-a51f-3d3f45d739ec-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb13aca8-a24f-4fc8-a51f-3d3f45d739ec-workload-socket\") pod \"eb13aca8-a24f-4fc8-a51f-3d3f45d739ec\" (UID: \"eb13aca8-a24f-4fc8-a51f-3d3f45d739ec\") "
Oct 31 01:34:11.617496 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:34:11.617311    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eb13aca8-a24f-4fc8-a51f-3d3f45d739ec-istio-token\" (UniqueName: \"kubernetes.io/projected/eb13aca8-a24f-4fc8-a51f-3d3f45d739ec-istio-token\") pod \"eb13aca8-a24f-4fc8-a51f-3d3f45d739ec\" (UID: \"eb13aca8-a24f-4fc8-a51f-3d3f45d739ec\") "
Oct 31 01:34:11.617496 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:34:11.617425    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eb13aca8-a24f-4fc8-a51f-3d3f45d739ec-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eb13aca8-a24f-4fc8-a51f-3d3f45d739ec-istio-data\") pod \"eb13aca8-a24f-4fc8-a51f-3d3f45d739ec\" (UID: \"eb13aca8-a24f-4fc8-a51f-3d3f45d739ec\") "
Oct 31 01:34:11.617496 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:34:11.617460    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eb13aca8-a24f-4fc8-a51f-3d3f45d739ec-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb13aca8-a24f-4fc8-a51f-3d3f45d739ec-credential-socket\") pod \"eb13aca8-a24f-4fc8-a51f-3d3f45d739ec\" (UID: \"eb13aca8-a24f-4fc8-a51f-3d3f45d739ec\") "
Oct 31 01:34:11.617496 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:34:11.617486    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eb13aca8-a24f-4fc8-a51f-3d3f45d739ec-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eb13aca8-a24f-4fc8-a51f-3d3f45d739ec-istio-envoy\") pod \"eb13aca8-a24f-4fc8-a51f-3d3f45d739ec\" (UID: \"eb13aca8-a24f-4fc8-a51f-3d3f45d739ec\") "
Oct 31 01:34:11.617752 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:34:11.617518    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eb13aca8-a24f-4fc8-a51f-3d3f45d739ec-kube-api-access-r6cvp\" (UniqueName: \"kubernetes.io/projected/eb13aca8-a24f-4fc8-a51f-3d3f45d739ec-kube-api-access-r6cvp\") pod \"eb13aca8-a24f-4fc8-a51f-3d3f45d739ec\" (UID: \"eb13aca8-a24f-4fc8-a51f-3d3f45d739ec\") "
Oct 31 01:34:11.617752 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:34:11.617600    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb13aca8-a24f-4fc8-a51f-3d3f45d739ec-workload-certs" pod "eb13aca8-a24f-4fc8-a51f-3d3f45d739ec" (UID: "eb13aca8-a24f-4fc8-a51f-3d3f45d739ec"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:34:11.617752 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:34:11.617616    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eb13aca8-a24f-4fc8-a51f-3d3f45d739ec-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eb13aca8-a24f-4fc8-a51f-3d3f45d739ec-istiod-ca-cert\") pod \"eb13aca8-a24f-4fc8-a51f-3d3f45d739ec\" (UID: \"eb13aca8-a24f-4fc8-a51f-3d3f45d739ec\") "
Oct 31 01:34:11.617913 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:34:11.617611    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb13aca8-a24f-4fc8-a51f-3d3f45d739ec-workload-socket" pod "eb13aca8-a24f-4fc8-a51f-3d3f45d739ec" (UID: "eb13aca8-a24f-4fc8-a51f-3d3f45d739ec"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:34:11.617913 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:34:11.617703    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb13aca8-a24f-4fc8-a51f-3d3f45d739ec-istio-data" pod "eb13aca8-a24f-4fc8-a51f-3d3f45d739ec" (UID: "eb13aca8-a24f-4fc8-a51f-3d3f45d739ec"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:34:11.618034 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:34:11.617942    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb13aca8-a24f-4fc8-a51f-3d3f45d739ec-credential-socket" pod "eb13aca8-a24f-4fc8-a51f-3d3f45d739ec" (UID: "eb13aca8-a24f-4fc8-a51f-3d3f45d739ec"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:34:11.618034 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:34:11.617985    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eb13aca8-a24f-4fc8-a51f-3d3f45d739ec-istiod-ca-cert" pod "eb13aca8-a24f-4fc8-a51f-3d3f45d739ec" (UID: "eb13aca8-a24f-4fc8-a51f-3d3f45d739ec"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:34:11.618430 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:34:11.618406    4964 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eb13aca8-a24f-4fc8-a51f-3d3f45d739ec-istio-data\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:34:11.618521 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:34:11.618436    4964 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb13aca8-a24f-4fc8-a51f-3d3f45d739ec-credential-socket\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:34:11.618521 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:34:11.618454    4964 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eb13aca8-a24f-4fc8-a51f-3d3f45d739ec-istiod-ca-cert\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:34:11.618521 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:34:11.618469    4964 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eb13aca8-a24f-4fc8-a51f-3d3f45d739ec-workload-certs\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:34:11.618521 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:34:11.618483    4964 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb13aca8-a24f-4fc8-a51f-3d3f45d739ec-workload-socket\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:34:11.619642 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:34:11.619621    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/eb13aca8-a24f-4fc8-a51f-3d3f45d739ec-istio-podinfo" pod "eb13aca8-a24f-4fc8-a51f-3d3f45d739ec" (UID: "eb13aca8-a24f-4fc8-a51f-3d3f45d739ec"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 01:34:11.619938 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:34:11.619920    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb13aca8-a24f-4fc8-a51f-3d3f45d739ec-kube-api-access-r6cvp" pod "eb13aca8-a24f-4fc8-a51f-3d3f45d739ec" (UID: "eb13aca8-a24f-4fc8-a51f-3d3f45d739ec"). InnerVolumeSpecName "kube-api-access-r6cvp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:34:11.620211 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:34:11.620186    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb13aca8-a24f-4fc8-a51f-3d3f45d739ec-istio-envoy" pod "eb13aca8-a24f-4fc8-a51f-3d3f45d739ec" (UID: "eb13aca8-a24f-4fc8-a51f-3d3f45d739ec"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:34:11.620211 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:34:11.620203    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb13aca8-a24f-4fc8-a51f-3d3f45d739ec-istio-token" pod "eb13aca8-a24f-4fc8-a51f-3d3f45d739ec" (UID: "eb13aca8-a24f-4fc8-a51f-3d3f45d739ec"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:34:11.720215 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:34:11.720179    4964 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eb13aca8-a24f-4fc8-a51f-3d3f45d739ec-istio-token\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:34:11.720215 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:34:11.720215    4964 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eb13aca8-a24f-4fc8-a51f-3d3f45d739ec-istio-envoy\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:34:11.720459 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:34:11.720230    4964 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r6cvp\" (UniqueName: \"kubernetes.io/projected/eb13aca8-a24f-4fc8-a51f-3d3f45d739ec-kube-api-access-r6cvp\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:34:11.720459 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:34:11.720239    4964 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eb13aca8-a24f-4fc8-a51f-3d3f45d739ec-istio-podinfo\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:34:11.735548 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:34:11.735515    4964 generic.go:356] "Generic (PLEG): container finished" podID="eb13aca8-a24f-4fc8-a51f-3d3f45d739ec" containerID="6432c11433f0325c461454a8717d858534f8b25b0cf44ae9a07089dfb9c9877c" exitCode=0
Oct 31 01:34:11.735702 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:34:11.735563    4964 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-e7eea9-openshift-default-76c54449f5-c6bvv"
Oct 31 01:34:11.735702 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:34:11.735567    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-e7eea9-openshift-default-76c54449f5-c6bvv" event={"ID":"eb13aca8-a24f-4fc8-a51f-3d3f45d739ec","Type":"ContainerDied","Data":"6432c11433f0325c461454a8717d858534f8b25b0cf44ae9a07089dfb9c9877c"}
Oct 31 01:34:11.735702 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:34:11.735625    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-e7eea9-openshift-default-76c54449f5-c6bvv" event={"ID":"eb13aca8-a24f-4fc8-a51f-3d3f45d739ec","Type":"ContainerDied","Data":"fe310b73fc4bb80eac8e38e3b45b33921d8dcd5293fbc5f35da5fb1730974a7e"}
Oct 31 01:34:11.735702 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:34:11.735653    4964 scope.go:122] "RemoveContainer" containerID="6432c11433f0325c461454a8717d858534f8b25b0cf44ae9a07089dfb9c9877c"
Oct 31 01:34:11.747871 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:34:11.747849    4964 scope.go:122] "RemoveContainer" containerID="1136ce451c0dddd2c9cbcfcb5f38288098a69c45635846edaf608507b1c4cff3"
Oct 31 01:34:11.757423 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:34:11.757407    4964 scope.go:122] "RemoveContainer" containerID="6432c11433f0325c461454a8717d858534f8b25b0cf44ae9a07089dfb9c9877c"
Oct 31 01:34:11.757674 ip-10-0-133-197 kubenswrapper[4964]: E1031 01:34:11.757658    4964 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6432c11433f0325c461454a8717d858534f8b25b0cf44ae9a07089dfb9c9877c\": container with ID starting with 6432c11433f0325c461454a8717d858534f8b25b0cf44ae9a07089dfb9c9877c not found: ID does not exist" containerID="6432c11433f0325c461454a8717d858534f8b25b0cf44ae9a07089dfb9c9877c"
Oct 31 01:34:11.757716 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:34:11.757683    4964 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6432c11433f0325c461454a8717d858534f8b25b0cf44ae9a07089dfb9c9877c"} err="failed to get container status \"6432c11433f0325c461454a8717d858534f8b25b0cf44ae9a07089dfb9c9877c\": rpc error: code = NotFound desc = could not find container \"6432c11433f0325c461454a8717d858534f8b25b0cf44ae9a07089dfb9c9877c\": container with ID starting with 6432c11433f0325c461454a8717d858534f8b25b0cf44ae9a07089dfb9c9877c not found: ID does not exist"
Oct 31 01:34:11.757716 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:34:11.757700    4964 scope.go:122] "RemoveContainer" containerID="1136ce451c0dddd2c9cbcfcb5f38288098a69c45635846edaf608507b1c4cff3"
Oct 31 01:34:11.757888 ip-10-0-133-197 kubenswrapper[4964]: E1031 01:34:11.757871    4964 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1136ce451c0dddd2c9cbcfcb5f38288098a69c45635846edaf608507b1c4cff3\": container with ID starting with 1136ce451c0dddd2c9cbcfcb5f38288098a69c45635846edaf608507b1c4cff3 not found: ID does not exist" containerID="1136ce451c0dddd2c9cbcfcb5f38288098a69c45635846edaf608507b1c4cff3"
Oct 31 01:34:11.757927 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:34:11.757894    4964 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1136ce451c0dddd2c9cbcfcb5f38288098a69c45635846edaf608507b1c4cff3"} err="failed to get container status \"1136ce451c0dddd2c9cbcfcb5f38288098a69c45635846edaf608507b1c4cff3\": rpc error: code = NotFound desc = could not find container \"1136ce451c0dddd2c9cbcfcb5f38288098a69c45635846edaf608507b1c4cff3\": container with ID starting with 1136ce451c0dddd2c9cbcfcb5f38288098a69c45635846edaf608507b1c4cff3 not found: ID does not exist"
Oct 31 01:34:11.903463 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:34:11.903385    4964 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-e7eea9-openshift-default-76c54449f5-c6bvv"]
Oct 31 01:34:11.908462 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:34:11.908439    4964 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-e7eea9-openshift-default-76c54449f5-c6bvv"]
Oct 31 01:34:13.777556 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:34:13.777526    4964 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eb13aca8-a24f-4fc8-a51f-3d3f45d739ec" path="/var/lib/kubelet/pods/eb13aca8-a24f-4fc8-a51f-3d3f45d739ec/volumes"
Oct 31 01:34:14.099909 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:34:14.099802    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:34:17.763218 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:34:17.763188    4964 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-9e5368-openshift-default-5ff7fc6fd7-fp6pw"]
Oct 31 01:34:17.763871 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:34:17.763855    4964 cpu_manager.go:414] "containerMap: removing container" podUID="eb13aca8-a24f-4fc8-a51f-3d3f45d739ec" containerName="istio-proxy"
Oct 31 01:34:17.763920 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:34:17.763873    4964 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eb13aca8-a24f-4fc8-a51f-3d3f45d739ec" containerName="istio-proxy"
Oct 31 01:34:17.763920 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:34:17.763884    4964 cpu_manager.go:414] "containerMap: removing container" podUID="d8871960-8a84-44b1-b24a-a6447283f329" containerName="payload-processing"
Oct 31 01:34:17.763920 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:34:17.763889    4964 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d8871960-8a84-44b1-b24a-a6447283f329" containerName="payload-processing"
Oct 31 01:34:17.763920 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:34:17.763899    4964 cpu_manager.go:414] "containerMap: removing container" podUID="a4153b03-5882-419a-bb5c-f1c9592b23fa" containerName="maas-api"
Oct 31 01:34:17.763920 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:34:17.763903    4964 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a4153b03-5882-419a-bb5c-f1c9592b23fa" containerName="maas-api"
Oct 31 01:34:17.763920 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:34:17.763911    4964 cpu_manager.go:414] "containerMap: removing container" podUID="2a53cf91-69f9-4c47-8271-79736ce35af9" containerName="istio-proxy"
Oct 31 01:34:17.763920 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:34:17.763916    4964 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2a53cf91-69f9-4c47-8271-79736ce35af9" containerName="istio-proxy"
Oct 31 01:34:17.764133 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:34:17.763924    4964 cpu_manager.go:414] "containerMap: removing container" podUID="fbf50441-0853-4ecf-99ea-38a47e9d5677" containerName="payload-processing"
Oct 31 01:34:17.764133 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:34:17.763929    4964 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fbf50441-0853-4ecf-99ea-38a47e9d5677" containerName="payload-processing"
Oct 31 01:34:17.764133 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:34:17.763940    4964 cpu_manager.go:414] "containerMap: removing container" podUID="eb13aca8-a24f-4fc8-a51f-3d3f45d739ec" containerName="istio-proxy"
Oct 31 01:34:17.764133 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:34:17.763944    4964 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eb13aca8-a24f-4fc8-a51f-3d3f45d739ec" containerName="istio-proxy"
Oct 31 01:34:17.764133 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:34:17.763958    4964 cpu_manager.go:414] "containerMap: removing container" podUID="2708a82e-32b4-443c-8cfe-97815ae5f85b" containerName="payload-pre-processing"
Oct 31 01:34:17.764133 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:34:17.763962    4964 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2708a82e-32b4-443c-8cfe-97815ae5f85b" containerName="payload-pre-processing"
Oct 31 01:34:17.764133 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:34:17.763969    4964 cpu_manager.go:414] "containerMap: removing container" podUID="f1770ebf-036d-4645-b3f2-c1f14574438f" containerName="payload-pre-processing"
Oct 31 01:34:17.764133 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:34:17.763976    4964 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f1770ebf-036d-4645-b3f2-c1f14574438f" containerName="payload-pre-processing"
Oct 31 01:34:17.764133 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:34:17.763992    4964 cpu_manager.go:414] "containerMap: removing container" podUID="c8cba8e6-afc6-459a-89a5-615f812b930c" containerName="maas-api"
Oct 31 01:34:17.764133 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:34:17.763996    4964 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c8cba8e6-afc6-459a-89a5-615f812b930c" containerName="maas-api"
Oct 31 01:34:17.764133 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:34:17.764008    4964 cpu_manager.go:414] "containerMap: removing container" podUID="da53c8d7-cc60-45cd-ab19-365d486484b1" containerName="istio-proxy"
Oct 31 01:34:17.764133 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:34:17.764013    4964 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="da53c8d7-cc60-45cd-ab19-365d486484b1" containerName="istio-proxy"
Oct 31 01:34:17.764133 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:34:17.764022    4964 cpu_manager.go:414] "containerMap: removing container" podUID="45450e7a-2b62-4417-acca-d370251f6d54" containerName="istio-proxy"
Oct 31 01:34:17.764133 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:34:17.764026    4964 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="45450e7a-2b62-4417-acca-d370251f6d54" containerName="istio-proxy"
Oct 31 01:34:17.764133 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:34:17.764036    4964 cpu_manager.go:414] "containerMap: removing container" podUID="145637b0-60a1-4990-8dfe-dd073d9b3929" containerName="istio-proxy"
Oct 31 01:34:17.764133 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:34:17.764039    4964 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="145637b0-60a1-4990-8dfe-dd073d9b3929" containerName="istio-proxy"
Oct 31 01:34:17.764133 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:34:17.764045    4964 cpu_manager.go:414] "containerMap: removing container" podUID="065217f5-7fda-424b-a2e7-fed363588c32" containerName="payload-pre-processing"
Oct 31 01:34:17.764133 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:34:17.764050    4964 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="065217f5-7fda-424b-a2e7-fed363588c32" containerName="payload-pre-processing"
Oct 31 01:34:17.764133 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:34:17.764058    4964 cpu_manager.go:414] "containerMap: removing container" podUID="45450e7a-2b62-4417-acca-d370251f6d54" containerName="istio-proxy"
Oct 31 01:34:17.764133 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:34:17.764064    4964 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="45450e7a-2b62-4417-acca-d370251f6d54" containerName="istio-proxy"
Oct 31 01:34:17.764785 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:34:17.764145    4964 memory_manager.go:365] "RemoveStaleState removing state" podUID="2708a82e-32b4-443c-8cfe-97815ae5f85b" containerName="payload-pre-processing"
Oct 31 01:34:17.764785 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:34:17.764153    4964 memory_manager.go:365] "RemoveStaleState removing state" podUID="f1770ebf-036d-4645-b3f2-c1f14574438f" containerName="payload-pre-processing"
Oct 31 01:34:17.764785 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:34:17.764160    4964 memory_manager.go:365] "RemoveStaleState removing state" podUID="c8cba8e6-afc6-459a-89a5-615f812b930c" containerName="maas-api"
Oct 31 01:34:17.764785 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:34:17.764167    4964 memory_manager.go:365] "RemoveStaleState removing state" podUID="da53c8d7-cc60-45cd-ab19-365d486484b1" containerName="istio-proxy"
Oct 31 01:34:17.764785 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:34:17.764174    4964 memory_manager.go:365] "RemoveStaleState removing state" podUID="45450e7a-2b62-4417-acca-d370251f6d54" containerName="istio-proxy"
Oct 31 01:34:17.764785 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:34:17.764179    4964 memory_manager.go:365] "RemoveStaleState removing state" podUID="145637b0-60a1-4990-8dfe-dd073d9b3929" containerName="istio-proxy"
Oct 31 01:34:17.764785 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:34:17.764185    4964 memory_manager.go:365] "RemoveStaleState removing state" podUID="065217f5-7fda-424b-a2e7-fed363588c32" containerName="payload-pre-processing"
Oct 31 01:34:17.764785 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:34:17.764191    4964 memory_manager.go:365] "RemoveStaleState removing state" podUID="45450e7a-2b62-4417-acca-d370251f6d54" containerName="istio-proxy"
Oct 31 01:34:17.764785 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:34:17.764197    4964 memory_manager.go:365] "RemoveStaleState removing state" podUID="eb13aca8-a24f-4fc8-a51f-3d3f45d739ec" containerName="istio-proxy"
Oct 31 01:34:17.764785 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:34:17.764202    4964 memory_manager.go:365] "RemoveStaleState removing state" podUID="d8871960-8a84-44b1-b24a-a6447283f329" containerName="payload-processing"
Oct 31 01:34:17.764785 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:34:17.764207    4964 memory_manager.go:365] "RemoveStaleState removing state" podUID="a4153b03-5882-419a-bb5c-f1c9592b23fa" containerName="maas-api"
Oct 31 01:34:17.764785 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:34:17.764212    4964 memory_manager.go:365] "RemoveStaleState removing state" podUID="2a53cf91-69f9-4c47-8271-79736ce35af9" containerName="istio-proxy"
Oct 31 01:34:17.764785 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:34:17.764217    4964 memory_manager.go:365] "RemoveStaleState removing state" podUID="fbf50441-0853-4ecf-99ea-38a47e9d5677" containerName="payload-processing"
Oct 31 01:34:17.769183 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:34:17.769158    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-9e5368-openshift-default-5ff7fc6fd7-fp6pw"
Oct 31 01:34:17.805192 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:34:17.804524    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5ab8d4bd-eefa-4a6c-8b3a-34593e689827-istio-token\") pod \"e2e-ipp-a-9e5368-openshift-default-5ff7fc6fd7-fp6pw\" (UID: \"5ab8d4bd-eefa-4a6c-8b3a-34593e689827\") " pod="openshift-ingress/e2e-ipp-a-9e5368-openshift-default-5ff7fc6fd7-fp6pw"
Oct 31 01:34:17.805192 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:34:17.804642    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ab8d4bd-eefa-4a6c-8b3a-34593e689827-credential-socket\") pod \"e2e-ipp-a-9e5368-openshift-default-5ff7fc6fd7-fp6pw\" (UID: \"5ab8d4bd-eefa-4a6c-8b3a-34593e689827\") " pod="openshift-ingress/e2e-ipp-a-9e5368-openshift-default-5ff7fc6fd7-fp6pw"
Oct 31 01:34:17.805192 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:34:17.804671    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ab8d4bd-eefa-4a6c-8b3a-34593e689827-workload-socket\") pod \"e2e-ipp-a-9e5368-openshift-default-5ff7fc6fd7-fp6pw\" (UID: \"5ab8d4bd-eefa-4a6c-8b3a-34593e689827\") " pod="openshift-ingress/e2e-ipp-a-9e5368-openshift-default-5ff7fc6fd7-fp6pw"
Oct 31 01:34:17.805192 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:34:17.804699    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ab8d4bd-eefa-4a6c-8b3a-34593e689827-istio-podinfo\") pod \"e2e-ipp-a-9e5368-openshift-default-5ff7fc6fd7-fp6pw\" (UID: \"5ab8d4bd-eefa-4a6c-8b3a-34593e689827\") " pod="openshift-ingress/e2e-ipp-a-9e5368-openshift-default-5ff7fc6fd7-fp6pw"
Oct 31 01:34:17.805192 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:34:17.804779    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-27vcq\" (UniqueName: \"kubernetes.io/projected/5ab8d4bd-eefa-4a6c-8b3a-34593e689827-kube-api-access-27vcq\") pod \"e2e-ipp-a-9e5368-openshift-default-5ff7fc6fd7-fp6pw\" (UID: \"5ab8d4bd-eefa-4a6c-8b3a-34593e689827\") " pod="openshift-ingress/e2e-ipp-a-9e5368-openshift-default-5ff7fc6fd7-fp6pw"
Oct 31 01:34:17.805192 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:34:17.804916    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ab8d4bd-eefa-4a6c-8b3a-34593e689827-istio-envoy\") pod \"e2e-ipp-a-9e5368-openshift-default-5ff7fc6fd7-fp6pw\" (UID: \"5ab8d4bd-eefa-4a6c-8b3a-34593e689827\") " pod="openshift-ingress/e2e-ipp-a-9e5368-openshift-default-5ff7fc6fd7-fp6pw"
Oct 31 01:34:17.805192 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:34:17.804952    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ab8d4bd-eefa-4a6c-8b3a-34593e689827-istio-data\") pod \"e2e-ipp-a-9e5368-openshift-default-5ff7fc6fd7-fp6pw\" (UID: \"5ab8d4bd-eefa-4a6c-8b3a-34593e689827\") " pod="openshift-ingress/e2e-ipp-a-9e5368-openshift-default-5ff7fc6fd7-fp6pw"
Oct 31 01:34:17.805192 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:34:17.804977    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ab8d4bd-eefa-4a6c-8b3a-34593e689827-workload-certs\") pod \"e2e-ipp-a-9e5368-openshift-default-5ff7fc6fd7-fp6pw\" (UID: \"5ab8d4bd-eefa-4a6c-8b3a-34593e689827\") " pod="openshift-ingress/e2e-ipp-a-9e5368-openshift-default-5ff7fc6fd7-fp6pw"
Oct 31 01:34:17.805192 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:34:17.805011    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ab8d4bd-eefa-4a6c-8b3a-34593e689827-istiod-ca-cert\") pod \"e2e-ipp-a-9e5368-openshift-default-5ff7fc6fd7-fp6pw\" (UID: \"5ab8d4bd-eefa-4a6c-8b3a-34593e689827\") " pod="openshift-ingress/e2e-ipp-a-9e5368-openshift-default-5ff7fc6fd7-fp6pw"
Oct 31 01:34:17.823337 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:34:17.822553    4964 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-9e5368-openshift-default-dockercfg-bk78g\""
Oct 31 01:34:17.898770 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:34:17.898732    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:34:17.905211 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:34:17.905166    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-9e5368-openshift-default-5ff7fc6fd7-fp6pw"]
Oct 31 01:34:17.906099 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:34:17.905963    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-27vcq\" (UniqueName: \"kubernetes.io/projected/5ab8d4bd-eefa-4a6c-8b3a-34593e689827-kube-api-access-27vcq\") pod \"e2e-ipp-a-9e5368-openshift-default-5ff7fc6fd7-fp6pw\" (UID: \"5ab8d4bd-eefa-4a6c-8b3a-34593e689827\") " pod="openshift-ingress/e2e-ipp-a-9e5368-openshift-default-5ff7fc6fd7-fp6pw"
Oct 31 01:34:17.906099 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:34:17.906057    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ab8d4bd-eefa-4a6c-8b3a-34593e689827-istio-envoy\") pod \"e2e-ipp-a-9e5368-openshift-default-5ff7fc6fd7-fp6pw\" (UID: \"5ab8d4bd-eefa-4a6c-8b3a-34593e689827\") " pod="openshift-ingress/e2e-ipp-a-9e5368-openshift-default-5ff7fc6fd7-fp6pw"
Oct 31 01:34:17.906258 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:34:17.906101    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ab8d4bd-eefa-4a6c-8b3a-34593e689827-istio-data\") pod \"e2e-ipp-a-9e5368-openshift-default-5ff7fc6fd7-fp6pw\" (UID: \"5ab8d4bd-eefa-4a6c-8b3a-34593e689827\") " pod="openshift-ingress/e2e-ipp-a-9e5368-openshift-default-5ff7fc6fd7-fp6pw"
Oct 31 01:34:17.906258 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:34:17.906129    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ab8d4bd-eefa-4a6c-8b3a-34593e689827-workload-certs\") pod \"e2e-ipp-a-9e5368-openshift-default-5ff7fc6fd7-fp6pw\" (UID: \"5ab8d4bd-eefa-4a6c-8b3a-34593e689827\") " pod="openshift-ingress/e2e-ipp-a-9e5368-openshift-default-5ff7fc6fd7-fp6pw"
Oct 31 01:34:17.906258 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:34:17.906163    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ab8d4bd-eefa-4a6c-8b3a-34593e689827-istiod-ca-cert\") pod \"e2e-ipp-a-9e5368-openshift-default-5ff7fc6fd7-fp6pw\" (UID: \"5ab8d4bd-eefa-4a6c-8b3a-34593e689827\") " pod="openshift-ingress/e2e-ipp-a-9e5368-openshift-default-5ff7fc6fd7-fp6pw"
Oct 31 01:34:17.906439 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:34:17.906300    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5ab8d4bd-eefa-4a6c-8b3a-34593e689827-istio-token\") pod \"e2e-ipp-a-9e5368-openshift-default-5ff7fc6fd7-fp6pw\" (UID: \"5ab8d4bd-eefa-4a6c-8b3a-34593e689827\") " pod="openshift-ingress/e2e-ipp-a-9e5368-openshift-default-5ff7fc6fd7-fp6pw"
Oct 31 01:34:17.906439 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:34:17.906427    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ab8d4bd-eefa-4a6c-8b3a-34593e689827-credential-socket\") pod \"e2e-ipp-a-9e5368-openshift-default-5ff7fc6fd7-fp6pw\" (UID: \"5ab8d4bd-eefa-4a6c-8b3a-34593e689827\") " pod="openshift-ingress/e2e-ipp-a-9e5368-openshift-default-5ff7fc6fd7-fp6pw"
Oct 31 01:34:17.906545 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:34:17.906455    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ab8d4bd-eefa-4a6c-8b3a-34593e689827-workload-socket\") pod \"e2e-ipp-a-9e5368-openshift-default-5ff7fc6fd7-fp6pw\" (UID: \"5ab8d4bd-eefa-4a6c-8b3a-34593e689827\") " pod="openshift-ingress/e2e-ipp-a-9e5368-openshift-default-5ff7fc6fd7-fp6pw"
Oct 31 01:34:17.906545 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:34:17.906488    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ab8d4bd-eefa-4a6c-8b3a-34593e689827-istio-podinfo\") pod \"e2e-ipp-a-9e5368-openshift-default-5ff7fc6fd7-fp6pw\" (UID: \"5ab8d4bd-eefa-4a6c-8b3a-34593e689827\") " pod="openshift-ingress/e2e-ipp-a-9e5368-openshift-default-5ff7fc6fd7-fp6pw"
Oct 31 01:34:17.907925 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:34:17.907904    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ab8d4bd-eefa-4a6c-8b3a-34593e689827-workload-certs\") pod \"e2e-ipp-a-9e5368-openshift-default-5ff7fc6fd7-fp6pw\" (UID: \"5ab8d4bd-eefa-4a6c-8b3a-34593e689827\") " pod="openshift-ingress/e2e-ipp-a-9e5368-openshift-default-5ff7fc6fd7-fp6pw"
Oct 31 01:34:17.909264 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:34:17.909239    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ab8d4bd-eefa-4a6c-8b3a-34593e689827-istio-podinfo\") pod \"e2e-ipp-a-9e5368-openshift-default-5ff7fc6fd7-fp6pw\" (UID: \"5ab8d4bd-eefa-4a6c-8b3a-34593e689827\") " pod="openshift-ingress/e2e-ipp-a-9e5368-openshift-default-5ff7fc6fd7-fp6pw"
Oct 31 01:34:17.910569 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:34:17.909747    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ab8d4bd-eefa-4a6c-8b3a-34593e689827-istiod-ca-cert\") pod \"e2e-ipp-a-9e5368-openshift-default-5ff7fc6fd7-fp6pw\" (UID: \"5ab8d4bd-eefa-4a6c-8b3a-34593e689827\") " pod="openshift-ingress/e2e-ipp-a-9e5368-openshift-default-5ff7fc6fd7-fp6pw"
Oct 31 01:34:17.910569 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:34:17.910116    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ab8d4bd-eefa-4a6c-8b3a-34593e689827-credential-socket\") pod \"e2e-ipp-a-9e5368-openshift-default-5ff7fc6fd7-fp6pw\" (UID: \"5ab8d4bd-eefa-4a6c-8b3a-34593e689827\") " pod="openshift-ingress/e2e-ipp-a-9e5368-openshift-default-5ff7fc6fd7-fp6pw"
Oct 31 01:34:17.910569 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:34:17.910273    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ab8d4bd-eefa-4a6c-8b3a-34593e689827-workload-socket\") pod \"e2e-ipp-a-9e5368-openshift-default-5ff7fc6fd7-fp6pw\" (UID: \"5ab8d4bd-eefa-4a6c-8b3a-34593e689827\") " pod="openshift-ingress/e2e-ipp-a-9e5368-openshift-default-5ff7fc6fd7-fp6pw"
Oct 31 01:34:17.910569 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:34:17.910517    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ab8d4bd-eefa-4a6c-8b3a-34593e689827-istio-data\") pod \"e2e-ipp-a-9e5368-openshift-default-5ff7fc6fd7-fp6pw\" (UID: \"5ab8d4bd-eefa-4a6c-8b3a-34593e689827\") " pod="openshift-ingress/e2e-ipp-a-9e5368-openshift-default-5ff7fc6fd7-fp6pw"
Oct 31 01:34:17.912199 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:34:17.912138    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ab8d4bd-eefa-4a6c-8b3a-34593e689827-istio-envoy\") pod \"e2e-ipp-a-9e5368-openshift-default-5ff7fc6fd7-fp6pw\" (UID: \"5ab8d4bd-eefa-4a6c-8b3a-34593e689827\") " pod="openshift-ingress/e2e-ipp-a-9e5368-openshift-default-5ff7fc6fd7-fp6pw"
Oct 31 01:34:17.988150 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:34:17.988119    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-27vcq\" (UniqueName: \"kubernetes.io/projected/5ab8d4bd-eefa-4a6c-8b3a-34593e689827-kube-api-access-27vcq\") pod \"e2e-ipp-a-9e5368-openshift-default-5ff7fc6fd7-fp6pw\" (UID: \"5ab8d4bd-eefa-4a6c-8b3a-34593e689827\") " pod="openshift-ingress/e2e-ipp-a-9e5368-openshift-default-5ff7fc6fd7-fp6pw"
Oct 31 01:34:17.988964 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:34:17.988541    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5ab8d4bd-eefa-4a6c-8b3a-34593e689827-istio-token\") pod \"e2e-ipp-a-9e5368-openshift-default-5ff7fc6fd7-fp6pw\" (UID: \"5ab8d4bd-eefa-4a6c-8b3a-34593e689827\") " pod="openshift-ingress/e2e-ipp-a-9e5368-openshift-default-5ff7fc6fd7-fp6pw"
Oct 31 01:34:18.162467 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:34:18.162378    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-9e5368-openshift-default-5ff7fc6fd7-fp6pw"
Oct 31 01:34:18.420205 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:34:18.420096    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:34:18.603457 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:34:18.603429    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-9e5368-openshift-default-5ff7fc6fd7-fp6pw"]
Oct 31 01:34:18.604200 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:34:18.604174    4964 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ab8d4bd_eefa_4a6c_8b3a_34593e689827.slice/crio-44e65863854dbee31a057d437c33450d426f61144222f7bc849fee49347591dc WatchSource:0}: Error finding container 44e65863854dbee31a057d437c33450d426f61144222f7bc849fee49347591dc: Status 404 returned error can't find the container with id 44e65863854dbee31a057d437c33450d426f61144222f7bc849fee49347591dc
Oct 31 01:34:18.606619 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:34:18.606587    4964 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 01:34:18.606713 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:34:18.606649    4964 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 01:34:18.606713 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:34:18.606676    4964 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 01:34:18.814442 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:34:18.814408    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-9e5368-openshift-default-5ff7fc6fd7-fp6pw" event={"ID":"5ab8d4bd-eefa-4a6c-8b3a-34593e689827","Type":"ContainerStarted","Data":"84b91dfbaff657c035aaadf786b3fadae6fc10b902a4346df8a0db03a6b1abcd"}
Oct 31 01:34:18.814943 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:34:18.814916    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-9e5368-openshift-default-5ff7fc6fd7-fp6pw" event={"ID":"5ab8d4bd-eefa-4a6c-8b3a-34593e689827","Type":"ContainerStarted","Data":"44e65863854dbee31a057d437c33450d426f61144222f7bc849fee49347591dc"}
Oct 31 01:34:18.889786 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:34:18.889755    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:34:18.933728 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:34:18.933664    4964 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-9e5368-openshift-default-5ff7fc6fd7-fp6pw" podStartSLOduration=1.933643534 podStartE2EDuration="1.933643534s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:34:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:34:18.931848703 +0000 UTC m=+1827.408415083" watchObservedRunningTime="2026-10-31 01:34:18.933643534 +0000 UTC m=+1827.410209919"
Oct 31 01:34:19.162595 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:34:19.162507    4964 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-9e5368-openshift-default-5ff7fc6fd7-fp6pw"
Oct 31 01:34:19.168819 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:34:19.168796    4964 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-9e5368-openshift-default-5ff7fc6fd7-fp6pw"
Oct 31 01:34:19.821444 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:34:19.821413    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-9e5368-openshift-default-5ff7fc6fd7-fp6pw"
Oct 31 01:34:19.822214 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:34:19.822197    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-9e5368-openshift-default-5ff7fc6fd7-fp6pw"
Oct 31 01:34:22.927099 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:34:22.927069    4964 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-42c96d-6d9dd859d6-z258p"]
Oct 31 01:34:22.927570 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:34:22.927293    4964 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-42c96d-6d9dd859d6-z258p" podUID="0d4e0035-30d3-472e-96e5-04df99e26e48" containerName="maas-api" containerID="cri-o://d078e600265fc8fafd5da89422fad7f6151a3b69c8f3786419f67837836ef3a6" gracePeriod=30
Oct 31 01:34:23.065934 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:34:23.065904    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:34:23.102863 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:34:23.102824    4964 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-42c96d-6d9dd859d6-z258p" podUID="0d4e0035-30d3-472e-96e5-04df99e26e48" containerName="maas-api" probeResult="failure" output="Get \"https://10.132.0.114:8443/health\": dial tcp 10.132.0.114:8443: connect: connection refused"
Oct 31 01:34:23.253974 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:34:23.253944    4964 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-42c96d-6d9dd859d6-z258p"
Oct 31 01:34:23.372725 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:34:23.372690    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0d4e0035-30d3-472e-96e5-04df99e26e48-metrics-certs\" (UniqueName: \"kubernetes.io/secret/0d4e0035-30d3-472e-96e5-04df99e26e48-metrics-certs\") pod \"0d4e0035-30d3-472e-96e5-04df99e26e48\" (UID: \"0d4e0035-30d3-472e-96e5-04df99e26e48\") "
Oct 31 01:34:23.372884 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:34:23.372774    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0d4e0035-30d3-472e-96e5-04df99e26e48-kube-api-access-54vq4\" (UniqueName: \"kubernetes.io/projected/0d4e0035-30d3-472e-96e5-04df99e26e48-kube-api-access-54vq4\") pod \"0d4e0035-30d3-472e-96e5-04df99e26e48\" (UID: \"0d4e0035-30d3-472e-96e5-04df99e26e48\") "
Oct 31 01:34:23.372996 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:34:23.372981    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0d4e0035-30d3-472e-96e5-04df99e26e48-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0d4e0035-30d3-472e-96e5-04df99e26e48-maas-api-tls\") pod \"0d4e0035-30d3-472e-96e5-04df99e26e48\" (UID: \"0d4e0035-30d3-472e-96e5-04df99e26e48\") "
Oct 31 01:34:23.373040 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:34:23.373009    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0d4e0035-30d3-472e-96e5-04df99e26e48-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0d4e0035-30d3-472e-96e5-04df99e26e48-openshift-service-ca\") pod \"0d4e0035-30d3-472e-96e5-04df99e26e48\" (UID: \"0d4e0035-30d3-472e-96e5-04df99e26e48\") "
Oct 31 01:34:23.373473 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:34:23.373433    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0d4e0035-30d3-472e-96e5-04df99e26e48-openshift-service-ca" pod "0d4e0035-30d3-472e-96e5-04df99e26e48" (UID: "0d4e0035-30d3-472e-96e5-04df99e26e48"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:34:23.373603 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:34:23.373589    4964 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0d4e0035-30d3-472e-96e5-04df99e26e48-openshift-service-ca\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:34:23.375053 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:34:23.375029    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0d4e0035-30d3-472e-96e5-04df99e26e48-metrics-certs" pod "0d4e0035-30d3-472e-96e5-04df99e26e48" (UID: "0d4e0035-30d3-472e-96e5-04df99e26e48"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:34:23.376336 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:34:23.375583    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d4e0035-30d3-472e-96e5-04df99e26e48-kube-api-access-54vq4" pod "0d4e0035-30d3-472e-96e5-04df99e26e48" (UID: "0d4e0035-30d3-472e-96e5-04df99e26e48"). InnerVolumeSpecName "kube-api-access-54vq4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:34:23.376336 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:34:23.375736    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0d4e0035-30d3-472e-96e5-04df99e26e48-maas-api-tls" pod "0d4e0035-30d3-472e-96e5-04df99e26e48" (UID: "0d4e0035-30d3-472e-96e5-04df99e26e48"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:34:23.474861 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:34:23.474827    4964 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0d4e0035-30d3-472e-96e5-04df99e26e48-metrics-certs\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:34:23.474861 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:34:23.474859    4964 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-54vq4\" (UniqueName: \"kubernetes.io/projected/0d4e0035-30d3-472e-96e5-04df99e26e48-kube-api-access-54vq4\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:34:23.475057 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:34:23.474874    4964 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0d4e0035-30d3-472e-96e5-04df99e26e48-maas-api-tls\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:34:23.853143 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:34:23.852618    4964 generic.go:356] "Generic (PLEG): container finished" podID="0d4e0035-30d3-472e-96e5-04df99e26e48" containerID="d078e600265fc8fafd5da89422fad7f6151a3b69c8f3786419f67837836ef3a6" exitCode=0
Oct 31 01:34:23.853143 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:34:23.852661    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-42c96d-6d9dd859d6-z258p" event={"ID":"0d4e0035-30d3-472e-96e5-04df99e26e48","Type":"ContainerDied","Data":"d078e600265fc8fafd5da89422fad7f6151a3b69c8f3786419f67837836ef3a6"}
Oct 31 01:34:23.853143 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:34:23.852713    4964 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-42c96d-6d9dd859d6-z258p"
Oct 31 01:34:23.853143 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:34:23.852729    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-42c96d-6d9dd859d6-z258p" event={"ID":"0d4e0035-30d3-472e-96e5-04df99e26e48","Type":"ContainerDied","Data":"e098cea11a8b504e646a0f9f6e0f8e9c691e1374e3af9896aa2f1c7b91745c2b"}
Oct 31 01:34:23.853143 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:34:23.852756    4964 scope.go:122] "RemoveContainer" containerID="d078e600265fc8fafd5da89422fad7f6151a3b69c8f3786419f67837836ef3a6"
Oct 31 01:34:23.908555 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:34:23.889740    4964 scope.go:122] "RemoveContainer" containerID="d078e600265fc8fafd5da89422fad7f6151a3b69c8f3786419f67837836ef3a6"
Oct 31 01:34:23.908555 ip-10-0-133-197 kubenswrapper[4964]: E1031 01:34:23.890124    4964 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d078e600265fc8fafd5da89422fad7f6151a3b69c8f3786419f67837836ef3a6\": container with ID starting with d078e600265fc8fafd5da89422fad7f6151a3b69c8f3786419f67837836ef3a6 not found: ID does not exist" containerID="d078e600265fc8fafd5da89422fad7f6151a3b69c8f3786419f67837836ef3a6"
Oct 31 01:34:23.908555 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:34:23.890157    4964 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d078e600265fc8fafd5da89422fad7f6151a3b69c8f3786419f67837836ef3a6"} err="failed to get container status \"d078e600265fc8fafd5da89422fad7f6151a3b69c8f3786419f67837836ef3a6\": rpc error: code = NotFound desc = could not find container \"d078e600265fc8fafd5da89422fad7f6151a3b69c8f3786419f67837836ef3a6\": container with ID starting with d078e600265fc8fafd5da89422fad7f6151a3b69c8f3786419f67837836ef3a6 not found: ID does not exist"
Oct 31 01:34:24.021433 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:34:24.021399    4964 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-42c96d-6d9dd859d6-z258p"]
Oct 31 01:34:24.068936 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:34:24.068860    4964 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-42c96d-6d9dd859d6-z258p"]
Oct 31 01:34:25.025841 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:34:25.025806    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:34:25.197519 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:34:25.197478    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:34:25.379530 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:34:25.379438    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:34:25.777060 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:34:25.777029    4964 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0d4e0035-30d3-472e-96e5-04df99e26e48" path="/var/lib/kubelet/pods/0d4e0035-30d3-472e-96e5-04df99e26e48/volumes"
Oct 31 01:34:25.910822 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:34:25.910782    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:34:30.242715 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:34:30.242621    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:34:30.407566 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:34:30.407531    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:34:30.590339 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:34:30.590234    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:34:30.768127 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:34:30.768094    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:34:32.082035 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:34:32.082004    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:34:32.308390 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:34:32.308351    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:34:32.416202 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:34:32.416108    4964 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-42c96d-openshift-default-69697959d9-p8ljq"]
Oct 31 01:34:32.416450 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:34:32.416343    4964 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-42c96d-openshift-default-69697959d9-p8ljq" podUID="50f61fea-7b83-48aa-ad74-ea2be71f683b" containerName="istio-proxy" containerID="cri-o://e949c066245e8dcc03f440670b60a4128e1960e39914711ffcf7327c1fcdf930" gracePeriod=30
Oct 31 01:34:32.489131 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:34:32.488721    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:34:33.739869 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:34:33.739821    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:34:33.965600 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:34:33.965560    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:34:34.116924 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:34:34.116838    4964 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-9e5368-7f65f554f5-g7mgs"]
Oct 31 01:34:34.117470 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:34:34.117454    4964 cpu_manager.go:414] "containerMap: removing container" podUID="0d4e0035-30d3-472e-96e5-04df99e26e48" containerName="maas-api"
Oct 31 01:34:34.117572 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:34:34.117471    4964 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0d4e0035-30d3-472e-96e5-04df99e26e48" containerName="maas-api"
Oct 31 01:34:34.117628 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:34:34.117585    4964 memory_manager.go:365] "RemoveStaleState removing state" podUID="eb13aca8-a24f-4fc8-a51f-3d3f45d739ec" containerName="istio-proxy"
Oct 31 01:34:34.117628 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:34:34.117603    4964 memory_manager.go:365] "RemoveStaleState removing state" podUID="0d4e0035-30d3-472e-96e5-04df99e26e48" containerName="maas-api"
Oct 31 01:34:34.122333 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:34:34.122295    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9e5368-7f65f554f5-g7mgs"
Oct 31 01:34:34.174024 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:34:34.173991    4964 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-9e5368\""
Oct 31 01:34:34.245894 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:34:34.245859    4964 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-9e5368-dockercfg-gh5s2\""
Oct 31 01:34:34.262520 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:34:34.262110    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hff9r\" (UniqueName: \"kubernetes.io/projected/16f5f3bb-ce5c-4e0d-aaca-8c796a1a9b1b-kube-api-access-hff9r\") pod \"payload-pre-processing-e2e-ipp-a-9e5368-7f65f554f5-g7mgs\" (UID: \"16f5f3bb-ce5c-4e0d-aaca-8c796a1a9b1b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9e5368-7f65f554f5-g7mgs"
Oct 31 01:34:34.262800 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:34:34.262772    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/16f5f3bb-ce5c-4e0d-aaca-8c796a1a9b1b-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-9e5368-7f65f554f5-g7mgs\" (UID: \"16f5f3bb-ce5c-4e0d-aaca-8c796a1a9b1b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9e5368-7f65f554f5-g7mgs"
Oct 31 01:34:34.292445 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:34:34.292400    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-9e5368-7f65f554f5-g7mgs"]
Oct 31 01:34:34.304454 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:34:34.304412    4964 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-9e5368-7b876c5d49-4zpqb"]
Oct 31 01:34:34.310298 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:34:34.310262    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-9e5368-7b876c5d49-4zpqb"
Oct 31 01:34:34.328991 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:34:34.328966    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-9e5368-7b876c5d49-4zpqb"]
Oct 31 01:34:34.364903 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:34:34.364873    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hff9r\" (UniqueName: \"kubernetes.io/projected/16f5f3bb-ce5c-4e0d-aaca-8c796a1a9b1b-kube-api-access-hff9r\") pod \"payload-pre-processing-e2e-ipp-a-9e5368-7f65f554f5-g7mgs\" (UID: \"16f5f3bb-ce5c-4e0d-aaca-8c796a1a9b1b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9e5368-7f65f554f5-g7mgs"
Oct 31 01:34:34.365077 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:34:34.364911    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/16f5f3bb-ce5c-4e0d-aaca-8c796a1a9b1b-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-9e5368-7f65f554f5-g7mgs\" (UID: \"16f5f3bb-ce5c-4e0d-aaca-8c796a1a9b1b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9e5368-7f65f554f5-g7mgs"
Oct 31 01:34:34.365610 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:34:34.365590    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/16f5f3bb-ce5c-4e0d-aaca-8c796a1a9b1b-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-9e5368-7f65f554f5-g7mgs\" (UID: \"16f5f3bb-ce5c-4e0d-aaca-8c796a1a9b1b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9e5368-7f65f554f5-g7mgs"
Oct 31 01:34:34.421238 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:34:34.421171    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hff9r\" (UniqueName: \"kubernetes.io/projected/16f5f3bb-ce5c-4e0d-aaca-8c796a1a9b1b-kube-api-access-hff9r\") pod \"payload-pre-processing-e2e-ipp-a-9e5368-7f65f554f5-g7mgs\" (UID: \"16f5f3bb-ce5c-4e0d-aaca-8c796a1a9b1b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9e5368-7f65f554f5-g7mgs"
Oct 31 01:34:34.437750 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:34:34.437723    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9e5368-7f65f554f5-g7mgs"
Oct 31 01:34:34.466436 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:34:34.466405    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qsphl\" (UniqueName: \"kubernetes.io/projected/8fb32c6d-7453-44d3-a54e-5470f274d555-kube-api-access-qsphl\") pod \"payload-processing-e2e-ipp-a-9e5368-7b876c5d49-4zpqb\" (UID: \"8fb32c6d-7453-44d3-a54e-5470f274d555\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-9e5368-7b876c5d49-4zpqb"
Oct 31 01:34:34.466721 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:34:34.466699    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8fb32c6d-7453-44d3-a54e-5470f274d555-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-9e5368-7b876c5d49-4zpqb\" (UID: \"8fb32c6d-7453-44d3-a54e-5470f274d555\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-9e5368-7b876c5d49-4zpqb"
Oct 31 01:34:34.567804 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:34:34.567760    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8fb32c6d-7453-44d3-a54e-5470f274d555-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-9e5368-7b876c5d49-4zpqb\" (UID: \"8fb32c6d-7453-44d3-a54e-5470f274d555\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-9e5368-7b876c5d49-4zpqb"
Oct 31 01:34:34.567961 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:34:34.567920    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qsphl\" (UniqueName: \"kubernetes.io/projected/8fb32c6d-7453-44d3-a54e-5470f274d555-kube-api-access-qsphl\") pod \"payload-processing-e2e-ipp-a-9e5368-7b876c5d49-4zpqb\" (UID: \"8fb32c6d-7453-44d3-a54e-5470f274d555\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-9e5368-7b876c5d49-4zpqb"
Oct 31 01:34:34.568755 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:34:34.568729    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8fb32c6d-7453-44d3-a54e-5470f274d555-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-9e5368-7b876c5d49-4zpqb\" (UID: \"8fb32c6d-7453-44d3-a54e-5470f274d555\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-9e5368-7b876c5d49-4zpqb"
Oct 31 01:34:34.624175 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:34:34.624146    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qsphl\" (UniqueName: \"kubernetes.io/projected/8fb32c6d-7453-44d3-a54e-5470f274d555-kube-api-access-qsphl\") pod \"payload-processing-e2e-ipp-a-9e5368-7b876c5d49-4zpqb\" (UID: \"8fb32c6d-7453-44d3-a54e-5470f274d555\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-9e5368-7b876c5d49-4zpqb"
Oct 31 01:34:34.628398 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:34:34.628368    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-9e5368-7b876c5d49-4zpqb"
Oct 31 01:34:34.819845 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:34:34.819811    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-9e5368-7f65f554f5-g7mgs"]
Oct 31 01:34:34.821697 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:34:34.821665    4964 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod16f5f3bb_ce5c_4e0d_aaca_8c796a1a9b1b.slice/crio-ddab1c82c3a9331ffc53ed13d297cb8b51a51783e7af0297742a382789805d80 WatchSource:0}: Error finding container ddab1c82c3a9331ffc53ed13d297cb8b51a51783e7af0297742a382789805d80: Status 404 returned error can't find the container with id ddab1c82c3a9331ffc53ed13d297cb8b51a51783e7af0297742a382789805d80
Oct 31 01:34:34.944015 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:34:34.943918    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9e5368-7f65f554f5-g7mgs" event={"ID":"16f5f3bb-ce5c-4e0d-aaca-8c796a1a9b1b","Type":"ContainerStarted","Data":"9433d37d5ddfe6011f14af53bed242ffa5cf296c97ed542d8ebd6fdcb087e19b"}
Oct 31 01:34:34.944015 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:34:34.943979    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9e5368-7f65f554f5-g7mgs" event={"ID":"16f5f3bb-ce5c-4e0d-aaca-8c796a1a9b1b","Type":"ContainerStarted","Data":"ddab1c82c3a9331ffc53ed13d297cb8b51a51783e7af0297742a382789805d80"}
Oct 31 01:34:34.944232 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:34:34.944143    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9e5368-7f65f554f5-g7mgs"
Oct 31 01:34:35.008900 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:34:35.008876    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-9e5368-7b876c5d49-4zpqb"]
Oct 31 01:34:35.011061 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:34:35.011035    4964 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8fb32c6d_7453_44d3_a54e_5470f274d555.slice/crio-7d7da6bbad9c862ccb5a423f6b451e00d6d55684942484ceb88b2424284ee8ff WatchSource:0}: Error finding container 7d7da6bbad9c862ccb5a423f6b451e00d6d55684942484ceb88b2424284ee8ff: Status 404 returned error can't find the container with id 7d7da6bbad9c862ccb5a423f6b451e00d6d55684942484ceb88b2424284ee8ff
Oct 31 01:34:35.059470 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:34:35.059427    4964 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9e5368-7f65f554f5-g7mgs" podStartSLOduration=1.059412961 podStartE2EDuration="1.059412961s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:34:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:34:35.056959872 +0000 UTC m=+1843.533526253" watchObservedRunningTime="2026-10-31 01:34:35.059412961 +0000 UTC m=+1843.535979341"
Oct 31 01:34:35.259032 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:34:35.259001    4964 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-5e916d-547c767844-llcts"]
Oct 31 01:34:35.263292 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:34:35.263275    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-5e916d-547c767844-llcts"
Oct 31 01:34:35.279275 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:34:35.279240    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7c420de6-67b9-4958-8083-ebd8bd939418-maas-api-tls\") pod \"maas-api-e2e-shared-a-w1-5e916d-547c767844-llcts\" (UID: \"7c420de6-67b9-4958-8083-ebd8bd939418\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-5e916d-547c767844-llcts"
Oct 31 01:34:35.279425 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:34:35.279352    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7c420de6-67b9-4958-8083-ebd8bd939418-metrics-certs\") pod \"maas-api-e2e-shared-a-w1-5e916d-547c767844-llcts\" (UID: \"7c420de6-67b9-4958-8083-ebd8bd939418\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-5e916d-547c767844-llcts"
Oct 31 01:34:35.279688 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:34:35.279652    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xxhbl\" (UniqueName: \"kubernetes.io/projected/7c420de6-67b9-4958-8083-ebd8bd939418-kube-api-access-xxhbl\") pod \"maas-api-e2e-shared-a-w1-5e916d-547c767844-llcts\" (UID: \"7c420de6-67b9-4958-8083-ebd8bd939418\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-5e916d-547c767844-llcts"
Oct 31 01:34:35.279816 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:34:35.279740    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7c420de6-67b9-4958-8083-ebd8bd939418-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w1-5e916d-547c767844-llcts\" (UID: \"7c420de6-67b9-4958-8083-ebd8bd939418\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-5e916d-547c767844-llcts"
Oct 31 01:34:35.316587 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:34:35.316500    4964 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w1-5e916d-serving-cert\""
Oct 31 01:34:35.321932 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:34:35.321902    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-5e916d-547c767844-llcts"]
Oct 31 01:34:35.380977 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:34:35.380941    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xxhbl\" (UniqueName: \"kubernetes.io/projected/7c420de6-67b9-4958-8083-ebd8bd939418-kube-api-access-xxhbl\") pod \"maas-api-e2e-shared-a-w1-5e916d-547c767844-llcts\" (UID: \"7c420de6-67b9-4958-8083-ebd8bd939418\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-5e916d-547c767844-llcts"
Oct 31 01:34:35.381131 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:34:35.380986    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7c420de6-67b9-4958-8083-ebd8bd939418-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w1-5e916d-547c767844-llcts\" (UID: \"7c420de6-67b9-4958-8083-ebd8bd939418\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-5e916d-547c767844-llcts"
Oct 31 01:34:35.381131 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:34:35.381074    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7c420de6-67b9-4958-8083-ebd8bd939418-maas-api-tls\") pod \"maas-api-e2e-shared-a-w1-5e916d-547c767844-llcts\" (UID: \"7c420de6-67b9-4958-8083-ebd8bd939418\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-5e916d-547c767844-llcts"
Oct 31 01:34:35.381131 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:34:35.381123    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7c420de6-67b9-4958-8083-ebd8bd939418-metrics-certs\") pod \"maas-api-e2e-shared-a-w1-5e916d-547c767844-llcts\" (UID: \"7c420de6-67b9-4958-8083-ebd8bd939418\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-5e916d-547c767844-llcts"
Oct 31 01:34:35.381265 ip-10-0-133-197 kubenswrapper[4964]: E1031 01:34:35.381243    4964 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-5e916d-serving-cert: secret "maas-api-e2e-shared-a-w1-5e916d-serving-cert" not found
Oct 31 01:34:35.381365 ip-10-0-133-197 kubenswrapper[4964]: E1031 01:34:35.381353    4964 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7c420de6-67b9-4958-8083-ebd8bd939418-maas-api-tls podName:7c420de6-67b9-4958-8083-ebd8bd939418 nodeName:}" failed. No retries permitted until 2026-10-31 01:34:35.881305032 +0000 UTC m=+1844.357871410 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/7c420de6-67b9-4958-8083-ebd8bd939418-maas-api-tls") pod "maas-api-e2e-shared-a-w1-5e916d-547c767844-llcts" (UID: "7c420de6-67b9-4958-8083-ebd8bd939418") : secret "maas-api-e2e-shared-a-w1-5e916d-serving-cert" not found
Oct 31 01:34:35.381765 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:34:35.381744    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7c420de6-67b9-4958-8083-ebd8bd939418-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w1-5e916d-547c767844-llcts\" (UID: \"7c420de6-67b9-4958-8083-ebd8bd939418\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-5e916d-547c767844-llcts"
Oct 31 01:34:35.383767 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:34:35.383749    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7c420de6-67b9-4958-8083-ebd8bd939418-metrics-certs\") pod \"maas-api-e2e-shared-a-w1-5e916d-547c767844-llcts\" (UID: \"7c420de6-67b9-4958-8083-ebd8bd939418\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-5e916d-547c767844-llcts"
Oct 31 01:34:35.437872 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:34:35.437842    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xxhbl\" (UniqueName: \"kubernetes.io/projected/7c420de6-67b9-4958-8083-ebd8bd939418-kube-api-access-xxhbl\") pod \"maas-api-e2e-shared-a-w1-5e916d-547c767844-llcts\" (UID: \"7c420de6-67b9-4958-8083-ebd8bd939418\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-5e916d-547c767844-llcts"
Oct 31 01:34:35.494991 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:34:35.494955    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:34:35.722428 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:34:35.719241    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:34:35.888595 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:34:35.888562    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7c420de6-67b9-4958-8083-ebd8bd939418-maas-api-tls\") pod \"maas-api-e2e-shared-a-w1-5e916d-547c767844-llcts\" (UID: \"7c420de6-67b9-4958-8083-ebd8bd939418\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-5e916d-547c767844-llcts"
Oct 31 01:34:35.891431 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:34:35.891403    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7c420de6-67b9-4958-8083-ebd8bd939418-maas-api-tls\") pod \"maas-api-e2e-shared-a-w1-5e916d-547c767844-llcts\" (UID: \"7c420de6-67b9-4958-8083-ebd8bd939418\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-5e916d-547c767844-llcts"
Oct 31 01:34:35.894100 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:34:35.894066    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:34:35.952888 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:34:35.952849    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-9e5368-7b876c5d49-4zpqb" event={"ID":"8fb32c6d-7453-44d3-a54e-5470f274d555","Type":"ContainerStarted","Data":"7a0be0aa3e5802ac6b06744841da7df51b1b27de45071d47ac935c4472fcefcd"}
Oct 31 01:34:35.953063 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:34:35.952923    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-9e5368-7b876c5d49-4zpqb"
Oct 31 01:34:35.953063 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:34:35.952943    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-9e5368-7b876c5d49-4zpqb" event={"ID":"8fb32c6d-7453-44d3-a54e-5470f274d555","Type":"ContainerStarted","Data":"7d7da6bbad9c862ccb5a423f6b451e00d6d55684942484ceb88b2424284ee8ff"}
Oct 31 01:34:36.069148 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:34:36.069049    4964 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-9e5368-7b876c5d49-4zpqb" podStartSLOduration=2.069035597 podStartE2EDuration="2.069035597s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:34:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:34:36.066050225 +0000 UTC m=+1844.542616608" watchObservedRunningTime="2026-10-31 01:34:36.069035597 +0000 UTC m=+1844.545601977"
Oct 31 01:34:36.179478 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:34:36.179446    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-5e916d-547c767844-llcts"
Oct 31 01:34:36.576631 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:34:36.576602    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-5e916d-547c767844-llcts"]
Oct 31 01:34:36.577778 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:34:36.577751    4964 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c420de6_67b9_4958_8083_ebd8bd939418.slice/crio-d4b0b242530e6afa02361b9137767df2c3f4863d1f9ae76fb8a46ab05ba7485a WatchSource:0}: Error finding container d4b0b242530e6afa02361b9137767df2c3f4863d1f9ae76fb8a46ab05ba7485a: Status 404 returned error can't find the container with id d4b0b242530e6afa02361b9137767df2c3f4863d1f9ae76fb8a46ab05ba7485a
Oct 31 01:34:36.579377 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:34:36.579356    4964 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 31 01:34:36.960899 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:34:36.960811    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-5e916d-547c767844-llcts" event={"ID":"7c420de6-67b9-4958-8083-ebd8bd939418","Type":"ContainerStarted","Data":"d4b0b242530e6afa02361b9137767df2c3f4863d1f9ae76fb8a46ab05ba7485a"}
Oct 31 01:34:37.710376 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:34:37.710352    4964 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-42c96d-openshift-default-69697959d9-p8ljq"
Oct 31 01:34:37.815953 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:34:37.815922    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/50f61fea-7b83-48aa-ad74-ea2be71f683b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/50f61fea-7b83-48aa-ad74-ea2be71f683b-istio-data\") pod \"50f61fea-7b83-48aa-ad74-ea2be71f683b\" (UID: \"50f61fea-7b83-48aa-ad74-ea2be71f683b\") "
Oct 31 01:34:37.816109 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:34:37.815990    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/50f61fea-7b83-48aa-ad74-ea2be71f683b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/50f61fea-7b83-48aa-ad74-ea2be71f683b-istiod-ca-cert\") pod \"50f61fea-7b83-48aa-ad74-ea2be71f683b\" (UID: \"50f61fea-7b83-48aa-ad74-ea2be71f683b\") "
Oct 31 01:34:37.816109 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:34:37.816011    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/50f61fea-7b83-48aa-ad74-ea2be71f683b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/50f61fea-7b83-48aa-ad74-ea2be71f683b-workload-socket\") pod \"50f61fea-7b83-48aa-ad74-ea2be71f683b\" (UID: \"50f61fea-7b83-48aa-ad74-ea2be71f683b\") "
Oct 31 01:34:37.816109 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:34:37.816045    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/50f61fea-7b83-48aa-ad74-ea2be71f683b-kube-api-access-h4xmz\" (UniqueName: \"kubernetes.io/projected/50f61fea-7b83-48aa-ad74-ea2be71f683b-kube-api-access-h4xmz\") pod \"50f61fea-7b83-48aa-ad74-ea2be71f683b\" (UID: \"50f61fea-7b83-48aa-ad74-ea2be71f683b\") "
Oct 31 01:34:37.816109 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:34:37.816064    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/50f61fea-7b83-48aa-ad74-ea2be71f683b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/50f61fea-7b83-48aa-ad74-ea2be71f683b-istio-podinfo\") pod \"50f61fea-7b83-48aa-ad74-ea2be71f683b\" (UID: \"50f61fea-7b83-48aa-ad74-ea2be71f683b\") "
Oct 31 01:34:37.816374 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:34:37.816133    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/50f61fea-7b83-48aa-ad74-ea2be71f683b-istio-token\" (UniqueName: \"kubernetes.io/projected/50f61fea-7b83-48aa-ad74-ea2be71f683b-istio-token\") pod \"50f61fea-7b83-48aa-ad74-ea2be71f683b\" (UID: \"50f61fea-7b83-48aa-ad74-ea2be71f683b\") "
Oct 31 01:34:37.816374 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:34:37.816186    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/50f61fea-7b83-48aa-ad74-ea2be71f683b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/50f61fea-7b83-48aa-ad74-ea2be71f683b-istio-envoy\") pod \"50f61fea-7b83-48aa-ad74-ea2be71f683b\" (UID: \"50f61fea-7b83-48aa-ad74-ea2be71f683b\") "
Oct 31 01:34:37.816374 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:34:37.816206    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50f61fea-7b83-48aa-ad74-ea2be71f683b-istio-data" pod "50f61fea-7b83-48aa-ad74-ea2be71f683b" (UID: "50f61fea-7b83-48aa-ad74-ea2be71f683b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:34:37.816374 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:34:37.816231    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/50f61fea-7b83-48aa-ad74-ea2be71f683b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/50f61fea-7b83-48aa-ad74-ea2be71f683b-credential-socket\") pod \"50f61fea-7b83-48aa-ad74-ea2be71f683b\" (UID: \"50f61fea-7b83-48aa-ad74-ea2be71f683b\") "
Oct 31 01:34:37.816374 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:34:37.816278    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/50f61fea-7b83-48aa-ad74-ea2be71f683b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/50f61fea-7b83-48aa-ad74-ea2be71f683b-workload-certs\") pod \"50f61fea-7b83-48aa-ad74-ea2be71f683b\" (UID: \"50f61fea-7b83-48aa-ad74-ea2be71f683b\") "
Oct 31 01:34:37.816374 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:34:37.816368    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50f61fea-7b83-48aa-ad74-ea2be71f683b-workload-socket" pod "50f61fea-7b83-48aa-ad74-ea2be71f683b" (UID: "50f61fea-7b83-48aa-ad74-ea2be71f683b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:34:37.816697 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:34:37.816482    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/50f61fea-7b83-48aa-ad74-ea2be71f683b-istiod-ca-cert" pod "50f61fea-7b83-48aa-ad74-ea2be71f683b" (UID: "50f61fea-7b83-48aa-ad74-ea2be71f683b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:34:37.816697 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:34:37.816576    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50f61fea-7b83-48aa-ad74-ea2be71f683b-credential-socket" pod "50f61fea-7b83-48aa-ad74-ea2be71f683b" (UID: "50f61fea-7b83-48aa-ad74-ea2be71f683b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:34:37.816847 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:34:37.816805    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50f61fea-7b83-48aa-ad74-ea2be71f683b-workload-certs" pod "50f61fea-7b83-48aa-ad74-ea2be71f683b" (UID: "50f61fea-7b83-48aa-ad74-ea2be71f683b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:34:37.817582 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:34:37.817269    4964 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/50f61fea-7b83-48aa-ad74-ea2be71f683b-istio-data\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:34:37.817582 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:34:37.817297    4964 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/50f61fea-7b83-48aa-ad74-ea2be71f683b-istiod-ca-cert\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:34:37.817582 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:34:37.817312    4964 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/50f61fea-7b83-48aa-ad74-ea2be71f683b-workload-socket\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:34:37.817582 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:34:37.817352    4964 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/50f61fea-7b83-48aa-ad74-ea2be71f683b-credential-socket\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:34:37.818880 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:34:37.818853    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/50f61fea-7b83-48aa-ad74-ea2be71f683b-istio-token" pod "50f61fea-7b83-48aa-ad74-ea2be71f683b" (UID: "50f61fea-7b83-48aa-ad74-ea2be71f683b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:34:37.818991 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:34:37.818962    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50f61fea-7b83-48aa-ad74-ea2be71f683b-istio-envoy" pod "50f61fea-7b83-48aa-ad74-ea2be71f683b" (UID: "50f61fea-7b83-48aa-ad74-ea2be71f683b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:34:37.819080 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:34:37.819066    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/50f61fea-7b83-48aa-ad74-ea2be71f683b-istio-podinfo" pod "50f61fea-7b83-48aa-ad74-ea2be71f683b" (UID: "50f61fea-7b83-48aa-ad74-ea2be71f683b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 01:34:37.819137 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:34:37.819119    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/50f61fea-7b83-48aa-ad74-ea2be71f683b-kube-api-access-h4xmz" pod "50f61fea-7b83-48aa-ad74-ea2be71f683b" (UID: "50f61fea-7b83-48aa-ad74-ea2be71f683b"). InnerVolumeSpecName "kube-api-access-h4xmz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:34:37.918431 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:34:37.918399    4964 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h4xmz\" (UniqueName: \"kubernetes.io/projected/50f61fea-7b83-48aa-ad74-ea2be71f683b-kube-api-access-h4xmz\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:34:37.918431 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:34:37.918427    4964 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/50f61fea-7b83-48aa-ad74-ea2be71f683b-istio-podinfo\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:34:37.918431 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:34:37.918437    4964 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/50f61fea-7b83-48aa-ad74-ea2be71f683b-istio-token\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:34:37.918682 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:34:37.918447    4964 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/50f61fea-7b83-48aa-ad74-ea2be71f683b-istio-envoy\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:34:37.918682 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:34:37.918456    4964 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/50f61fea-7b83-48aa-ad74-ea2be71f683b-workload-certs\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:34:37.969993 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:34:37.969961    4964 generic.go:356] "Generic (PLEG): container finished" podID="50f61fea-7b83-48aa-ad74-ea2be71f683b" containerID="e949c066245e8dcc03f440670b60a4128e1960e39914711ffcf7327c1fcdf930" exitCode=0
Oct 31 01:34:37.970463 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:34:37.970026    4964 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-42c96d-openshift-default-69697959d9-p8ljq"
Oct 31 01:34:37.970463 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:34:37.970043    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-42c96d-openshift-default-69697959d9-p8ljq" event={"ID":"50f61fea-7b83-48aa-ad74-ea2be71f683b","Type":"ContainerDied","Data":"e949c066245e8dcc03f440670b60a4128e1960e39914711ffcf7327c1fcdf930"}
Oct 31 01:34:37.970463 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:34:37.970092    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-42c96d-openshift-default-69697959d9-p8ljq" event={"ID":"50f61fea-7b83-48aa-ad74-ea2be71f683b","Type":"ContainerDied","Data":"3d61406606430beaf52c28bf589ca2f21fbe0b8c1b53cdcb6a730f1a899e1af5"}
Oct 31 01:34:37.970463 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:34:37.970114    4964 scope.go:122] "RemoveContainer" containerID="e949c066245e8dcc03f440670b60a4128e1960e39914711ffcf7327c1fcdf930"
Oct 31 01:34:37.971980 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:34:37.971958    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-5e916d-547c767844-llcts" event={"ID":"7c420de6-67b9-4958-8083-ebd8bd939418","Type":"ContainerStarted","Data":"192a883bc42d0d874efdf649d8e388469bbae96ef777de96d86c6f727b6cc7f5"}
Oct 31 01:34:37.972228 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:34:37.972210    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-5e916d-547c767844-llcts"
Oct 31 01:34:37.982035 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:34:37.982018    4964 scope.go:122] "RemoveContainer" containerID="9eeac5cf406f68573cb3279b07b12b7f3068677631a160ef21978be114b06a04"
Oct 31 01:34:37.992834 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:34:37.992814    4964 scope.go:122] "RemoveContainer" containerID="e949c066245e8dcc03f440670b60a4128e1960e39914711ffcf7327c1fcdf930"
Oct 31 01:34:37.993119 ip-10-0-133-197 kubenswrapper[4964]: E1031 01:34:37.993099    4964 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e949c066245e8dcc03f440670b60a4128e1960e39914711ffcf7327c1fcdf930\": container with ID starting with e949c066245e8dcc03f440670b60a4128e1960e39914711ffcf7327c1fcdf930 not found: ID does not exist" containerID="e949c066245e8dcc03f440670b60a4128e1960e39914711ffcf7327c1fcdf930"
Oct 31 01:34:37.993196 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:34:37.993135    4964 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e949c066245e8dcc03f440670b60a4128e1960e39914711ffcf7327c1fcdf930"} err="failed to get container status \"e949c066245e8dcc03f440670b60a4128e1960e39914711ffcf7327c1fcdf930\": rpc error: code = NotFound desc = could not find container \"e949c066245e8dcc03f440670b60a4128e1960e39914711ffcf7327c1fcdf930\": container with ID starting with e949c066245e8dcc03f440670b60a4128e1960e39914711ffcf7327c1fcdf930 not found: ID does not exist"
Oct 31 01:34:37.993196 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:34:37.993153    4964 scope.go:122] "RemoveContainer" containerID="9eeac5cf406f68573cb3279b07b12b7f3068677631a160ef21978be114b06a04"
Oct 31 01:34:37.993448 ip-10-0-133-197 kubenswrapper[4964]: E1031 01:34:37.993431    4964 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9eeac5cf406f68573cb3279b07b12b7f3068677631a160ef21978be114b06a04\": container with ID starting with 9eeac5cf406f68573cb3279b07b12b7f3068677631a160ef21978be114b06a04 not found: ID does not exist" containerID="9eeac5cf406f68573cb3279b07b12b7f3068677631a160ef21978be114b06a04"
Oct 31 01:34:37.993502 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:34:37.993455    4964 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9eeac5cf406f68573cb3279b07b12b7f3068677631a160ef21978be114b06a04"} err="failed to get container status \"9eeac5cf406f68573cb3279b07b12b7f3068677631a160ef21978be114b06a04\": rpc error: code = NotFound desc = could not find container \"9eeac5cf406f68573cb3279b07b12b7f3068677631a160ef21978be114b06a04\": container with ID starting with 9eeac5cf406f68573cb3279b07b12b7f3068677631a160ef21978be114b06a04 not found: ID does not exist"
Oct 31 01:34:38.137248 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:34:38.137197    4964 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-5e916d-547c767844-llcts" podStartSLOduration=2.203502772 podStartE2EDuration="3.137183252s" totalImagesPullingTime="933.68048ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:34:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:34:38.134795545 +0000 UTC m=+1846.611361925" watchObservedRunningTime="2026-10-31 01:34:38.137183252 +0000 UTC m=+1846.613749632"
Oct 31 01:34:38.250201 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:34:38.250087    4964 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-42c96d-openshift-default-69697959d9-p8ljq"]
Oct 31 01:34:38.254486 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:34:38.254458    4964 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-42c96d-openshift-default-69697959d9-p8ljq"]
Oct 31 01:34:39.777440 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:34:39.777405    4964 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="50f61fea-7b83-48aa-ad74-ea2be71f683b" path="/var/lib/kubelet/pods/50f61fea-7b83-48aa-ad74-ea2be71f683b/volumes"
Oct 31 01:34:40.852215 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:34:40.852178    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:34:41.240141 ip-10-0-133-197 kubenswrapper[4964]: E1031 01:34:41.240106    4964 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q56x5" containerName="extractor"
Oct 31 01:34:43.987771 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:34:43.987741    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-5e916d-547c767844-llcts"
Oct 31 01:34:45.128588 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:34:45.128553    4964 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-bc2046-579d7d879-mkdpx"]
Oct 31 01:34:45.529044 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:34:45.529004    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:34:45.700453 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:34:45.700421    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:34:45.956597 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:34:45.956526    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9e5368-7f65f554f5-g7mgs"
Oct 31 01:34:46.078969 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:34:46.078928    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:34:46.964832 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:34:46.964802    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-9e5368-7b876c5d49-4zpqb"
Oct 31 01:34:50.129209 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:34:50.129172    4964 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-bc2046-579d7d879-mkdpx" podUID="3a504885-cccb-4330-a15e-2f121ad43896" containerName="payload-processing" containerID="cri-o://6cedfd16ebdd9d9788c36cedd84b6ae6d386ef24878991c62b3fccceed6377dd" gracePeriod=25
Oct 31 01:34:50.413654 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:34:50.413624    4964 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-bc2046-579d7d879-mkdpx"
Oct 31 01:34:50.590716 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:34:50.590685    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3a504885-cccb-4330-a15e-2f121ad43896-kube-api-access-df6qp\" (UniqueName: \"kubernetes.io/projected/3a504885-cccb-4330-a15e-2f121ad43896-kube-api-access-df6qp\") pod \"3a504885-cccb-4330-a15e-2f121ad43896\" (UID: \"3a504885-cccb-4330-a15e-2f121ad43896\") "
Oct 31 01:34:50.590891 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:34:50.590726    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3a504885-cccb-4330-a15e-2f121ad43896-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3a504885-cccb-4330-a15e-2f121ad43896-plugins-config-volume\") pod \"3a504885-cccb-4330-a15e-2f121ad43896\" (UID: \"3a504885-cccb-4330-a15e-2f121ad43896\") "
Oct 31 01:34:50.591213 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:34:50.591175    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3a504885-cccb-4330-a15e-2f121ad43896-plugins-config-volume" pod "3a504885-cccb-4330-a15e-2f121ad43896" (UID: "3a504885-cccb-4330-a15e-2f121ad43896"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:34:50.591374 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:34:50.591259    4964 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3a504885-cccb-4330-a15e-2f121ad43896-plugins-config-volume\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:34:50.593135 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:34:50.593113    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a504885-cccb-4330-a15e-2f121ad43896-kube-api-access-df6qp" pod "3a504885-cccb-4330-a15e-2f121ad43896" (UID: "3a504885-cccb-4330-a15e-2f121ad43896"). InnerVolumeSpecName "kube-api-access-df6qp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:34:50.692900 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:34:50.692820    4964 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-df6qp\" (UniqueName: \"kubernetes.io/projected/3a504885-cccb-4330-a15e-2f121ad43896-kube-api-access-df6qp\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:34:51.065489 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:34:51.065455    4964 generic.go:356] "Generic (PLEG): container finished" podID="3a504885-cccb-4330-a15e-2f121ad43896" containerID="6cedfd16ebdd9d9788c36cedd84b6ae6d386ef24878991c62b3fccceed6377dd" exitCode=0
Oct 31 01:34:51.065698 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:34:51.065507    4964 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-bc2046-579d7d879-mkdpx"
Oct 31 01:34:51.065698 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:34:51.065545    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-bc2046-579d7d879-mkdpx" event={"ID":"3a504885-cccb-4330-a15e-2f121ad43896","Type":"ContainerDied","Data":"6cedfd16ebdd9d9788c36cedd84b6ae6d386ef24878991c62b3fccceed6377dd"}
Oct 31 01:34:51.065698 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:34:51.065609    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-bc2046-579d7d879-mkdpx" event={"ID":"3a504885-cccb-4330-a15e-2f121ad43896","Type":"ContainerDied","Data":"d0599c144e7e496f61eccd077feaa44ac9f019e0d78de9ecb93b82e228a3a67f"}
Oct 31 01:34:51.065698 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:34:51.065637    4964 scope.go:122] "RemoveContainer" containerID="6cedfd16ebdd9d9788c36cedd84b6ae6d386ef24878991c62b3fccceed6377dd"
Oct 31 01:34:51.076626 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:34:51.076601    4964 scope.go:122] "RemoveContainer" containerID="6cedfd16ebdd9d9788c36cedd84b6ae6d386ef24878991c62b3fccceed6377dd"
Oct 31 01:34:51.076887 ip-10-0-133-197 kubenswrapper[4964]: E1031 01:34:51.076866    4964 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6cedfd16ebdd9d9788c36cedd84b6ae6d386ef24878991c62b3fccceed6377dd\": container with ID starting with 6cedfd16ebdd9d9788c36cedd84b6ae6d386ef24878991c62b3fccceed6377dd not found: ID does not exist" containerID="6cedfd16ebdd9d9788c36cedd84b6ae6d386ef24878991c62b3fccceed6377dd"
Oct 31 01:34:51.076945 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:34:51.076895    4964 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6cedfd16ebdd9d9788c36cedd84b6ae6d386ef24878991c62b3fccceed6377dd"} err="failed to get container status \"6cedfd16ebdd9d9788c36cedd84b6ae6d386ef24878991c62b3fccceed6377dd\": rpc error: code = NotFound desc = could not find container \"6cedfd16ebdd9d9788c36cedd84b6ae6d386ef24878991c62b3fccceed6377dd\": container with ID starting with 6cedfd16ebdd9d9788c36cedd84b6ae6d386ef24878991c62b3fccceed6377dd not found: ID does not exist"
Oct 31 01:34:51.130124 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:34:51.130093    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:34:51.229410 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:34:51.229373    4964 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-bc2046-579d7d879-mkdpx"]
Oct 31 01:34:51.277885 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:34:51.277847    4964 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-bc2046-579d7d879-mkdpx"]
Oct 31 01:34:51.777916 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:34:51.777886    4964 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3a504885-cccb-4330-a15e-2f121ad43896" path="/var/lib/kubelet/pods/3a504885-cccb-4330-a15e-2f121ad43896/volumes"
Oct 31 01:34:52.387655 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:34:52.387623    4964 scope.go:122] "RemoveContainer" containerID="6cf56edb7185751ae6e9c849d46c2c29a2f0ebbde3be50c5fccb8d36c77ab8f9"
Oct 31 01:34:55.757178 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:34:55.757132    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:34:57.377767 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:34:57.377726    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:34:57.675093 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:34:57.675013    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:34:57.852862 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:34:57.852826    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:34:58.391921 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:34:58.391873    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:35:02.253130 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:35:02.253096    4964 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-14ca09-54b56b9987-h9ws9"]
Oct 31 01:35:02.253569 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:35:02.253309    4964 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-14ca09-54b56b9987-h9ws9" podUID="21025738-82f8-45ef-a53f-f7761d96ca2f" containerName="maas-api" containerID="cri-o://f449f0541bca60268967a885fb2a45fa7ae04797f042e45c7ba6607f66bbd664" gracePeriod=30
Oct 31 01:35:02.395228 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:35:02.395187    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:35:02.559721 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:35:02.559694    4964 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-14ca09-54b56b9987-h9ws9"
Oct 31 01:35:02.661351 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:35:02.658706    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/21025738-82f8-45ef-a53f-f7761d96ca2f-kube-api-access-lknnw\" (UniqueName: \"kubernetes.io/projected/21025738-82f8-45ef-a53f-f7761d96ca2f-kube-api-access-lknnw\") pod \"21025738-82f8-45ef-a53f-f7761d96ca2f\" (UID: \"21025738-82f8-45ef-a53f-f7761d96ca2f\") "
Oct 31 01:35:02.661351 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:35:02.658763    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/21025738-82f8-45ef-a53f-f7761d96ca2f-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/21025738-82f8-45ef-a53f-f7761d96ca2f-openshift-service-ca\") pod \"21025738-82f8-45ef-a53f-f7761d96ca2f\" (UID: \"21025738-82f8-45ef-a53f-f7761d96ca2f\") "
Oct 31 01:35:02.661351 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:35:02.658818    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/21025738-82f8-45ef-a53f-f7761d96ca2f-metrics-certs\" (UniqueName: \"kubernetes.io/secret/21025738-82f8-45ef-a53f-f7761d96ca2f-metrics-certs\") pod \"21025738-82f8-45ef-a53f-f7761d96ca2f\" (UID: \"21025738-82f8-45ef-a53f-f7761d96ca2f\") "
Oct 31 01:35:02.661351 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:35:02.658964    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/21025738-82f8-45ef-a53f-f7761d96ca2f-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/21025738-82f8-45ef-a53f-f7761d96ca2f-maas-api-tls\") pod \"21025738-82f8-45ef-a53f-f7761d96ca2f\" (UID: \"21025738-82f8-45ef-a53f-f7761d96ca2f\") "
Oct 31 01:35:02.661884 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:35:02.661845    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/21025738-82f8-45ef-a53f-f7761d96ca2f-openshift-service-ca" pod "21025738-82f8-45ef-a53f-f7761d96ca2f" (UID: "21025738-82f8-45ef-a53f-f7761d96ca2f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:35:02.663103 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:35:02.663074    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/21025738-82f8-45ef-a53f-f7761d96ca2f-maas-api-tls" pod "21025738-82f8-45ef-a53f-f7761d96ca2f" (UID: "21025738-82f8-45ef-a53f-f7761d96ca2f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:35:02.671599 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:35:02.671552    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/21025738-82f8-45ef-a53f-f7761d96ca2f-kube-api-access-lknnw" pod "21025738-82f8-45ef-a53f-f7761d96ca2f" (UID: "21025738-82f8-45ef-a53f-f7761d96ca2f"). InnerVolumeSpecName "kube-api-access-lknnw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:35:02.672637 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:35:02.672574    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/21025738-82f8-45ef-a53f-f7761d96ca2f-metrics-certs" pod "21025738-82f8-45ef-a53f-f7761d96ca2f" (UID: "21025738-82f8-45ef-a53f-f7761d96ca2f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:35:02.760654 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:35:02.760622    4964 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/21025738-82f8-45ef-a53f-f7761d96ca2f-maas-api-tls\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:35:02.760654 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:35:02.760649    4964 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lknnw\" (UniqueName: \"kubernetes.io/projected/21025738-82f8-45ef-a53f-f7761d96ca2f-kube-api-access-lknnw\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:35:02.760654 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:35:02.760660    4964 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/21025738-82f8-45ef-a53f-f7761d96ca2f-openshift-service-ca\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:35:02.760872 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:35:02.760669    4964 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/21025738-82f8-45ef-a53f-f7761d96ca2f-metrics-certs\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:35:03.152465 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:35:03.152431    4964 generic.go:356] "Generic (PLEG): container finished" podID="21025738-82f8-45ef-a53f-f7761d96ca2f" containerID="f449f0541bca60268967a885fb2a45fa7ae04797f042e45c7ba6607f66bbd664" exitCode=0
Oct 31 01:35:03.152684 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:35:03.152477    4964 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-14ca09-54b56b9987-h9ws9"
Oct 31 01:35:03.152684 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:35:03.152520    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-14ca09-54b56b9987-h9ws9" event={"ID":"21025738-82f8-45ef-a53f-f7761d96ca2f","Type":"ContainerDied","Data":"f449f0541bca60268967a885fb2a45fa7ae04797f042e45c7ba6607f66bbd664"}
Oct 31 01:35:03.152684 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:35:03.152576    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-14ca09-54b56b9987-h9ws9" event={"ID":"21025738-82f8-45ef-a53f-f7761d96ca2f","Type":"ContainerDied","Data":"1a04f380ec23d3bf336b0603ceeacfa61dec31bb420b7e417f32aa111b3d1ec1"}
Oct 31 01:35:03.152684 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:35:03.152604    4964 scope.go:122] "RemoveContainer" containerID="f449f0541bca60268967a885fb2a45fa7ae04797f042e45c7ba6607f66bbd664"
Oct 31 01:35:03.163085 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:35:03.163068    4964 scope.go:122] "RemoveContainer" containerID="f449f0541bca60268967a885fb2a45fa7ae04797f042e45c7ba6607f66bbd664"
Oct 31 01:35:03.163359 ip-10-0-133-197 kubenswrapper[4964]: E1031 01:35:03.163310    4964 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f449f0541bca60268967a885fb2a45fa7ae04797f042e45c7ba6607f66bbd664\": container with ID starting with f449f0541bca60268967a885fb2a45fa7ae04797f042e45c7ba6607f66bbd664 not found: ID does not exist" containerID="f449f0541bca60268967a885fb2a45fa7ae04797f042e45c7ba6607f66bbd664"
Oct 31 01:35:03.163397 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:35:03.163368    4964 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f449f0541bca60268967a885fb2a45fa7ae04797f042e45c7ba6607f66bbd664"} err="failed to get container status \"f449f0541bca60268967a885fb2a45fa7ae04797f042e45c7ba6607f66bbd664\": rpc error: code = NotFound desc = could not find container \"f449f0541bca60268967a885fb2a45fa7ae04797f042e45c7ba6607f66bbd664\": container with ID starting with f449f0541bca60268967a885fb2a45fa7ae04797f042e45c7ba6607f66bbd664 not found: ID does not exist"
Oct 31 01:35:03.319186 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:35:03.319152    4964 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-14ca09-54b56b9987-h9ws9"]
Oct 31 01:35:03.322213 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:35:03.322188    4964 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-14ca09-54b56b9987-h9ws9"]
Oct 31 01:35:03.777841 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:35:03.777812    4964 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="21025738-82f8-45ef-a53f-f7761d96ca2f" path="/var/lib/kubelet/pods/21025738-82f8-45ef-a53f-f7761d96ca2f/volumes"
Oct 31 01:35:03.976861 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:35:03.976831    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:35:04.214233 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:35:04.214142    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:35:04.399928 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:35:04.399896    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:35:04.578607 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:35:04.578576    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:35:04.827803 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:35:04.827770    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:35:05.007266 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:35:05.007216    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:35:05.500711 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:35:05.500669    4964 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-14ca09-54b56b9987-h9ws9" podUID="21025738-82f8-45ef-a53f-f7761d96ca2f" containerName="maas-api" probeResult="failure" output="Get \"https://10.132.0.117:8443/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:35:09.031794 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:35:09.031753    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:35:09.353431 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:35:09.353337    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:35:10.738052 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:35:10.738016    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:35:10.923365 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:35:10.923306    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:35:11.209520 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:35:11.209424    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:35:11.311771 ip-10-0-133-197 kubenswrapper[4964]: E1031 01:35:11.311736    4964 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q56x5" containerName="extractor"
Oct 31 01:35:11.401761 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:35:11.401729    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:35:12.176230 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:35:12.176192    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:35:12.826694 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:35:12.824754    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:35:13.016656 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:35:13.016621    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:35:14.626612 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:35:14.621855    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:35:16.873213 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:35:16.873174    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:35:17.118899 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:35:17.118850    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:35:17.311981 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:35:17.311945    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:35:17.730843 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:35:17.730813    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:35:17.969378 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:35:17.969299    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:35:19.500984 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:35:19.500952    4964 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-9e5368-565c76c-jrl5z"]
Oct 31 01:35:19.501563 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:35:19.501546    4964 cpu_manager.go:414] "containerMap: removing container" podUID="3a504885-cccb-4330-a15e-2f121ad43896" containerName="payload-processing"
Oct 31 01:35:19.501563 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:35:19.501562    4964 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3a504885-cccb-4330-a15e-2f121ad43896" containerName="payload-processing"
Oct 31 01:35:19.501688 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:35:19.501589    4964 cpu_manager.go:414] "containerMap: removing container" podUID="21025738-82f8-45ef-a53f-f7761d96ca2f" containerName="maas-api"
Oct 31 01:35:19.501688 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:35:19.501595    4964 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="21025738-82f8-45ef-a53f-f7761d96ca2f" containerName="maas-api"
Oct 31 01:35:19.501688 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:35:19.501611    4964 cpu_manager.go:414] "containerMap: removing container" podUID="50f61fea-7b83-48aa-ad74-ea2be71f683b" containerName="istio-proxy"
Oct 31 01:35:19.501688 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:35:19.501617    4964 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="50f61fea-7b83-48aa-ad74-ea2be71f683b" containerName="istio-proxy"
Oct 31 01:35:19.501688 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:35:19.501624    4964 cpu_manager.go:414] "containerMap: removing container" podUID="50f61fea-7b83-48aa-ad74-ea2be71f683b" containerName="istio-proxy"
Oct 31 01:35:19.501688 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:35:19.501629    4964 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="50f61fea-7b83-48aa-ad74-ea2be71f683b" containerName="istio-proxy"
Oct 31 01:35:19.501989 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:35:19.501715    4964 memory_manager.go:365] "RemoveStaleState removing state" podUID="3a504885-cccb-4330-a15e-2f121ad43896" containerName="payload-processing"
Oct 31 01:35:19.501989 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:35:19.501727    4964 memory_manager.go:365] "RemoveStaleState removing state" podUID="21025738-82f8-45ef-a53f-f7761d96ca2f" containerName="maas-api"
Oct 31 01:35:19.501989 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:35:19.501735    4964 memory_manager.go:365] "RemoveStaleState removing state" podUID="50f61fea-7b83-48aa-ad74-ea2be71f683b" containerName="istio-proxy"
Oct 31 01:35:19.501989 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:35:19.501740    4964 memory_manager.go:365] "RemoveStaleState removing state" podUID="50f61fea-7b83-48aa-ad74-ea2be71f683b" containerName="istio-proxy"
Oct 31 01:35:19.505429 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:35:19.505405    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-9e5368-565c76c-jrl5z"
Oct 31 01:35:19.557879 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:35:19.557837    4964 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-9e5368-serving-cert\""
Oct 31 01:35:19.602826 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:35:19.602794    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c5mjk\" (UniqueName: \"kubernetes.io/projected/c2c5cd7c-7776-4e22-b91b-34fd4e71da5d-kube-api-access-c5mjk\") pod \"maas-api-e2e-ipp-a-9e5368-565c76c-jrl5z\" (UID: \"c2c5cd7c-7776-4e22-b91b-34fd4e71da5d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-9e5368-565c76c-jrl5z"
Oct 31 01:35:19.602967 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:35:19.602888    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c2c5cd7c-7776-4e22-b91b-34fd4e71da5d-maas-api-tls\") pod \"maas-api-e2e-ipp-a-9e5368-565c76c-jrl5z\" (UID: \"c2c5cd7c-7776-4e22-b91b-34fd4e71da5d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-9e5368-565c76c-jrl5z"
Oct 31 01:35:19.602967 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:35:19.602928    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c2c5cd7c-7776-4e22-b91b-34fd4e71da5d-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-9e5368-565c76c-jrl5z\" (UID: \"c2c5cd7c-7776-4e22-b91b-34fd4e71da5d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-9e5368-565c76c-jrl5z"
Oct 31 01:35:19.603049 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:35:19.603018    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c2c5cd7c-7776-4e22-b91b-34fd4e71da5d-metrics-certs\") pod \"maas-api-e2e-ipp-a-9e5368-565c76c-jrl5z\" (UID: \"c2c5cd7c-7776-4e22-b91b-34fd4e71da5d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-9e5368-565c76c-jrl5z"
Oct 31 01:35:19.610581 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:35:19.610555    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-9e5368-565c76c-jrl5z"]
Oct 31 01:35:19.704343 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:35:19.704294    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c5mjk\" (UniqueName: \"kubernetes.io/projected/c2c5cd7c-7776-4e22-b91b-34fd4e71da5d-kube-api-access-c5mjk\") pod \"maas-api-e2e-ipp-a-9e5368-565c76c-jrl5z\" (UID: \"c2c5cd7c-7776-4e22-b91b-34fd4e71da5d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-9e5368-565c76c-jrl5z"
Oct 31 01:35:19.704529 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:35:19.704393    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c2c5cd7c-7776-4e22-b91b-34fd4e71da5d-maas-api-tls\") pod \"maas-api-e2e-ipp-a-9e5368-565c76c-jrl5z\" (UID: \"c2c5cd7c-7776-4e22-b91b-34fd4e71da5d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-9e5368-565c76c-jrl5z"
Oct 31 01:35:19.704529 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:35:19.704429    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c2c5cd7c-7776-4e22-b91b-34fd4e71da5d-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-9e5368-565c76c-jrl5z\" (UID: \"c2c5cd7c-7776-4e22-b91b-34fd4e71da5d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-9e5368-565c76c-jrl5z"
Oct 31 01:35:19.704529 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:35:19.704509    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c2c5cd7c-7776-4e22-b91b-34fd4e71da5d-metrics-certs\") pod \"maas-api-e2e-ipp-a-9e5368-565c76c-jrl5z\" (UID: \"c2c5cd7c-7776-4e22-b91b-34fd4e71da5d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-9e5368-565c76c-jrl5z"
Oct 31 01:35:19.704705 ip-10-0-133-197 kubenswrapper[4964]: E1031 01:35:19.704628    4964 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-a-9e5368-serving-cert: secret "maas-api-e2e-ipp-a-9e5368-serving-cert" not found
Oct 31 01:35:19.704705 ip-10-0-133-197 kubenswrapper[4964]: E1031 01:35:19.704692    4964 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c2c5cd7c-7776-4e22-b91b-34fd4e71da5d-maas-api-tls podName:c2c5cd7c-7776-4e22-b91b-34fd4e71da5d nodeName:}" failed. No retries permitted until 2026-10-31 01:35:20.204679395 +0000 UTC m=+1888.681245753 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/c2c5cd7c-7776-4e22-b91b-34fd4e71da5d-maas-api-tls") pod "maas-api-e2e-ipp-a-9e5368-565c76c-jrl5z" (UID: "c2c5cd7c-7776-4e22-b91b-34fd4e71da5d") : secret "maas-api-e2e-ipp-a-9e5368-serving-cert" not found
Oct 31 01:35:19.705690 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:35:19.705662    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c2c5cd7c-7776-4e22-b91b-34fd4e71da5d-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-9e5368-565c76c-jrl5z\" (UID: \"c2c5cd7c-7776-4e22-b91b-34fd4e71da5d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-9e5368-565c76c-jrl5z"
Oct 31 01:35:19.707406 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:35:19.707383    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c2c5cd7c-7776-4e22-b91b-34fd4e71da5d-metrics-certs\") pod \"maas-api-e2e-ipp-a-9e5368-565c76c-jrl5z\" (UID: \"c2c5cd7c-7776-4e22-b91b-34fd4e71da5d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-9e5368-565c76c-jrl5z"
Oct 31 01:35:19.742056 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:35:19.742028    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:35:19.760991 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:35:19.760930    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c5mjk\" (UniqueName: \"kubernetes.io/projected/c2c5cd7c-7776-4e22-b91b-34fd4e71da5d-kube-api-access-c5mjk\") pod \"maas-api-e2e-ipp-a-9e5368-565c76c-jrl5z\" (UID: \"c2c5cd7c-7776-4e22-b91b-34fd4e71da5d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-9e5368-565c76c-jrl5z"
Oct 31 01:35:19.925455 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:35:19.925418    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:35:20.211667 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:35:20.211580    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c2c5cd7c-7776-4e22-b91b-34fd4e71da5d-maas-api-tls\") pod \"maas-api-e2e-ipp-a-9e5368-565c76c-jrl5z\" (UID: \"c2c5cd7c-7776-4e22-b91b-34fd4e71da5d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-9e5368-565c76c-jrl5z"
Oct 31 01:35:20.214236 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:35:20.214207    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c2c5cd7c-7776-4e22-b91b-34fd4e71da5d-maas-api-tls\") pod \"maas-api-e2e-ipp-a-9e5368-565c76c-jrl5z\" (UID: \"c2c5cd7c-7776-4e22-b91b-34fd4e71da5d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-9e5368-565c76c-jrl5z"
Oct 31 01:35:20.420162 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:35:20.420131    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-9e5368-565c76c-jrl5z"
Oct 31 01:35:20.814856 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:35:20.814827    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-9e5368-565c76c-jrl5z"]
Oct 31 01:35:20.816330 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:35:20.816292    4964 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc2c5cd7c_7776_4e22_b91b_34fd4e71da5d.slice/crio-aac449051b7d640f1d5870684b81b059d47e41153e95dfb0c25c84c250e65d79 WatchSource:0}: Error finding container aac449051b7d640f1d5870684b81b059d47e41153e95dfb0c25c84c250e65d79: Status 404 returned error can't find the container with id aac449051b7d640f1d5870684b81b059d47e41153e95dfb0c25c84c250e65d79
Oct 31 01:35:21.276971 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:35:21.276939    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-9e5368-565c76c-jrl5z" event={"ID":"c2c5cd7c-7776-4e22-b91b-34fd4e71da5d","Type":"ContainerStarted","Data":"aac449051b7d640f1d5870684b81b059d47e41153e95dfb0c25c84c250e65d79"}
Oct 31 01:35:22.284884 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:35:22.284849    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-9e5368-565c76c-jrl5z" event={"ID":"c2c5cd7c-7776-4e22-b91b-34fd4e71da5d","Type":"ContainerStarted","Data":"57ec4f181bb5826be30b5ec62c2ca406344c851af9678c8ef1c23990232c45c0"}
Oct 31 01:35:22.285356 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:35:22.284920    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-9e5368-565c76c-jrl5z"
Oct 31 01:35:22.403191 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:35:22.403133    4964 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-9e5368-565c76c-jrl5z" podStartSLOduration=2.575318991 podStartE2EDuration="3.403119478s" totalImagesPullingTime="827.800487ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:35:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:35:22.399034879 +0000 UTC m=+1890.875601262" watchObservedRunningTime="2026-10-31 01:35:22.403119478 +0000 UTC m=+1890.879685858"
Oct 31 01:35:23.525450 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:35:23.525417    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:35:23.958842 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:35:23.958760    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:35:25.439149 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:35:25.439113    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:35:28.298061 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:35:28.298033    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-9e5368-565c76c-jrl5z"
Oct 31 01:35:30.387767 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:35:30.387721    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:35:30.700057 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:35:30.699969    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:35:35.207549 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:35:35.207502    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:35:36.835294 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:35:36.835256    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:35:37.387237 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:35:37.387195    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:35:38.091607 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:35:38.091571    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:35:38.275800 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:35:38.275769    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:35:40.386962 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:35:40.386925    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:35:41.244253 ip-10-0-133-197 kubenswrapper[4964]: E1031 01:35:41.244217    4964 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q56x5" containerName="extractor"
Oct 31 01:35:42.281678 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:35:42.281637    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:35:42.477246 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:35:42.477208    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:35:45.655680 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:35:45.655629    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:35:45.828231 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:35:45.828091    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:35:49.822725 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:35:49.822642    4964 patch_prober.go:29] interesting pod/e2e-ipp-a-9e5368-openshift-default-5ff7fc6fd7-fp6pw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" start-of-body=
Oct 31 01:35:49.822725 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:35:49.822705    4964 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-9e5368-openshift-default-5ff7fc6fd7-fp6pw" podUID="5ab8d4bd-eefa-4a6c-8b3a-34593e689827" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused"
Oct 31 01:35:50.502722 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:35:50.502687    4964 generic.go:356] "Generic (PLEG): container finished" podID="5ab8d4bd-eefa-4a6c-8b3a-34593e689827" containerID="84b91dfbaff657c035aaadf786b3fadae6fc10b902a4346df8a0db03a6b1abcd" exitCode=0
Oct 31 01:35:50.502892 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:35:50.502760    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-9e5368-openshift-default-5ff7fc6fd7-fp6pw" event={"ID":"5ab8d4bd-eefa-4a6c-8b3a-34593e689827","Type":"ContainerDied","Data":"84b91dfbaff657c035aaadf786b3fadae6fc10b902a4346df8a0db03a6b1abcd"}
Oct 31 01:35:50.503143 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:35:50.503130    4964 scope.go:122] "RemoveContainer" containerID="84b91dfbaff657c035aaadf786b3fadae6fc10b902a4346df8a0db03a6b1abcd"
Oct 31 01:35:50.504580 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:35:50.504554    4964 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 01:35:50.504668 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:35:50.504616    4964 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 01:35:50.504668 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:35:50.504649    4964 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 01:35:50.654958 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:35:50.654717    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:35:51.517307 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:35:51.517271    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-9e5368-openshift-default-5ff7fc6fd7-fp6pw" event={"ID":"5ab8d4bd-eefa-4a6c-8b3a-34593e689827","Type":"ContainerStarted","Data":"38e6ea22ddbc2d642aac2580828fee2853259d7990f397bb2948d7de2c0e4c11"}
Oct 31 01:35:52.162645 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:35:52.162614    4964 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-9e5368-openshift-default-5ff7fc6fd7-fp6pw"
Oct 31 01:35:52.162875 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:35:52.162853    4964 patch_prober.go:29] interesting pod/e2e-ipp-a-9e5368-openshift-default-5ff7fc6fd7-fp6pw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" start-of-body=
Oct 31 01:35:52.162920 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:35:52.162901    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-9e5368-openshift-default-5ff7fc6fd7-fp6pw" podUID="5ab8d4bd-eefa-4a6c-8b3a-34593e689827" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused"
Oct 31 01:35:53.163563 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:35:53.163526    4964 patch_prober.go:29] interesting pod/e2e-ipp-a-9e5368-openshift-default-5ff7fc6fd7-fp6pw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" start-of-body=
Oct 31 01:35:53.164043 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:35:53.163587    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-9e5368-openshift-default-5ff7fc6fd7-fp6pw" podUID="5ab8d4bd-eefa-4a6c-8b3a-34593e689827" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused"
Oct 31 01:35:54.163672 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:35:54.163639    4964 patch_prober.go:29] interesting pod/e2e-ipp-a-9e5368-openshift-default-5ff7fc6fd7-fp6pw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" start-of-body=
Oct 31 01:35:54.164065 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:35:54.163695    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-9e5368-openshift-default-5ff7fc6fd7-fp6pw" podUID="5ab8d4bd-eefa-4a6c-8b3a-34593e689827" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused"
Oct 31 01:35:55.163517 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:35:55.163475    4964 patch_prober.go:29] interesting pod/e2e-ipp-a-9e5368-openshift-default-5ff7fc6fd7-fp6pw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" start-of-body=
Oct 31 01:35:55.163705 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:35:55.163563    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-9e5368-openshift-default-5ff7fc6fd7-fp6pw" podUID="5ab8d4bd-eefa-4a6c-8b3a-34593e689827" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused"
Oct 31 01:35:55.424524 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:35:55.424415    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:35:56.162897 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:35:56.162868    4964 patch_prober.go:29] interesting pod/e2e-ipp-a-9e5368-openshift-default-5ff7fc6fd7-fp6pw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" start-of-body=
Oct 31 01:35:56.163083 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:35:56.162932    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-9e5368-openshift-default-5ff7fc6fd7-fp6pw" podUID="5ab8d4bd-eefa-4a6c-8b3a-34593e689827" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused"
Oct 31 01:35:57.167998 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:35:57.167968    4964 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-9e5368-openshift-default-5ff7fc6fd7-fp6pw"
Oct 31 01:35:57.168438 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:35:57.168384    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-9e5368-openshift-default-5ff7fc6fd7-fp6pw"
Oct 31 01:35:57.169075 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:35:57.169055    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-9e5368-openshift-default-5ff7fc6fd7-fp6pw"
Oct 31 01:35:59.918881 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:35:59.918837    4964 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-3d3023-f97fd979b-b7trx"]
Oct 31 01:35:59.922477 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:35:59.922459    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3d3023-f97fd979b-b7trx"
Oct 31 01:36:00.025368 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:36:00.025312    4964 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-3d3023-serving-cert\""
Oct 31 01:36:00.028815 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:36:00.028787    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-3d3023-f97fd979b-b7trx"]
Oct 31 01:36:00.030054 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:36:00.030034    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xqcfz\" (UniqueName: \"kubernetes.io/projected/26492a66-0b7e-4fe0-a2e1-a52b345c12e7-kube-api-access-xqcfz\") pod \"maas-api-e2e-ipp-a-3d3023-f97fd979b-b7trx\" (UID: \"26492a66-0b7e-4fe0-a2e1-a52b345c12e7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3d3023-f97fd979b-b7trx"
Oct 31 01:36:00.030137 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:36:00.030067    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/26492a66-0b7e-4fe0-a2e1-a52b345c12e7-maas-api-tls\") pod \"maas-api-e2e-ipp-a-3d3023-f97fd979b-b7trx\" (UID: \"26492a66-0b7e-4fe0-a2e1-a52b345c12e7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3d3023-f97fd979b-b7trx"
Oct 31 01:36:00.030137 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:36:00.030094    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/26492a66-0b7e-4fe0-a2e1-a52b345c12e7-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-3d3023-f97fd979b-b7trx\" (UID: \"26492a66-0b7e-4fe0-a2e1-a52b345c12e7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3d3023-f97fd979b-b7trx"
Oct 31 01:36:00.030228 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:36:00.030195    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/26492a66-0b7e-4fe0-a2e1-a52b345c12e7-metrics-certs\") pod \"maas-api-e2e-ipp-a-3d3023-f97fd979b-b7trx\" (UID: \"26492a66-0b7e-4fe0-a2e1-a52b345c12e7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3d3023-f97fd979b-b7trx"
Oct 31 01:36:00.131210 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:36:00.131173    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xqcfz\" (UniqueName: \"kubernetes.io/projected/26492a66-0b7e-4fe0-a2e1-a52b345c12e7-kube-api-access-xqcfz\") pod \"maas-api-e2e-ipp-a-3d3023-f97fd979b-b7trx\" (UID: \"26492a66-0b7e-4fe0-a2e1-a52b345c12e7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3d3023-f97fd979b-b7trx"
Oct 31 01:36:00.131210 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:36:00.131211    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/26492a66-0b7e-4fe0-a2e1-a52b345c12e7-maas-api-tls\") pod \"maas-api-e2e-ipp-a-3d3023-f97fd979b-b7trx\" (UID: \"26492a66-0b7e-4fe0-a2e1-a52b345c12e7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3d3023-f97fd979b-b7trx"
Oct 31 01:36:00.131451 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:36:00.131248    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/26492a66-0b7e-4fe0-a2e1-a52b345c12e7-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-3d3023-f97fd979b-b7trx\" (UID: \"26492a66-0b7e-4fe0-a2e1-a52b345c12e7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3d3023-f97fd979b-b7trx"
Oct 31 01:36:00.131451 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:36:00.131281    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/26492a66-0b7e-4fe0-a2e1-a52b345c12e7-metrics-certs\") pod \"maas-api-e2e-ipp-a-3d3023-f97fd979b-b7trx\" (UID: \"26492a66-0b7e-4fe0-a2e1-a52b345c12e7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3d3023-f97fd979b-b7trx"
Oct 31 01:36:00.131451 ip-10-0-133-197 kubenswrapper[4964]: E1031 01:36:00.131428    4964 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-a-3d3023-serving-cert: secret "maas-api-e2e-ipp-a-3d3023-serving-cert" not found
Oct 31 01:36:00.131614 ip-10-0-133-197 kubenswrapper[4964]: E1031 01:36:00.131495    4964 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/26492a66-0b7e-4fe0-a2e1-a52b345c12e7-maas-api-tls podName:26492a66-0b7e-4fe0-a2e1-a52b345c12e7 nodeName:}" failed. No retries permitted until 2026-10-31 01:36:00.631473613 +0000 UTC m=+1929.108039975 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/26492a66-0b7e-4fe0-a2e1-a52b345c12e7-maas-api-tls") pod "maas-api-e2e-ipp-a-3d3023-f97fd979b-b7trx" (UID: "26492a66-0b7e-4fe0-a2e1-a52b345c12e7") : secret "maas-api-e2e-ipp-a-3d3023-serving-cert" not found
Oct 31 01:36:00.132144 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:36:00.132070    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/26492a66-0b7e-4fe0-a2e1-a52b345c12e7-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-3d3023-f97fd979b-b7trx\" (UID: \"26492a66-0b7e-4fe0-a2e1-a52b345c12e7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3d3023-f97fd979b-b7trx"
Oct 31 01:36:00.134182 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:36:00.134155    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/26492a66-0b7e-4fe0-a2e1-a52b345c12e7-metrics-certs\") pod \"maas-api-e2e-ipp-a-3d3023-f97fd979b-b7trx\" (UID: \"26492a66-0b7e-4fe0-a2e1-a52b345c12e7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3d3023-f97fd979b-b7trx"
Oct 31 01:36:00.160629 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:36:00.160584    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:36:00.192802 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:36:00.192679    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xqcfz\" (UniqueName: \"kubernetes.io/projected/26492a66-0b7e-4fe0-a2e1-a52b345c12e7-kube-api-access-xqcfz\") pod \"maas-api-e2e-ipp-a-3d3023-f97fd979b-b7trx\" (UID: \"26492a66-0b7e-4fe0-a2e1-a52b345c12e7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3d3023-f97fd979b-b7trx"
Oct 31 01:36:00.382213 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:36:00.382134    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:36:00.575512 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:36:00.575475    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:36:00.637438 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:36:00.637394    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/26492a66-0b7e-4fe0-a2e1-a52b345c12e7-maas-api-tls\") pod \"maas-api-e2e-ipp-a-3d3023-f97fd979b-b7trx\" (UID: \"26492a66-0b7e-4fe0-a2e1-a52b345c12e7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3d3023-f97fd979b-b7trx"
Oct 31 01:36:00.640031 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:36:00.640000    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/26492a66-0b7e-4fe0-a2e1-a52b345c12e7-maas-api-tls\") pod \"maas-api-e2e-ipp-a-3d3023-f97fd979b-b7trx\" (UID: \"26492a66-0b7e-4fe0-a2e1-a52b345c12e7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3d3023-f97fd979b-b7trx"
Oct 31 01:36:00.835358 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:36:00.835233    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3d3023-f97fd979b-b7trx"
Oct 31 01:36:01.216400 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:36:01.216376    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-3d3023-f97fd979b-b7trx"]
Oct 31 01:36:01.218144 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:36:01.218111    4964 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26492a66_0b7e_4fe0_a2e1_a52b345c12e7.slice/crio-a67b731eaef5dde9e931aa0062ab3492c1b5e9941c345dcc35ed9876e8f9e74b WatchSource:0}: Error finding container a67b731eaef5dde9e931aa0062ab3492c1b5e9941c345dcc35ed9876e8f9e74b: Status 404 returned error can't find the container with id a67b731eaef5dde9e931aa0062ab3492c1b5e9941c345dcc35ed9876e8f9e74b
Oct 31 01:36:01.596748 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:36:01.596719    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3d3023-f97fd979b-b7trx" event={"ID":"26492a66-0b7e-4fe0-a2e1-a52b345c12e7","Type":"ContainerStarted","Data":"a67b731eaef5dde9e931aa0062ab3492c1b5e9941c345dcc35ed9876e8f9e74b"}
Oct 31 01:36:02.268942 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:36:02.268909    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:36:02.454606 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:36:02.454523    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:36:02.605930 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:36:02.605884    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3d3023-f97fd979b-b7trx" event={"ID":"26492a66-0b7e-4fe0-a2e1-a52b345c12e7","Type":"ContainerStarted","Data":"98b079983ed765fda422c2fde7b30d74757fa0d8d3cc47e82de9f616926706fb"}
Oct 31 01:36:02.606135 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:36:02.606018    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3d3023-f97fd979b-b7trx"
Oct 31 01:36:02.721832 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:36:02.721783    4964 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3d3023-f97fd979b-b7trx" podStartSLOduration=2.8764870030000003 podStartE2EDuration="3.721766978s" totalImagesPullingTime="845.279975ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:35:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:36:02.720407876 +0000 UTC m=+1931.196974257" watchObservedRunningTime="2026-10-31 01:36:02.721766978 +0000 UTC m=+1931.198333359"
Oct 31 01:36:04.123841 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:36:04.123807    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:36:05.204760 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:36:05.204390    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:36:07.562855 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:36:07.562817    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:36:07.769835 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:36:07.769794    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:36:08.619234 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:36:08.619206    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3d3023-f97fd979b-b7trx"
Oct 31 01:36:09.368919 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:36:09.368881    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:36:10.011972 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:36:10.011923    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:36:11.239420 ip-10-0-133-197 kubenswrapper[4964]: E1031 01:36:11.239386    4964 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q56x5" containerName="extractor"
Oct 31 01:36:11.673062 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:36:11.672968    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:36:15.098955 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:36:15.098917    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:36:19.128432 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:36:19.128399    4964 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-bc2046-7988f9b774-pz82x"]
Oct 31 01:36:19.128943 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:36:19.128607    4964 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bc2046-7988f9b774-pz82x" podUID="65cf4fa6-618d-44d2-91a6-2b8d7b1dcde2" containerName="maas-api" containerID="cri-o://c7a8511df73e48da71ad92a53f2441f1b3ac4d0d8bfd14baa046a5f3e8ac3604" gracePeriod=30
Oct 31 01:36:19.268540 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:36:19.268505    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:36:19.429128 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:36:19.429102    4964 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bc2046-7988f9b774-pz82x"
Oct 31 01:36:19.464464 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:36:19.464435    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/65cf4fa6-618d-44d2-91a6-2b8d7b1dcde2-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/65cf4fa6-618d-44d2-91a6-2b8d7b1dcde2-maas-api-tls\") pod \"65cf4fa6-618d-44d2-91a6-2b8d7b1dcde2\" (UID: \"65cf4fa6-618d-44d2-91a6-2b8d7b1dcde2\") "
Oct 31 01:36:19.464685 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:36:19.464599    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/65cf4fa6-618d-44d2-91a6-2b8d7b1dcde2-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/65cf4fa6-618d-44d2-91a6-2b8d7b1dcde2-openshift-service-ca\") pod \"65cf4fa6-618d-44d2-91a6-2b8d7b1dcde2\" (UID: \"65cf4fa6-618d-44d2-91a6-2b8d7b1dcde2\") "
Oct 31 01:36:19.464775 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:36:19.464724    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/65cf4fa6-618d-44d2-91a6-2b8d7b1dcde2-kube-api-access-nr2bv\" (UniqueName: \"kubernetes.io/projected/65cf4fa6-618d-44d2-91a6-2b8d7b1dcde2-kube-api-access-nr2bv\") pod \"65cf4fa6-618d-44d2-91a6-2b8d7b1dcde2\" (UID: \"65cf4fa6-618d-44d2-91a6-2b8d7b1dcde2\") "
Oct 31 01:36:19.464775 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:36:19.464765    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/65cf4fa6-618d-44d2-91a6-2b8d7b1dcde2-metrics-certs\" (UniqueName: \"kubernetes.io/secret/65cf4fa6-618d-44d2-91a6-2b8d7b1dcde2-metrics-certs\") pod \"65cf4fa6-618d-44d2-91a6-2b8d7b1dcde2\" (UID: \"65cf4fa6-618d-44d2-91a6-2b8d7b1dcde2\") "
Oct 31 01:36:19.464931 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:36:19.464911    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/65cf4fa6-618d-44d2-91a6-2b8d7b1dcde2-openshift-service-ca" pod "65cf4fa6-618d-44d2-91a6-2b8d7b1dcde2" (UID: "65cf4fa6-618d-44d2-91a6-2b8d7b1dcde2"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:36:19.465207 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:36:19.465187    4964 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/65cf4fa6-618d-44d2-91a6-2b8d7b1dcde2-openshift-service-ca\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:36:19.466701 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:36:19.466677    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/65cf4fa6-618d-44d2-91a6-2b8d7b1dcde2-maas-api-tls" pod "65cf4fa6-618d-44d2-91a6-2b8d7b1dcde2" (UID: "65cf4fa6-618d-44d2-91a6-2b8d7b1dcde2"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:36:19.466881 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:36:19.466861    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/65cf4fa6-618d-44d2-91a6-2b8d7b1dcde2-metrics-certs" pod "65cf4fa6-618d-44d2-91a6-2b8d7b1dcde2" (UID: "65cf4fa6-618d-44d2-91a6-2b8d7b1dcde2"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:36:19.466980 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:36:19.466966    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/65cf4fa6-618d-44d2-91a6-2b8d7b1dcde2-kube-api-access-nr2bv" pod "65cf4fa6-618d-44d2-91a6-2b8d7b1dcde2" (UID: "65cf4fa6-618d-44d2-91a6-2b8d7b1dcde2"). InnerVolumeSpecName "kube-api-access-nr2bv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:36:19.567965 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:36:19.566901    4964 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/65cf4fa6-618d-44d2-91a6-2b8d7b1dcde2-maas-api-tls\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:36:19.567965 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:36:19.566936    4964 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nr2bv\" (UniqueName: \"kubernetes.io/projected/65cf4fa6-618d-44d2-91a6-2b8d7b1dcde2-kube-api-access-nr2bv\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:36:19.567965 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:36:19.566951    4964 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/65cf4fa6-618d-44d2-91a6-2b8d7b1dcde2-metrics-certs\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:36:19.733716 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:36:19.733679    4964 generic.go:356] "Generic (PLEG): container finished" podID="65cf4fa6-618d-44d2-91a6-2b8d7b1dcde2" containerID="c7a8511df73e48da71ad92a53f2441f1b3ac4d0d8bfd14baa046a5f3e8ac3604" exitCode=0
Oct 31 01:36:19.733911 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:36:19.733740    4964 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bc2046-7988f9b774-pz82x"
Oct 31 01:36:19.733911 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:36:19.733794    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bc2046-7988f9b774-pz82x" event={"ID":"65cf4fa6-618d-44d2-91a6-2b8d7b1dcde2","Type":"ContainerDied","Data":"c7a8511df73e48da71ad92a53f2441f1b3ac4d0d8bfd14baa046a5f3e8ac3604"}
Oct 31 01:36:19.733911 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:36:19.733845    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bc2046-7988f9b774-pz82x" event={"ID":"65cf4fa6-618d-44d2-91a6-2b8d7b1dcde2","Type":"ContainerDied","Data":"a2e756ab327ef7ff406a4df2ea329002c8632b5f6975f7bd05ae414e83dfbd61"}
Oct 31 01:36:19.733911 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:36:19.733872    4964 scope.go:122] "RemoveContainer" containerID="c7a8511df73e48da71ad92a53f2441f1b3ac4d0d8bfd14baa046a5f3e8ac3604"
Oct 31 01:36:19.745675 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:36:19.745657    4964 scope.go:122] "RemoveContainer" containerID="fdcdb07f25434852f6a2fccd7cff41e1655d9fa1884d5084b9d0694840dd4f53"
Oct 31 01:36:19.754548 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:36:19.754531    4964 scope.go:122] "RemoveContainer" containerID="c7a8511df73e48da71ad92a53f2441f1b3ac4d0d8bfd14baa046a5f3e8ac3604"
Oct 31 01:36:19.754832 ip-10-0-133-197 kubenswrapper[4964]: E1031 01:36:19.754811    4964 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c7a8511df73e48da71ad92a53f2441f1b3ac4d0d8bfd14baa046a5f3e8ac3604\": container with ID starting with c7a8511df73e48da71ad92a53f2441f1b3ac4d0d8bfd14baa046a5f3e8ac3604 not found: ID does not exist" containerID="c7a8511df73e48da71ad92a53f2441f1b3ac4d0d8bfd14baa046a5f3e8ac3604"
Oct 31 01:36:19.754883 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:36:19.754841    4964 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c7a8511df73e48da71ad92a53f2441f1b3ac4d0d8bfd14baa046a5f3e8ac3604"} err="failed to get container status \"c7a8511df73e48da71ad92a53f2441f1b3ac4d0d8bfd14baa046a5f3e8ac3604\": rpc error: code = NotFound desc = could not find container \"c7a8511df73e48da71ad92a53f2441f1b3ac4d0d8bfd14baa046a5f3e8ac3604\": container with ID starting with c7a8511df73e48da71ad92a53f2441f1b3ac4d0d8bfd14baa046a5f3e8ac3604 not found: ID does not exist"
Oct 31 01:36:19.754883 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:36:19.754857    4964 scope.go:122] "RemoveContainer" containerID="fdcdb07f25434852f6a2fccd7cff41e1655d9fa1884d5084b9d0694840dd4f53"
Oct 31 01:36:19.755130 ip-10-0-133-197 kubenswrapper[4964]: E1031 01:36:19.755111    4964 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fdcdb07f25434852f6a2fccd7cff41e1655d9fa1884d5084b9d0694840dd4f53\": container with ID starting with fdcdb07f25434852f6a2fccd7cff41e1655d9fa1884d5084b9d0694840dd4f53 not found: ID does not exist" containerID="fdcdb07f25434852f6a2fccd7cff41e1655d9fa1884d5084b9d0694840dd4f53"
Oct 31 01:36:19.755173 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:36:19.755138    4964 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fdcdb07f25434852f6a2fccd7cff41e1655d9fa1884d5084b9d0694840dd4f53"} err="failed to get container status \"fdcdb07f25434852f6a2fccd7cff41e1655d9fa1884d5084b9d0694840dd4f53\": rpc error: code = NotFound desc = could not find container \"fdcdb07f25434852f6a2fccd7cff41e1655d9fa1884d5084b9d0694840dd4f53\": container with ID starting with fdcdb07f25434852f6a2fccd7cff41e1655d9fa1884d5084b9d0694840dd4f53 not found: ID does not exist"
Oct 31 01:36:19.899484 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:36:19.899449    4964 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-bc2046-7988f9b774-pz82x"]
Oct 31 01:36:19.902631 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:36:19.902606    4964 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-bc2046-7988f9b774-pz82x"]
Oct 31 01:36:20.815521 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:36:20.815488    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:36:21.064509 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:36:21.064476    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:36:21.268148 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:36:21.268077    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:36:21.760612 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:36:21.760581    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:36:21.777287 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:36:21.777218    4964 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="65cf4fa6-618d-44d2-91a6-2b8d7b1dcde2" path="/var/lib/kubelet/pods/65cf4fa6-618d-44d2-91a6-2b8d7b1dcde2/volumes"
Oct 31 01:36:22.747344 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:36:22.747296    4964 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-9e5368-7b876c5d49-4zpqb"]
Oct 31 01:36:22.892670 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:36:22.892638    4964 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-9e5368-7f65f554f5-g7mgs"]
Oct 31 01:36:23.056222 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:36:23.056139    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:36:23.241682 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:36:23.241641    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:36:23.427154 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:36:23.427055    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:36:25.989333 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:36:25.989293    4964 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-71bc98-7fbd6bc889-kc2f4"]
Oct 31 01:36:25.989884 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:36:25.989869    4964 cpu_manager.go:414] "containerMap: removing container" podUID="65cf4fa6-618d-44d2-91a6-2b8d7b1dcde2" containerName="maas-api"
Oct 31 01:36:25.989984 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:36:25.989885    4964 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="65cf4fa6-618d-44d2-91a6-2b8d7b1dcde2" containerName="maas-api"
Oct 31 01:36:25.989984 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:36:25.989904    4964 cpu_manager.go:414] "containerMap: removing container" podUID="65cf4fa6-618d-44d2-91a6-2b8d7b1dcde2" containerName="maas-api"
Oct 31 01:36:25.989984 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:36:25.989909    4964 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="65cf4fa6-618d-44d2-91a6-2b8d7b1dcde2" containerName="maas-api"
Oct 31 01:36:25.990079 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:36:25.990031    4964 memory_manager.go:365] "RemoveStaleState removing state" podUID="65cf4fa6-618d-44d2-91a6-2b8d7b1dcde2" containerName="maas-api"
Oct 31 01:36:25.990079 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:36:25.990040    4964 memory_manager.go:365] "RemoveStaleState removing state" podUID="65cf4fa6-618d-44d2-91a6-2b8d7b1dcde2" containerName="maas-api"
Oct 31 01:36:25.994945 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:36:25.994926    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-71bc98-7fbd6bc889-kc2f4"
Oct 31 01:36:26.050907 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:36:26.050880    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-71bc98-7fbd6bc889-kc2f4"]
Oct 31 01:36:26.069518 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:36:26.069486    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/95f88410-b151-4896-af94-ab455dfc8ad2-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w1-71bc98-7fbd6bc889-kc2f4\" (UID: \"95f88410-b151-4896-af94-ab455dfc8ad2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-71bc98-7fbd6bc889-kc2f4"
Oct 31 01:36:26.069666 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:36:26.069585    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/95f88410-b151-4896-af94-ab455dfc8ad2-maas-api-tls\") pod \"maas-api-e2e-shared-b-w1-71bc98-7fbd6bc889-kc2f4\" (UID: \"95f88410-b151-4896-af94-ab455dfc8ad2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-71bc98-7fbd6bc889-kc2f4"
Oct 31 01:36:26.069747 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:36:26.069716    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rg7j6\" (UniqueName: \"kubernetes.io/projected/95f88410-b151-4896-af94-ab455dfc8ad2-kube-api-access-rg7j6\") pod \"maas-api-e2e-shared-b-w1-71bc98-7fbd6bc889-kc2f4\" (UID: \"95f88410-b151-4896-af94-ab455dfc8ad2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-71bc98-7fbd6bc889-kc2f4"
Oct 31 01:36:26.069809 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:36:26.069749    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/95f88410-b151-4896-af94-ab455dfc8ad2-metrics-certs\") pod \"maas-api-e2e-shared-b-w1-71bc98-7fbd6bc889-kc2f4\" (UID: \"95f88410-b151-4896-af94-ab455dfc8ad2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-71bc98-7fbd6bc889-kc2f4"
Oct 31 01:36:26.118128 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:36:26.118102    4964 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w1-71bc98-serving-cert\""
Oct 31 01:36:26.170683 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:36:26.170655    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/95f88410-b151-4896-af94-ab455dfc8ad2-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w1-71bc98-7fbd6bc889-kc2f4\" (UID: \"95f88410-b151-4896-af94-ab455dfc8ad2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-71bc98-7fbd6bc889-kc2f4"
Oct 31 01:36:26.170849 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:36:26.170776    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/95f88410-b151-4896-af94-ab455dfc8ad2-maas-api-tls\") pod \"maas-api-e2e-shared-b-w1-71bc98-7fbd6bc889-kc2f4\" (UID: \"95f88410-b151-4896-af94-ab455dfc8ad2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-71bc98-7fbd6bc889-kc2f4"
Oct 31 01:36:26.170924 ip-10-0-133-197 kubenswrapper[4964]: E1031 01:36:26.170898    4964 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-71bc98-serving-cert: secret "maas-api-e2e-shared-b-w1-71bc98-serving-cert" not found
Oct 31 01:36:26.170981 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:36:26.170922    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rg7j6\" (UniqueName: \"kubernetes.io/projected/95f88410-b151-4896-af94-ab455dfc8ad2-kube-api-access-rg7j6\") pod \"maas-api-e2e-shared-b-w1-71bc98-7fbd6bc889-kc2f4\" (UID: \"95f88410-b151-4896-af94-ab455dfc8ad2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-71bc98-7fbd6bc889-kc2f4"
Oct 31 01:36:26.170981 ip-10-0-133-197 kubenswrapper[4964]: E1031 01:36:26.170957    4964 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/95f88410-b151-4896-af94-ab455dfc8ad2-maas-api-tls podName:95f88410-b151-4896-af94-ab455dfc8ad2 nodeName:}" failed. No retries permitted until 2026-10-31 01:36:26.670942565 +0000 UTC m=+1955.147508922 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/95f88410-b151-4896-af94-ab455dfc8ad2-maas-api-tls") pod "maas-api-e2e-shared-b-w1-71bc98-7fbd6bc889-kc2f4" (UID: "95f88410-b151-4896-af94-ab455dfc8ad2") : secret "maas-api-e2e-shared-b-w1-71bc98-serving-cert" not found
Oct 31 01:36:26.171098 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:36:26.170979    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/95f88410-b151-4896-af94-ab455dfc8ad2-metrics-certs\") pod \"maas-api-e2e-shared-b-w1-71bc98-7fbd6bc889-kc2f4\" (UID: \"95f88410-b151-4896-af94-ab455dfc8ad2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-71bc98-7fbd6bc889-kc2f4"
Oct 31 01:36:26.171435 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:36:26.171410    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/95f88410-b151-4896-af94-ab455dfc8ad2-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w1-71bc98-7fbd6bc889-kc2f4\" (UID: \"95f88410-b151-4896-af94-ab455dfc8ad2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-71bc98-7fbd6bc889-kc2f4"
Oct 31 01:36:26.173504 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:36:26.173486    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/95f88410-b151-4896-af94-ab455dfc8ad2-metrics-certs\") pod \"maas-api-e2e-shared-b-w1-71bc98-7fbd6bc889-kc2f4\" (UID: \"95f88410-b151-4896-af94-ab455dfc8ad2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-71bc98-7fbd6bc889-kc2f4"
Oct 31 01:36:26.229811 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:36:26.229780    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rg7j6\" (UniqueName: \"kubernetes.io/projected/95f88410-b151-4896-af94-ab455dfc8ad2-kube-api-access-rg7j6\") pod \"maas-api-e2e-shared-b-w1-71bc98-7fbd6bc889-kc2f4\" (UID: \"95f88410-b151-4896-af94-ab455dfc8ad2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-71bc98-7fbd6bc889-kc2f4"
Oct 31 01:36:26.243714 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:36:26.243647    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:36:26.472194 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:36:26.472152    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:36:26.677449 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:36:26.677355    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/95f88410-b151-4896-af94-ab455dfc8ad2-maas-api-tls\") pod \"maas-api-e2e-shared-b-w1-71bc98-7fbd6bc889-kc2f4\" (UID: \"95f88410-b151-4896-af94-ab455dfc8ad2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-71bc98-7fbd6bc889-kc2f4"
Oct 31 01:36:26.679901 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:36:26.679879    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/95f88410-b151-4896-af94-ab455dfc8ad2-maas-api-tls\") pod \"maas-api-e2e-shared-b-w1-71bc98-7fbd6bc889-kc2f4\" (UID: \"95f88410-b151-4896-af94-ab455dfc8ad2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-71bc98-7fbd6bc889-kc2f4"
Oct 31 01:36:26.910372 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:36:26.910309    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-71bc98-7fbd6bc889-kc2f4"
Oct 31 01:36:27.297360 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:36:27.297333    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-71bc98-7fbd6bc889-kc2f4"]
Oct 31 01:36:27.298776 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:36:27.298743    4964 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod95f88410_b151_4896_af94_ab455dfc8ad2.slice/crio-25d5aba1ecfe6c2b01d99a835a72692320fc048d953d597bbc72c0a55e9a7bc7 WatchSource:0}: Error finding container 25d5aba1ecfe6c2b01d99a835a72692320fc048d953d597bbc72c0a55e9a7bc7: Status 404 returned error can't find the container with id 25d5aba1ecfe6c2b01d99a835a72692320fc048d953d597bbc72c0a55e9a7bc7
Oct 31 01:36:27.748452 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:36:27.748413    4964 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-9e5368-7b876c5d49-4zpqb" podUID="8fb32c6d-7453-44d3-a54e-5470f274d555" containerName="payload-processing" containerID="cri-o://7a0be0aa3e5802ac6b06744841da7df51b1b27de45071d47ac935c4472fcefcd" gracePeriod=25
Oct 31 01:36:27.759442 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:36:27.759287    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:36:27.796114 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:36:27.796079    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-71bc98-7fbd6bc889-kc2f4" event={"ID":"95f88410-b151-4896-af94-ab455dfc8ad2","Type":"ContainerStarted","Data":"25d5aba1ecfe6c2b01d99a835a72692320fc048d953d597bbc72c0a55e9a7bc7"}
Oct 31 01:36:27.893505 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:36:27.893468    4964 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9e5368-7f65f554f5-g7mgs" podUID="16f5f3bb-ce5c-4e0d-aaca-8c796a1a9b1b" containerName="payload-pre-processing" containerID="cri-o://9433d37d5ddfe6011f14af53bed242ffa5cf296c97ed542d8ebd6fdcb087e19b" gracePeriod=25
Oct 31 01:36:28.065301 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:36:28.049275    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:36:28.076622 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:36:28.076573    4964 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-9e5368-7b876c5d49-4zpqb"
Oct 31 01:36:28.261169 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:36:28.259889    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8fb32c6d-7453-44d3-a54e-5470f274d555-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8fb32c6d-7453-44d3-a54e-5470f274d555-plugins-config-volume\") pod \"8fb32c6d-7453-44d3-a54e-5470f274d555\" (UID: \"8fb32c6d-7453-44d3-a54e-5470f274d555\") "
Oct 31 01:36:28.261169 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:36:28.259990    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8fb32c6d-7453-44d3-a54e-5470f274d555-kube-api-access-qsphl\" (UniqueName: \"kubernetes.io/projected/8fb32c6d-7453-44d3-a54e-5470f274d555-kube-api-access-qsphl\") pod \"8fb32c6d-7453-44d3-a54e-5470f274d555\" (UID: \"8fb32c6d-7453-44d3-a54e-5470f274d555\") "
Oct 31 01:36:28.261703 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:36:28.261656    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8fb32c6d-7453-44d3-a54e-5470f274d555-plugins-config-volume" pod "8fb32c6d-7453-44d3-a54e-5470f274d555" (UID: "8fb32c6d-7453-44d3-a54e-5470f274d555"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:36:28.273643 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:36:28.268476    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8fb32c6d-7453-44d3-a54e-5470f274d555-kube-api-access-qsphl" pod "8fb32c6d-7453-44d3-a54e-5470f274d555" (UID: "8fb32c6d-7453-44d3-a54e-5470f274d555"). InnerVolumeSpecName "kube-api-access-qsphl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:36:28.273784 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:36:28.273696    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:36:28.282806 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:36:28.282783    4964 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9e5368-7f65f554f5-g7mgs"
Oct 31 01:36:28.361541 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:36:28.361511    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/16f5f3bb-ce5c-4e0d-aaca-8c796a1a9b1b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/16f5f3bb-ce5c-4e0d-aaca-8c796a1a9b1b-plugins-config-volume\") pod \"16f5f3bb-ce5c-4e0d-aaca-8c796a1a9b1b\" (UID: \"16f5f3bb-ce5c-4e0d-aaca-8c796a1a9b1b\") "
Oct 31 01:36:28.361865 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:36:28.361604    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/16f5f3bb-ce5c-4e0d-aaca-8c796a1a9b1b-kube-api-access-hff9r\" (UniqueName: \"kubernetes.io/projected/16f5f3bb-ce5c-4e0d-aaca-8c796a1a9b1b-kube-api-access-hff9r\") pod \"16f5f3bb-ce5c-4e0d-aaca-8c796a1a9b1b\" (UID: \"16f5f3bb-ce5c-4e0d-aaca-8c796a1a9b1b\") "
Oct 31 01:36:28.362152 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:36:28.362123    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/16f5f3bb-ce5c-4e0d-aaca-8c796a1a9b1b-plugins-config-volume" pod "16f5f3bb-ce5c-4e0d-aaca-8c796a1a9b1b" (UID: "16f5f3bb-ce5c-4e0d-aaca-8c796a1a9b1b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:36:28.362531 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:36:28.362508    4964 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8fb32c6d-7453-44d3-a54e-5470f274d555-plugins-config-volume\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:36:28.362597 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:36:28.362538    4964 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qsphl\" (UniqueName: \"kubernetes.io/projected/8fb32c6d-7453-44d3-a54e-5470f274d555-kube-api-access-qsphl\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:36:28.366216 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:36:28.366186    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/16f5f3bb-ce5c-4e0d-aaca-8c796a1a9b1b-kube-api-access-hff9r" pod "16f5f3bb-ce5c-4e0d-aaca-8c796a1a9b1b" (UID: "16f5f3bb-ce5c-4e0d-aaca-8c796a1a9b1b"). InnerVolumeSpecName "kube-api-access-hff9r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:36:28.463582 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:36:28.463556    4964 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/16f5f3bb-ce5c-4e0d-aaca-8c796a1a9b1b-plugins-config-volume\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:36:28.463582 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:36:28.463581    4964 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hff9r\" (UniqueName: \"kubernetes.io/projected/16f5f3bb-ce5c-4e0d-aaca-8c796a1a9b1b-kube-api-access-hff9r\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:36:28.803929 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:36:28.803899    4964 generic.go:356] "Generic (PLEG): container finished" podID="16f5f3bb-ce5c-4e0d-aaca-8c796a1a9b1b" containerID="9433d37d5ddfe6011f14af53bed242ffa5cf296c97ed542d8ebd6fdcb087e19b" exitCode=0
Oct 31 01:36:28.804115 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:36:28.803948    4964 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9e5368-7f65f554f5-g7mgs"
Oct 31 01:36:28.804115 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:36:28.803978    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9e5368-7f65f554f5-g7mgs" event={"ID":"16f5f3bb-ce5c-4e0d-aaca-8c796a1a9b1b","Type":"ContainerDied","Data":"9433d37d5ddfe6011f14af53bed242ffa5cf296c97ed542d8ebd6fdcb087e19b"}
Oct 31 01:36:28.804115 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:36:28.804044    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9e5368-7f65f554f5-g7mgs" event={"ID":"16f5f3bb-ce5c-4e0d-aaca-8c796a1a9b1b","Type":"ContainerDied","Data":"ddab1c82c3a9331ffc53ed13d297cb8b51a51783e7af0297742a382789805d80"}
Oct 31 01:36:28.804115 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:36:28.804074    4964 scope.go:122] "RemoveContainer" containerID="9433d37d5ddfe6011f14af53bed242ffa5cf296c97ed542d8ebd6fdcb087e19b"
Oct 31 01:36:28.805799 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:36:28.805774    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-71bc98-7fbd6bc889-kc2f4" event={"ID":"95f88410-b151-4896-af94-ab455dfc8ad2","Type":"ContainerStarted","Data":"c79b5eb636d9b814a65a6ae238c71056061895b813fb44067e89f857fd8ec3ef"}
Oct 31 01:36:28.805970 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:36:28.805946    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-71bc98-7fbd6bc889-kc2f4"
Oct 31 01:36:28.807384 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:36:28.807358    4964 generic.go:356] "Generic (PLEG): container finished" podID="8fb32c6d-7453-44d3-a54e-5470f274d555" containerID="7a0be0aa3e5802ac6b06744841da7df51b1b27de45071d47ac935c4472fcefcd" exitCode=0
Oct 31 01:36:28.807494 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:36:28.807442    4964 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-9e5368-7b876c5d49-4zpqb"
Oct 31 01:36:28.807494 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:36:28.807445    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-9e5368-7b876c5d49-4zpqb" event={"ID":"8fb32c6d-7453-44d3-a54e-5470f274d555","Type":"ContainerDied","Data":"7a0be0aa3e5802ac6b06744841da7df51b1b27de45071d47ac935c4472fcefcd"}
Oct 31 01:36:28.807596 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:36:28.807506    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-9e5368-7b876c5d49-4zpqb" event={"ID":"8fb32c6d-7453-44d3-a54e-5470f274d555","Type":"ContainerDied","Data":"7d7da6bbad9c862ccb5a423f6b451e00d6d55684942484ceb88b2424284ee8ff"}
Oct 31 01:36:28.821522 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:36:28.821504    4964 scope.go:122] "RemoveContainer" containerID="9433d37d5ddfe6011f14af53bed242ffa5cf296c97ed542d8ebd6fdcb087e19b"
Oct 31 01:36:28.821799 ip-10-0-133-197 kubenswrapper[4964]: E1031 01:36:28.821781    4964 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9433d37d5ddfe6011f14af53bed242ffa5cf296c97ed542d8ebd6fdcb087e19b\": container with ID starting with 9433d37d5ddfe6011f14af53bed242ffa5cf296c97ed542d8ebd6fdcb087e19b not found: ID does not exist" containerID="9433d37d5ddfe6011f14af53bed242ffa5cf296c97ed542d8ebd6fdcb087e19b"
Oct 31 01:36:28.821864 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:36:28.821806    4964 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9433d37d5ddfe6011f14af53bed242ffa5cf296c97ed542d8ebd6fdcb087e19b"} err="failed to get container status \"9433d37d5ddfe6011f14af53bed242ffa5cf296c97ed542d8ebd6fdcb087e19b\": rpc error: code = NotFound desc = could not find container \"9433d37d5ddfe6011f14af53bed242ffa5cf296c97ed542d8ebd6fdcb087e19b\": container with ID starting with 9433d37d5ddfe6011f14af53bed242ffa5cf296c97ed542d8ebd6fdcb087e19b not found: ID does not exist"
Oct 31 01:36:28.821864 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:36:28.821823    4964 scope.go:122] "RemoveContainer" containerID="7a0be0aa3e5802ac6b06744841da7df51b1b27de45071d47ac935c4472fcefcd"
Oct 31 01:36:28.831249 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:36:28.831233    4964 scope.go:122] "RemoveContainer" containerID="7a0be0aa3e5802ac6b06744841da7df51b1b27de45071d47ac935c4472fcefcd"
Oct 31 01:36:28.831627 ip-10-0-133-197 kubenswrapper[4964]: E1031 01:36:28.831605    4964 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7a0be0aa3e5802ac6b06744841da7df51b1b27de45071d47ac935c4472fcefcd\": container with ID starting with 7a0be0aa3e5802ac6b06744841da7df51b1b27de45071d47ac935c4472fcefcd not found: ID does not exist" containerID="7a0be0aa3e5802ac6b06744841da7df51b1b27de45071d47ac935c4472fcefcd"
Oct 31 01:36:28.831710 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:36:28.831632    4964 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7a0be0aa3e5802ac6b06744841da7df51b1b27de45071d47ac935c4472fcefcd"} err="failed to get container status \"7a0be0aa3e5802ac6b06744841da7df51b1b27de45071d47ac935c4472fcefcd\": rpc error: code = NotFound desc = could not find container \"7a0be0aa3e5802ac6b06744841da7df51b1b27de45071d47ac935c4472fcefcd\": container with ID starting with 7a0be0aa3e5802ac6b06744841da7df51b1b27de45071d47ac935c4472fcefcd not found: ID does not exist"
Oct 31 01:36:28.968504 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:36:28.968474    4964 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-9e5368-7f65f554f5-g7mgs"]
Oct 31 01:36:28.972022 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:36:28.971997    4964 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-9e5368-7f65f554f5-g7mgs"]
Oct 31 01:36:29.080890 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:36:29.080805    4964 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-9e5368-7b876c5d49-4zpqb"]
Oct 31 01:36:29.083864 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:36:29.083838    4964 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-9e5368-7b876c5d49-4zpqb"]
Oct 31 01:36:29.200737 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:36:29.200668    4964 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-71bc98-7fbd6bc889-kc2f4" podStartSLOduration=3.194121759 podStartE2EDuration="4.200649757s" totalImagesPullingTime="1.006527998s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:36:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:36:29.195216741 +0000 UTC m=+1957.671783122" watchObservedRunningTime="2026-10-31 01:36:29.200649757 +0000 UTC m=+1957.677216138"
Oct 31 01:36:29.369048 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:36:29.368952    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:36:29.778796 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:36:29.778761    4964 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="16f5f3bb-ce5c-4e0d-aaca-8c796a1a9b1b" path="/var/lib/kubelet/pods/16f5f3bb-ce5c-4e0d-aaca-8c796a1a9b1b/volumes"
Oct 31 01:36:29.779342 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:36:29.779304    4964 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8fb32c6d-7453-44d3-a54e-5470f274d555" path="/var/lib/kubelet/pods/8fb32c6d-7453-44d3-a54e-5470f274d555/volumes"
Oct 31 01:36:29.977947 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:36:29.977911    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:36:30.399593 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:36:30.399555    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:36:30.404779 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:36:30.404752    4964 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-dtxgt"]
Oct 31 01:36:30.405015 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:36:30.404991    4964 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-dtxgt" podUID="b9feacee-50cf-472f-94c7-9d15afb5aefc" containerName="istio-proxy" containerID="cri-o://09baf1c6431b481e836ba2ce93d08e410a6af1927ad72a9341e7d4460ce74c1e" gracePeriod=30
Oct 31 01:36:31.507977 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:36:31.507942    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:36:32.495987 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:36:32.495952    4964 patch_prober.go:29] interesting pod/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-dtxgt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 01:36:32.496159 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:36:32.496002    4964 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-dtxgt" podUID="b9feacee-50cf-472f-94c7-9d15afb5aefc" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 01:36:34.599765 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:36:34.599733    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:36:34.823989 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:36:34.823959    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-71bc98-7fbd6bc889-kc2f4"
Oct 31 01:36:34.827704 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:36:34.827676    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:36:35.673339 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:36:35.673245    4964 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-dtxgt"
Oct 31 01:36:35.752058 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:36:35.752026    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b9feacee-50cf-472f-94c7-9d15afb5aefc-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b9feacee-50cf-472f-94c7-9d15afb5aefc-workload-certs\") pod \"b9feacee-50cf-472f-94c7-9d15afb5aefc\" (UID: \"b9feacee-50cf-472f-94c7-9d15afb5aefc\") "
Oct 31 01:36:35.752238 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:36:35.752078    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b9feacee-50cf-472f-94c7-9d15afb5aefc-istio-token\" (UniqueName: \"kubernetes.io/projected/b9feacee-50cf-472f-94c7-9d15afb5aefc-istio-token\") pod \"b9feacee-50cf-472f-94c7-9d15afb5aefc\" (UID: \"b9feacee-50cf-472f-94c7-9d15afb5aefc\") "
Oct 31 01:36:35.752238 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:36:35.752109    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b9feacee-50cf-472f-94c7-9d15afb5aefc-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9feacee-50cf-472f-94c7-9d15afb5aefc-workload-socket\") pod \"b9feacee-50cf-472f-94c7-9d15afb5aefc\" (UID: \"b9feacee-50cf-472f-94c7-9d15afb5aefc\") "
Oct 31 01:36:35.752238 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:36:35.752145    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b9feacee-50cf-472f-94c7-9d15afb5aefc-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b9feacee-50cf-472f-94c7-9d15afb5aefc-istiod-ca-cert\") pod \"b9feacee-50cf-472f-94c7-9d15afb5aefc\" (UID: \"b9feacee-50cf-472f-94c7-9d15afb5aefc\") "
Oct 31 01:36:35.752238 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:36:35.752217    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b9feacee-50cf-472f-94c7-9d15afb5aefc-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b9feacee-50cf-472f-94c7-9d15afb5aefc-istio-podinfo\") pod \"b9feacee-50cf-472f-94c7-9d15afb5aefc\" (UID: \"b9feacee-50cf-472f-94c7-9d15afb5aefc\") "
Oct 31 01:36:35.752543 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:36:35.752278    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b9feacee-50cf-472f-94c7-9d15afb5aefc-kube-api-access-vfpzx\" (UniqueName: \"kubernetes.io/projected/b9feacee-50cf-472f-94c7-9d15afb5aefc-kube-api-access-vfpzx\") pod \"b9feacee-50cf-472f-94c7-9d15afb5aefc\" (UID: \"b9feacee-50cf-472f-94c7-9d15afb5aefc\") "
Oct 31 01:36:35.752543 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:36:35.752303    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b9feacee-50cf-472f-94c7-9d15afb5aefc-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b9feacee-50cf-472f-94c7-9d15afb5aefc-istio-data\") pod \"b9feacee-50cf-472f-94c7-9d15afb5aefc\" (UID: \"b9feacee-50cf-472f-94c7-9d15afb5aefc\") "
Oct 31 01:36:35.752543 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:36:35.752301    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9feacee-50cf-472f-94c7-9d15afb5aefc-workload-certs" pod "b9feacee-50cf-472f-94c7-9d15afb5aefc" (UID: "b9feacee-50cf-472f-94c7-9d15afb5aefc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:36:35.752543 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:36:35.752401    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b9feacee-50cf-472f-94c7-9d15afb5aefc-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9feacee-50cf-472f-94c7-9d15afb5aefc-credential-socket\") pod \"b9feacee-50cf-472f-94c7-9d15afb5aefc\" (UID: \"b9feacee-50cf-472f-94c7-9d15afb5aefc\") "
Oct 31 01:36:35.752543 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:36:35.752483    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9feacee-50cf-472f-94c7-9d15afb5aefc-workload-socket" pod "b9feacee-50cf-472f-94c7-9d15afb5aefc" (UID: "b9feacee-50cf-472f-94c7-9d15afb5aefc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:36:35.752543 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:36:35.752503    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b9feacee-50cf-472f-94c7-9d15afb5aefc-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b9feacee-50cf-472f-94c7-9d15afb5aefc-istio-envoy\") pod \"b9feacee-50cf-472f-94c7-9d15afb5aefc\" (UID: \"b9feacee-50cf-472f-94c7-9d15afb5aefc\") "
Oct 31 01:36:35.752861 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:36:35.752584    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b9feacee-50cf-472f-94c7-9d15afb5aefc-istiod-ca-cert" pod "b9feacee-50cf-472f-94c7-9d15afb5aefc" (UID: "b9feacee-50cf-472f-94c7-9d15afb5aefc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:36:35.752861 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:36:35.752697    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9feacee-50cf-472f-94c7-9d15afb5aefc-credential-socket" pod "b9feacee-50cf-472f-94c7-9d15afb5aefc" (UID: "b9feacee-50cf-472f-94c7-9d15afb5aefc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:36:35.752861 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:36:35.752768    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9feacee-50cf-472f-94c7-9d15afb5aefc-istio-data" pod "b9feacee-50cf-472f-94c7-9d15afb5aefc" (UID: "b9feacee-50cf-472f-94c7-9d15afb5aefc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:36:35.753120 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:36:35.753098    4964 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9feacee-50cf-472f-94c7-9d15afb5aefc-credential-socket\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:36:35.753187 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:36:35.753129    4964 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b9feacee-50cf-472f-94c7-9d15afb5aefc-workload-certs\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:36:35.753187 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:36:35.753144    4964 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9feacee-50cf-472f-94c7-9d15afb5aefc-workload-socket\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:36:35.753187 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:36:35.753159    4964 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b9feacee-50cf-472f-94c7-9d15afb5aefc-istiod-ca-cert\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:36:35.753286 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:36:35.753174    4964 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b9feacee-50cf-472f-94c7-9d15afb5aefc-istio-data\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:36:35.754761 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:36:35.754735    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b9feacee-50cf-472f-94c7-9d15afb5aefc-istio-podinfo" pod "b9feacee-50cf-472f-94c7-9d15afb5aefc" (UID: "b9feacee-50cf-472f-94c7-9d15afb5aefc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 01:36:35.754864 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:36:35.754775    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9feacee-50cf-472f-94c7-9d15afb5aefc-istio-envoy" pod "b9feacee-50cf-472f-94c7-9d15afb5aefc" (UID: "b9feacee-50cf-472f-94c7-9d15afb5aefc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:36:35.755065 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:36:35.755032    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b9feacee-50cf-472f-94c7-9d15afb5aefc-kube-api-access-vfpzx" pod "b9feacee-50cf-472f-94c7-9d15afb5aefc" (UID: "b9feacee-50cf-472f-94c7-9d15afb5aefc"). InnerVolumeSpecName "kube-api-access-vfpzx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:36:35.755163 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:36:35.755070    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b9feacee-50cf-472f-94c7-9d15afb5aefc-istio-token" pod "b9feacee-50cf-472f-94c7-9d15afb5aefc" (UID: "b9feacee-50cf-472f-94c7-9d15afb5aefc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:36:35.854581 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:36:35.854549    4964 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b9feacee-50cf-472f-94c7-9d15afb5aefc-istio-podinfo\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:36:35.854581 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:36:35.854581    4964 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vfpzx\" (UniqueName: \"kubernetes.io/projected/b9feacee-50cf-472f-94c7-9d15afb5aefc-kube-api-access-vfpzx\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:36:35.854804 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:36:35.854599    4964 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b9feacee-50cf-472f-94c7-9d15afb5aefc-istio-envoy\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:36:35.854804 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:36:35.854613    4964 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b9feacee-50cf-472f-94c7-9d15afb5aefc-istio-token\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:36:35.860711 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:36:35.860683    4964 generic.go:356] "Generic (PLEG): container finished" podID="b9feacee-50cf-472f-94c7-9d15afb5aefc" containerID="09baf1c6431b481e836ba2ce93d08e410a6af1927ad72a9341e7d4460ce74c1e" exitCode=0
Oct 31 01:36:35.860838 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:36:35.860745    4964 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-dtxgt"
Oct 31 01:36:35.860838 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:36:35.860755    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-dtxgt" event={"ID":"b9feacee-50cf-472f-94c7-9d15afb5aefc","Type":"ContainerDied","Data":"09baf1c6431b481e836ba2ce93d08e410a6af1927ad72a9341e7d4460ce74c1e"}
Oct 31 01:36:35.860838 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:36:35.860799    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-dtxgt" event={"ID":"b9feacee-50cf-472f-94c7-9d15afb5aefc","Type":"ContainerDied","Data":"8b988eea33693eb4b3dfbdacf2ef776c0ff40b055525ad3bdc0648b141d8c098"}
Oct 31 01:36:35.860838 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:36:35.860820    4964 scope.go:122] "RemoveContainer" containerID="09baf1c6431b481e836ba2ce93d08e410a6af1927ad72a9341e7d4460ce74c1e"
Oct 31 01:36:35.873563 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:36:35.873546    4964 scope.go:122] "RemoveContainer" containerID="09baf1c6431b481e836ba2ce93d08e410a6af1927ad72a9341e7d4460ce74c1e"
Oct 31 01:36:35.873843 ip-10-0-133-197 kubenswrapper[4964]: E1031 01:36:35.873824    4964 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"09baf1c6431b481e836ba2ce93d08e410a6af1927ad72a9341e7d4460ce74c1e\": container with ID starting with 09baf1c6431b481e836ba2ce93d08e410a6af1927ad72a9341e7d4460ce74c1e not found: ID does not exist" containerID="09baf1c6431b481e836ba2ce93d08e410a6af1927ad72a9341e7d4460ce74c1e"
Oct 31 01:36:35.873921 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:36:35.873855    4964 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"09baf1c6431b481e836ba2ce93d08e410a6af1927ad72a9341e7d4460ce74c1e"} err="failed to get container status \"09baf1c6431b481e836ba2ce93d08e410a6af1927ad72a9341e7d4460ce74c1e\": rpc error: code = NotFound desc = could not find container \"09baf1c6431b481e836ba2ce93d08e410a6af1927ad72a9341e7d4460ce74c1e\": container with ID starting with 09baf1c6431b481e836ba2ce93d08e410a6af1927ad72a9341e7d4460ce74c1e not found: ID does not exist"
Oct 31 01:36:35.930456 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:36:35.930388    4964 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-57ea84-bd5bb4cd7-vkghl"]
Oct 31 01:36:35.930616 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:36:35.930596    4964 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-57ea84-bd5bb4cd7-vkghl" podUID="afa6c9e7-5acc-4894-ab75-38c00dc49ae2" containerName="maas-api" containerID="cri-o://8018a3510633d3e7a49693b3367564bae9905e35511caf06884f1b84756b0126" gracePeriod=30
Oct 31 01:36:36.034738 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:36:36.034705    4964 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-dtxgt"]
Oct 31 01:36:36.042194 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:36:36.042159    4964 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-dtxgt"]
Oct 31 01:36:36.050146 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:36:36.050077    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:36:36.223641 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:36:36.223618    4964 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-57ea84-bd5bb4cd7-vkghl"
Oct 31 01:36:36.361893 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:36:36.361864    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/afa6c9e7-5acc-4894-ab75-38c00dc49ae2-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/afa6c9e7-5acc-4894-ab75-38c00dc49ae2-openshift-service-ca\") pod \"afa6c9e7-5acc-4894-ab75-38c00dc49ae2\" (UID: \"afa6c9e7-5acc-4894-ab75-38c00dc49ae2\") "
Oct 31 01:36:36.362041 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:36:36.361969    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/afa6c9e7-5acc-4894-ab75-38c00dc49ae2-metrics-certs\" (UniqueName: \"kubernetes.io/secret/afa6c9e7-5acc-4894-ab75-38c00dc49ae2-metrics-certs\") pod \"afa6c9e7-5acc-4894-ab75-38c00dc49ae2\" (UID: \"afa6c9e7-5acc-4894-ab75-38c00dc49ae2\") "
Oct 31 01:36:36.362041 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:36:36.362035    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/afa6c9e7-5acc-4894-ab75-38c00dc49ae2-kube-api-access-dzq5w\" (UniqueName: \"kubernetes.io/projected/afa6c9e7-5acc-4894-ab75-38c00dc49ae2-kube-api-access-dzq5w\") pod \"afa6c9e7-5acc-4894-ab75-38c00dc49ae2\" (UID: \"afa6c9e7-5acc-4894-ab75-38c00dc49ae2\") "
Oct 31 01:36:36.362141 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:36:36.362122    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/afa6c9e7-5acc-4894-ab75-38c00dc49ae2-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/afa6c9e7-5acc-4894-ab75-38c00dc49ae2-maas-api-tls\") pod \"afa6c9e7-5acc-4894-ab75-38c00dc49ae2\" (UID: \"afa6c9e7-5acc-4894-ab75-38c00dc49ae2\") "
Oct 31 01:36:36.362558 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:36:36.362527    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/afa6c9e7-5acc-4894-ab75-38c00dc49ae2-openshift-service-ca" pod "afa6c9e7-5acc-4894-ab75-38c00dc49ae2" (UID: "afa6c9e7-5acc-4894-ab75-38c00dc49ae2"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:36:36.362709 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:36:36.362684    4964 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/afa6c9e7-5acc-4894-ab75-38c00dc49ae2-openshift-service-ca\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:36:36.364938 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:36:36.364915    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/afa6c9e7-5acc-4894-ab75-38c00dc49ae2-metrics-certs" pod "afa6c9e7-5acc-4894-ab75-38c00dc49ae2" (UID: "afa6c9e7-5acc-4894-ab75-38c00dc49ae2"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:36:36.365022 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:36:36.364938    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/afa6c9e7-5acc-4894-ab75-38c00dc49ae2-maas-api-tls" pod "afa6c9e7-5acc-4894-ab75-38c00dc49ae2" (UID: "afa6c9e7-5acc-4894-ab75-38c00dc49ae2"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:36:36.365022 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:36:36.365006    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/afa6c9e7-5acc-4894-ab75-38c00dc49ae2-kube-api-access-dzq5w" pod "afa6c9e7-5acc-4894-ab75-38c00dc49ae2" (UID: "afa6c9e7-5acc-4894-ab75-38c00dc49ae2"). InnerVolumeSpecName "kube-api-access-dzq5w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:36:36.464226 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:36:36.464195    4964 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/afa6c9e7-5acc-4894-ab75-38c00dc49ae2-metrics-certs\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:36:36.464226 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:36:36.464223    4964 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dzq5w\" (UniqueName: \"kubernetes.io/projected/afa6c9e7-5acc-4894-ab75-38c00dc49ae2-kube-api-access-dzq5w\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:36:36.464431 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:36:36.464234    4964 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/afa6c9e7-5acc-4894-ab75-38c00dc49ae2-maas-api-tls\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:36:36.869767 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:36:36.869739    4964 generic.go:356] "Generic (PLEG): container finished" podID="afa6c9e7-5acc-4894-ab75-38c00dc49ae2" containerID="8018a3510633d3e7a49693b3367564bae9905e35511caf06884f1b84756b0126" exitCode=0
Oct 31 01:36:36.870221 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:36:36.869789    4964 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-57ea84-bd5bb4cd7-vkghl"
Oct 31 01:36:36.870221 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:36:36.869817    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-57ea84-bd5bb4cd7-vkghl" event={"ID":"afa6c9e7-5acc-4894-ab75-38c00dc49ae2","Type":"ContainerDied","Data":"8018a3510633d3e7a49693b3367564bae9905e35511caf06884f1b84756b0126"}
Oct 31 01:36:36.870221 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:36:36.869871    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-57ea84-bd5bb4cd7-vkghl" event={"ID":"afa6c9e7-5acc-4894-ab75-38c00dc49ae2","Type":"ContainerDied","Data":"daaff8873aad701d00b643b32a704d38b3687e3328fbdd97138d79e17497b369"}
Oct 31 01:36:36.870221 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:36:36.869891    4964 scope.go:122] "RemoveContainer" containerID="8018a3510633d3e7a49693b3367564bae9905e35511caf06884f1b84756b0126"
Oct 31 01:36:36.880734 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:36:36.880717    4964 scope.go:122] "RemoveContainer" containerID="8018a3510633d3e7a49693b3367564bae9905e35511caf06884f1b84756b0126"
Oct 31 01:36:36.881026 ip-10-0-133-197 kubenswrapper[4964]: E1031 01:36:36.881008    4964 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8018a3510633d3e7a49693b3367564bae9905e35511caf06884f1b84756b0126\": container with ID starting with 8018a3510633d3e7a49693b3367564bae9905e35511caf06884f1b84756b0126 not found: ID does not exist" containerID="8018a3510633d3e7a49693b3367564bae9905e35511caf06884f1b84756b0126"
Oct 31 01:36:36.881084 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:36:36.881033    4964 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8018a3510633d3e7a49693b3367564bae9905e35511caf06884f1b84756b0126"} err="failed to get container status \"8018a3510633d3e7a49693b3367564bae9905e35511caf06884f1b84756b0126\": rpc error: code = NotFound desc = could not find container \"8018a3510633d3e7a49693b3367564bae9905e35511caf06884f1b84756b0126\": container with ID starting with 8018a3510633d3e7a49693b3367564bae9905e35511caf06884f1b84756b0126 not found: ID does not exist"
Oct 31 01:36:37.038491 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:36:37.038448    4964 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-57ea84-bd5bb4cd7-vkghl"]
Oct 31 01:36:37.042850 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:36:37.042813    4964 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-57ea84-bd5bb4cd7-vkghl"]
Oct 31 01:36:37.778242 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:36:37.778212    4964 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="afa6c9e7-5acc-4894-ab75-38c00dc49ae2" path="/var/lib/kubelet/pods/afa6c9e7-5acc-4894-ab75-38c00dc49ae2/volumes"
Oct 31 01:36:37.778719 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:36:37.778703    4964 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b9feacee-50cf-472f-94c7-9d15afb5aefc" path="/var/lib/kubelet/pods/b9feacee-50cf-472f-94c7-9d15afb5aefc/volumes"
Oct 31 01:36:37.953098 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:36:37.953060    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:36:38.137017 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:36:38.136915    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:36:38.326622 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:36:38.326586    4964 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-99c8bd-77f6bdf9dc-tqmfm"]
Oct 31 01:36:38.327574 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:36:38.327550    4964 cpu_manager.go:414] "containerMap: removing container" podUID="b9feacee-50cf-472f-94c7-9d15afb5aefc" containerName="istio-proxy"
Oct 31 01:36:38.327704 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:36:38.327575    4964 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b9feacee-50cf-472f-94c7-9d15afb5aefc" containerName="istio-proxy"
Oct 31 01:36:38.327704 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:36:38.327591    4964 cpu_manager.go:414] "containerMap: removing container" podUID="afa6c9e7-5acc-4894-ab75-38c00dc49ae2" containerName="maas-api"
Oct 31 01:36:38.327704 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:36:38.327599    4964 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="afa6c9e7-5acc-4894-ab75-38c00dc49ae2" containerName="maas-api"
Oct 31 01:36:38.327704 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:36:38.327637    4964 cpu_manager.go:414] "containerMap: removing container" podUID="16f5f3bb-ce5c-4e0d-aaca-8c796a1a9b1b" containerName="payload-pre-processing"
Oct 31 01:36:38.327704 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:36:38.327646    4964 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="16f5f3bb-ce5c-4e0d-aaca-8c796a1a9b1b" containerName="payload-pre-processing"
Oct 31 01:36:38.327704 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:36:38.327663    4964 cpu_manager.go:414] "containerMap: removing container" podUID="8fb32c6d-7453-44d3-a54e-5470f274d555" containerName="payload-processing"
Oct 31 01:36:38.327704 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:36:38.327670    4964 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8fb32c6d-7453-44d3-a54e-5470f274d555" containerName="payload-processing"
Oct 31 01:36:38.328075 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:36:38.327820    4964 memory_manager.go:365] "RemoveStaleState removing state" podUID="afa6c9e7-5acc-4894-ab75-38c00dc49ae2" containerName="maas-api"
Oct 31 01:36:38.328075 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:36:38.327838    4964 memory_manager.go:365] "RemoveStaleState removing state" podUID="16f5f3bb-ce5c-4e0d-aaca-8c796a1a9b1b" containerName="payload-pre-processing"
Oct 31 01:36:38.328075 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:36:38.327849    4964 memory_manager.go:365] "RemoveStaleState removing state" podUID="8fb32c6d-7453-44d3-a54e-5470f274d555" containerName="payload-processing"
Oct 31 01:36:38.328075 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:36:38.327867    4964 memory_manager.go:365] "RemoveStaleState removing state" podUID="b9feacee-50cf-472f-94c7-9d15afb5aefc" containerName="istio-proxy"
Oct 31 01:36:38.333499 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:36:38.333467    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-99c8bd-77f6bdf9dc-tqmfm"
Oct 31 01:36:38.385615 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:36:38.385589    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zk98f\" (UniqueName: \"kubernetes.io/projected/7c87ec79-9698-470d-a9b2-f119731eb072-kube-api-access-zk98f\") pod \"payload-pre-processing-e2e-ipp-a-99c8bd-77f6bdf9dc-tqmfm\" (UID: \"7c87ec79-9698-470d-a9b2-f119731eb072\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-99c8bd-77f6bdf9dc-tqmfm"
Oct 31 01:36:38.385768 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:36:38.385740    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7c87ec79-9698-470d-a9b2-f119731eb072-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-99c8bd-77f6bdf9dc-tqmfm\" (UID: \"7c87ec79-9698-470d-a9b2-f119731eb072\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-99c8bd-77f6bdf9dc-tqmfm"
Oct 31 01:36:38.386448 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:36:38.386429    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-99c8bd-77f6bdf9dc-tqmfm"]
Oct 31 01:36:38.415384 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:36:38.411496    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:36:38.432465 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:36:38.432430    4964 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-99c8bd\""
Oct 31 01:36:38.432625 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:36:38.432508    4964 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-99c8bd-dockercfg-9xht2\""
Oct 31 01:36:38.487257 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:36:38.487214    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zk98f\" (UniqueName: \"kubernetes.io/projected/7c87ec79-9698-470d-a9b2-f119731eb072-kube-api-access-zk98f\") pod \"payload-pre-processing-e2e-ipp-a-99c8bd-77f6bdf9dc-tqmfm\" (UID: \"7c87ec79-9698-470d-a9b2-f119731eb072\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-99c8bd-77f6bdf9dc-tqmfm"
Oct 31 01:36:38.487485 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:36:38.487386    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7c87ec79-9698-470d-a9b2-f119731eb072-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-99c8bd-77f6bdf9dc-tqmfm\" (UID: \"7c87ec79-9698-470d-a9b2-f119731eb072\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-99c8bd-77f6bdf9dc-tqmfm"
Oct 31 01:36:38.488170 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:36:38.488143    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7c87ec79-9698-470d-a9b2-f119731eb072-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-99c8bd-77f6bdf9dc-tqmfm\" (UID: \"7c87ec79-9698-470d-a9b2-f119731eb072\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-99c8bd-77f6bdf9dc-tqmfm"
Oct 31 01:36:38.489078 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:36:38.489057    4964 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-99c8bd-54b85b659f-9kck6"]
Oct 31 01:36:38.528443 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:36:38.527770    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-99c8bd-54b85b659f-9kck6"
Oct 31 01:36:38.548885 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:36:38.548853    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-99c8bd-54b85b659f-9kck6"]
Oct 31 01:36:38.555023 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:36:38.554990    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zk98f\" (UniqueName: \"kubernetes.io/projected/7c87ec79-9698-470d-a9b2-f119731eb072-kube-api-access-zk98f\") pod \"payload-pre-processing-e2e-ipp-a-99c8bd-77f6bdf9dc-tqmfm\" (UID: \"7c87ec79-9698-470d-a9b2-f119731eb072\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-99c8bd-77f6bdf9dc-tqmfm"
Oct 31 01:36:38.589297 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:36:38.589269    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6ac8d799-a669-446b-950c-69fc4d4fb26b-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-99c8bd-54b85b659f-9kck6\" (UID: \"6ac8d799-a669-446b-950c-69fc4d4fb26b\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-99c8bd-54b85b659f-9kck6"
Oct 31 01:36:38.589572 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:36:38.589552    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9tnrq\" (UniqueName: \"kubernetes.io/projected/6ac8d799-a669-446b-950c-69fc4d4fb26b-kube-api-access-9tnrq\") pod \"payload-processing-e2e-ipp-a-99c8bd-54b85b659f-9kck6\" (UID: \"6ac8d799-a669-446b-950c-69fc4d4fb26b\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-99c8bd-54b85b659f-9kck6"
Oct 31 01:36:38.647672 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:36:38.647645    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-99c8bd-77f6bdf9dc-tqmfm"
Oct 31 01:36:38.691116 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:36:38.691033    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9tnrq\" (UniqueName: \"kubernetes.io/projected/6ac8d799-a669-446b-950c-69fc4d4fb26b-kube-api-access-9tnrq\") pod \"payload-processing-e2e-ipp-a-99c8bd-54b85b659f-9kck6\" (UID: \"6ac8d799-a669-446b-950c-69fc4d4fb26b\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-99c8bd-54b85b659f-9kck6"
Oct 31 01:36:38.691922 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:36:38.691457    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6ac8d799-a669-446b-950c-69fc4d4fb26b-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-99c8bd-54b85b659f-9kck6\" (UID: \"6ac8d799-a669-446b-950c-69fc4d4fb26b\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-99c8bd-54b85b659f-9kck6"
Oct 31 01:36:38.692251 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:36:38.692229    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6ac8d799-a669-446b-950c-69fc4d4fb26b-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-99c8bd-54b85b659f-9kck6\" (UID: \"6ac8d799-a669-446b-950c-69fc4d4fb26b\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-99c8bd-54b85b659f-9kck6"
Oct 31 01:36:38.722387 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:36:38.716312    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:36:38.773101 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:36:38.773041    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9tnrq\" (UniqueName: \"kubernetes.io/projected/6ac8d799-a669-446b-950c-69fc4d4fb26b-kube-api-access-9tnrq\") pod \"payload-processing-e2e-ipp-a-99c8bd-54b85b659f-9kck6\" (UID: \"6ac8d799-a669-446b-950c-69fc4d4fb26b\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-99c8bd-54b85b659f-9kck6"
Oct 31 01:36:38.860646 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:36:38.860614    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-99c8bd-54b85b659f-9kck6"
Oct 31 01:36:39.037983 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:36:39.037951    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-99c8bd-77f6bdf9dc-tqmfm"]
Oct 31 01:36:39.039435 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:36:39.039411    4964 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c87ec79_9698_470d_a9b2_f119731eb072.slice/crio-5f3ab02fe0ca9fdec5e949abdedf7d22bce20daabfeb7d382a78ba772176b940 WatchSource:0}: Error finding container 5f3ab02fe0ca9fdec5e949abdedf7d22bce20daabfeb7d382a78ba772176b940: Status 404 returned error can't find the container with id 5f3ab02fe0ca9fdec5e949abdedf7d22bce20daabfeb7d382a78ba772176b940
Oct 31 01:36:39.253924 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:36:39.253898    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-99c8bd-54b85b659f-9kck6"]
Oct 31 01:36:39.256345 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:36:39.256285    4964 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ac8d799_a669_446b_950c_69fc4d4fb26b.slice/crio-be57b1725a3e57cc6b78a30098c71f5d3ad4339bb4502de8706ca20b305f0a64 WatchSource:0}: Error finding container be57b1725a3e57cc6b78a30098c71f5d3ad4339bb4502de8706ca20b305f0a64: Status 404 returned error can't find the container with id be57b1725a3e57cc6b78a30098c71f5d3ad4339bb4502de8706ca20b305f0a64
Oct 31 01:36:39.372887 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:36:39.372854    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:36:39.897773 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:36:39.897739    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-99c8bd-54b85b659f-9kck6" event={"ID":"6ac8d799-a669-446b-950c-69fc4d4fb26b","Type":"ContainerStarted","Data":"5e8ce5b77a5aac53b6b469b107586ccf7c6fed5c67b4012e36dc945f2b059ba3"}
Oct 31 01:36:39.897960 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:36:39.897803    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-99c8bd-54b85b659f-9kck6"
Oct 31 01:36:39.897960 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:36:39.897826    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-99c8bd-54b85b659f-9kck6" event={"ID":"6ac8d799-a669-446b-950c-69fc4d4fb26b","Type":"ContainerStarted","Data":"be57b1725a3e57cc6b78a30098c71f5d3ad4339bb4502de8706ca20b305f0a64"}
Oct 31 01:36:39.899707 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:36:39.899676    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-99c8bd-77f6bdf9dc-tqmfm" event={"ID":"7c87ec79-9698-470d-a9b2-f119731eb072","Type":"ContainerStarted","Data":"94cfa99d78de213db2dccd70495c199f2b70b23a0d7e1c601ef7e8060ad9e919"}
Oct 31 01:36:39.899871 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:36:39.899729    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-99c8bd-77f6bdf9dc-tqmfm"
Oct 31 01:36:39.899871 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:36:39.899749    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-99c8bd-77f6bdf9dc-tqmfm" event={"ID":"7c87ec79-9698-470d-a9b2-f119731eb072","Type":"ContainerStarted","Data":"5f3ab02fe0ca9fdec5e949abdedf7d22bce20daabfeb7d382a78ba772176b940"}
Oct 31 01:36:40.016199 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:36:40.016150    4964 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-99c8bd-54b85b659f-9kck6" podStartSLOduration=2.016137668 podStartE2EDuration="2.016137668s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:36:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:36:40.01164713 +0000 UTC m=+1968.488213511" watchObservedRunningTime="2026-10-31 01:36:40.016137668 +0000 UTC m=+1968.492704086"
Oct 31 01:36:40.130111 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:36:40.130051    4964 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-99c8bd-77f6bdf9dc-tqmfm" podStartSLOduration=2.130034415 podStartE2EDuration="2.130034415s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:36:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:36:40.127156655 +0000 UTC m=+1968.603723036" watchObservedRunningTime="2026-10-31 01:36:40.130034415 +0000 UTC m=+1968.606600844"
Oct 31 01:36:41.239357 ip-10-0-133-197 kubenswrapper[4964]: E1031 01:36:41.239301    4964 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q56x5" containerName="extractor"
Oct 31 01:36:43.206476 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:36:43.206437    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:36:45.390311 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:36:45.390280    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:36:45.612181 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:36:45.612144    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:36:45.797341 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:36:45.792034    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:36:46.689750 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:36:46.689720    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:36:47.185175 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:36:47.185145    4964 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-53af08-openshift-default-556ff7748f-4nqv7"]
Oct 31 01:36:47.189289 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:36:47.189266    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-53af08-openshift-default-556ff7748f-4nqv7"
Oct 31 01:36:47.260673 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:36:47.260639    4964 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-53af08-openshift-default-dockercfg-fsdjd\""
Oct 31 01:36:47.298902 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:36:47.298865    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/778c73f2-648f-4909-a1e9-a685d1d9a2c7-workload-socket\") pod \"e2e-ipp-a-53af08-openshift-default-556ff7748f-4nqv7\" (UID: \"778c73f2-648f-4909-a1e9-a685d1d9a2c7\") " pod="openshift-ingress/e2e-ipp-a-53af08-openshift-default-556ff7748f-4nqv7"
Oct 31 01:36:47.299206 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:36:47.299187    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/778c73f2-648f-4909-a1e9-a685d1d9a2c7-istio-token\") pod \"e2e-ipp-a-53af08-openshift-default-556ff7748f-4nqv7\" (UID: \"778c73f2-648f-4909-a1e9-a685d1d9a2c7\") " pod="openshift-ingress/e2e-ipp-a-53af08-openshift-default-556ff7748f-4nqv7"
Oct 31 01:36:47.299391 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:36:47.299377    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/778c73f2-648f-4909-a1e9-a685d1d9a2c7-istio-data\") pod \"e2e-ipp-a-53af08-openshift-default-556ff7748f-4nqv7\" (UID: \"778c73f2-648f-4909-a1e9-a685d1d9a2c7\") " pod="openshift-ingress/e2e-ipp-a-53af08-openshift-default-556ff7748f-4nqv7"
Oct 31 01:36:47.299600 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:36:47.299583    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7pw4z\" (UniqueName: \"kubernetes.io/projected/778c73f2-648f-4909-a1e9-a685d1d9a2c7-kube-api-access-7pw4z\") pod \"e2e-ipp-a-53af08-openshift-default-556ff7748f-4nqv7\" (UID: \"778c73f2-648f-4909-a1e9-a685d1d9a2c7\") " pod="openshift-ingress/e2e-ipp-a-53af08-openshift-default-556ff7748f-4nqv7"
Oct 31 01:36:47.299740 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:36:47.299727    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/778c73f2-648f-4909-a1e9-a685d1d9a2c7-istiod-ca-cert\") pod \"e2e-ipp-a-53af08-openshift-default-556ff7748f-4nqv7\" (UID: \"778c73f2-648f-4909-a1e9-a685d1d9a2c7\") " pod="openshift-ingress/e2e-ipp-a-53af08-openshift-default-556ff7748f-4nqv7"
Oct 31 01:36:47.299949 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:36:47.299936    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/778c73f2-648f-4909-a1e9-a685d1d9a2c7-credential-socket\") pod \"e2e-ipp-a-53af08-openshift-default-556ff7748f-4nqv7\" (UID: \"778c73f2-648f-4909-a1e9-a685d1d9a2c7\") " pod="openshift-ingress/e2e-ipp-a-53af08-openshift-default-556ff7748f-4nqv7"
Oct 31 01:36:47.300099 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:36:47.300086    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/778c73f2-648f-4909-a1e9-a685d1d9a2c7-istio-podinfo\") pod \"e2e-ipp-a-53af08-openshift-default-556ff7748f-4nqv7\" (UID: \"778c73f2-648f-4909-a1e9-a685d1d9a2c7\") " pod="openshift-ingress/e2e-ipp-a-53af08-openshift-default-556ff7748f-4nqv7"
Oct 31 01:36:47.300236 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:36:47.300222    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/778c73f2-648f-4909-a1e9-a685d1d9a2c7-workload-certs\") pod \"e2e-ipp-a-53af08-openshift-default-556ff7748f-4nqv7\" (UID: \"778c73f2-648f-4909-a1e9-a685d1d9a2c7\") " pod="openshift-ingress/e2e-ipp-a-53af08-openshift-default-556ff7748f-4nqv7"
Oct 31 01:36:47.300410 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:36:47.300392    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/778c73f2-648f-4909-a1e9-a685d1d9a2c7-istio-envoy\") pod \"e2e-ipp-a-53af08-openshift-default-556ff7748f-4nqv7\" (UID: \"778c73f2-648f-4909-a1e9-a685d1d9a2c7\") " pod="openshift-ingress/e2e-ipp-a-53af08-openshift-default-556ff7748f-4nqv7"
Oct 31 01:36:47.325411 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:36:47.325376    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-53af08-openshift-default-556ff7748f-4nqv7"]
Oct 31 01:36:47.329769 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:36:47.329071    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:36:47.401709 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:36:47.401672    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/778c73f2-648f-4909-a1e9-a685d1d9a2c7-istio-podinfo\") pod \"e2e-ipp-a-53af08-openshift-default-556ff7748f-4nqv7\" (UID: \"778c73f2-648f-4909-a1e9-a685d1d9a2c7\") " pod="openshift-ingress/e2e-ipp-a-53af08-openshift-default-556ff7748f-4nqv7"
Oct 31 01:36:47.401872 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:36:47.401742    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/778c73f2-648f-4909-a1e9-a685d1d9a2c7-workload-certs\") pod \"e2e-ipp-a-53af08-openshift-default-556ff7748f-4nqv7\" (UID: \"778c73f2-648f-4909-a1e9-a685d1d9a2c7\") " pod="openshift-ingress/e2e-ipp-a-53af08-openshift-default-556ff7748f-4nqv7"
Oct 31 01:36:47.401872 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:36:47.401797    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/778c73f2-648f-4909-a1e9-a685d1d9a2c7-istio-envoy\") pod \"e2e-ipp-a-53af08-openshift-default-556ff7748f-4nqv7\" (UID: \"778c73f2-648f-4909-a1e9-a685d1d9a2c7\") " pod="openshift-ingress/e2e-ipp-a-53af08-openshift-default-556ff7748f-4nqv7"
Oct 31 01:36:47.401872 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:36:47.401853    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/778c73f2-648f-4909-a1e9-a685d1d9a2c7-workload-socket\") pod \"e2e-ipp-a-53af08-openshift-default-556ff7748f-4nqv7\" (UID: \"778c73f2-648f-4909-a1e9-a685d1d9a2c7\") " pod="openshift-ingress/e2e-ipp-a-53af08-openshift-default-556ff7748f-4nqv7"
Oct 31 01:36:47.402043 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:36:47.401897    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/778c73f2-648f-4909-a1e9-a685d1d9a2c7-istio-token\") pod \"e2e-ipp-a-53af08-openshift-default-556ff7748f-4nqv7\" (UID: \"778c73f2-648f-4909-a1e9-a685d1d9a2c7\") " pod="openshift-ingress/e2e-ipp-a-53af08-openshift-default-556ff7748f-4nqv7"
Oct 31 01:36:47.402043 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:36:47.401947    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/778c73f2-648f-4909-a1e9-a685d1d9a2c7-istio-data\") pod \"e2e-ipp-a-53af08-openshift-default-556ff7748f-4nqv7\" (UID: \"778c73f2-648f-4909-a1e9-a685d1d9a2c7\") " pod="openshift-ingress/e2e-ipp-a-53af08-openshift-default-556ff7748f-4nqv7"
Oct 31 01:36:47.402043 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:36:47.402017    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7pw4z\" (UniqueName: \"kubernetes.io/projected/778c73f2-648f-4909-a1e9-a685d1d9a2c7-kube-api-access-7pw4z\") pod \"e2e-ipp-a-53af08-openshift-default-556ff7748f-4nqv7\" (UID: \"778c73f2-648f-4909-a1e9-a685d1d9a2c7\") " pod="openshift-ingress/e2e-ipp-a-53af08-openshift-default-556ff7748f-4nqv7"
Oct 31 01:36:47.402198 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:36:47.402050    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/778c73f2-648f-4909-a1e9-a685d1d9a2c7-istiod-ca-cert\") pod \"e2e-ipp-a-53af08-openshift-default-556ff7748f-4nqv7\" (UID: \"778c73f2-648f-4909-a1e9-a685d1d9a2c7\") " pod="openshift-ingress/e2e-ipp-a-53af08-openshift-default-556ff7748f-4nqv7"
Oct 31 01:36:47.402198 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:36:47.402154    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/778c73f2-648f-4909-a1e9-a685d1d9a2c7-workload-certs\") pod \"e2e-ipp-a-53af08-openshift-default-556ff7748f-4nqv7\" (UID: \"778c73f2-648f-4909-a1e9-a685d1d9a2c7\") " pod="openshift-ingress/e2e-ipp-a-53af08-openshift-default-556ff7748f-4nqv7"
Oct 31 01:36:47.402198 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:36:47.402174    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/778c73f2-648f-4909-a1e9-a685d1d9a2c7-credential-socket\") pod \"e2e-ipp-a-53af08-openshift-default-556ff7748f-4nqv7\" (UID: \"778c73f2-648f-4909-a1e9-a685d1d9a2c7\") " pod="openshift-ingress/e2e-ipp-a-53af08-openshift-default-556ff7748f-4nqv7"
Oct 31 01:36:47.402655 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:36:47.402625    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/778c73f2-648f-4909-a1e9-a685d1d9a2c7-istio-data\") pod \"e2e-ipp-a-53af08-openshift-default-556ff7748f-4nqv7\" (UID: \"778c73f2-648f-4909-a1e9-a685d1d9a2c7\") " pod="openshift-ingress/e2e-ipp-a-53af08-openshift-default-556ff7748f-4nqv7"
Oct 31 01:36:47.402794 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:36:47.402673    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/778c73f2-648f-4909-a1e9-a685d1d9a2c7-credential-socket\") pod \"e2e-ipp-a-53af08-openshift-default-556ff7748f-4nqv7\" (UID: \"778c73f2-648f-4909-a1e9-a685d1d9a2c7\") " pod="openshift-ingress/e2e-ipp-a-53af08-openshift-default-556ff7748f-4nqv7"
Oct 31 01:36:47.402794 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:36:47.402731    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/778c73f2-648f-4909-a1e9-a685d1d9a2c7-workload-socket\") pod \"e2e-ipp-a-53af08-openshift-default-556ff7748f-4nqv7\" (UID: \"778c73f2-648f-4909-a1e9-a685d1d9a2c7\") " pod="openshift-ingress/e2e-ipp-a-53af08-openshift-default-556ff7748f-4nqv7"
Oct 31 01:36:47.402794 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:36:47.402746    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/778c73f2-648f-4909-a1e9-a685d1d9a2c7-istiod-ca-cert\") pod \"e2e-ipp-a-53af08-openshift-default-556ff7748f-4nqv7\" (UID: \"778c73f2-648f-4909-a1e9-a685d1d9a2c7\") " pod="openshift-ingress/e2e-ipp-a-53af08-openshift-default-556ff7748f-4nqv7"
Oct 31 01:36:47.404699 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:36:47.404675    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/778c73f2-648f-4909-a1e9-a685d1d9a2c7-istio-envoy\") pod \"e2e-ipp-a-53af08-openshift-default-556ff7748f-4nqv7\" (UID: \"778c73f2-648f-4909-a1e9-a685d1d9a2c7\") " pod="openshift-ingress/e2e-ipp-a-53af08-openshift-default-556ff7748f-4nqv7"
Oct 31 01:36:47.405463 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:36:47.405438    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/778c73f2-648f-4909-a1e9-a685d1d9a2c7-istio-podinfo\") pod \"e2e-ipp-a-53af08-openshift-default-556ff7748f-4nqv7\" (UID: \"778c73f2-648f-4909-a1e9-a685d1d9a2c7\") " pod="openshift-ingress/e2e-ipp-a-53af08-openshift-default-556ff7748f-4nqv7"
Oct 31 01:36:47.460242 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:36:47.460163    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/778c73f2-648f-4909-a1e9-a685d1d9a2c7-istio-token\") pod \"e2e-ipp-a-53af08-openshift-default-556ff7748f-4nqv7\" (UID: \"778c73f2-648f-4909-a1e9-a685d1d9a2c7\") " pod="openshift-ingress/e2e-ipp-a-53af08-openshift-default-556ff7748f-4nqv7"
Oct 31 01:36:47.460436 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:36:47.460241    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7pw4z\" (UniqueName: \"kubernetes.io/projected/778c73f2-648f-4909-a1e9-a685d1d9a2c7-kube-api-access-7pw4z\") pod \"e2e-ipp-a-53af08-openshift-default-556ff7748f-4nqv7\" (UID: \"778c73f2-648f-4909-a1e9-a685d1d9a2c7\") " pod="openshift-ingress/e2e-ipp-a-53af08-openshift-default-556ff7748f-4nqv7"
Oct 31 01:36:47.505442 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:36:47.505406    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-53af08-openshift-default-556ff7748f-4nqv7"
Oct 31 01:36:47.545639 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:36:47.545601    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:36:47.939328 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:36:47.939280    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-53af08-openshift-default-556ff7748f-4nqv7"]
Oct 31 01:36:47.940187 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:36:47.940162    4964 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod778c73f2_648f_4909_a1e9_a685d1d9a2c7.slice/crio-e5e289dbedca40981ad06b1b5f235d7be84f51451f66b56d5dc1b50e1ad7b4cc WatchSource:0}: Error finding container e5e289dbedca40981ad06b1b5f235d7be84f51451f66b56d5dc1b50e1ad7b4cc: Status 404 returned error can't find the container with id e5e289dbedca40981ad06b1b5f235d7be84f51451f66b56d5dc1b50e1ad7b4cc
Oct 31 01:36:47.942841 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:36:47.942804    4964 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 01:36:47.942902 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:36:47.942868    4964 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 01:36:47.942960 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:36:47.942900    4964 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 01:36:47.963583 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:36:47.963554    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-53af08-openshift-default-556ff7748f-4nqv7" event={"ID":"778c73f2-648f-4909-a1e9-a685d1d9a2c7","Type":"ContainerStarted","Data":"e5e289dbedca40981ad06b1b5f235d7be84f51451f66b56d5dc1b50e1ad7b4cc"}
Oct 31 01:36:48.564436 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:36:48.564392    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:36:48.972646 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:36:48.972612    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-53af08-openshift-default-556ff7748f-4nqv7" event={"ID":"778c73f2-648f-4909-a1e9-a685d1d9a2c7","Type":"ContainerStarted","Data":"537884ca81ca127b04c754bf17d7fe77bcbae562cc0dcd4adf9033d91990d433"}
Oct 31 01:36:49.191512 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:36:49.191464    4964 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-53af08-openshift-default-556ff7748f-4nqv7" podStartSLOduration=2.191448968 podStartE2EDuration="2.191448968s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:36:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:36:49.139953711 +0000 UTC m=+1977.616520090" watchObservedRunningTime="2026-10-31 01:36:49.191448968 +0000 UTC m=+1977.668015347"
Oct 31 01:36:49.506150 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:36:49.506114    4964 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-53af08-openshift-default-556ff7748f-4nqv7"
Oct 31 01:36:49.512807 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:36:49.512777    4964 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-53af08-openshift-default-556ff7748f-4nqv7"
Oct 31 01:36:49.979237 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:36:49.979210    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-53af08-openshift-default-556ff7748f-4nqv7"
Oct 31 01:36:49.980193 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:36:49.980175    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-53af08-openshift-default-556ff7748f-4nqv7"
Oct 31 01:36:50.917987 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:36:50.917956    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-99c8bd-54b85b659f-9kck6"
Oct 31 01:36:50.918147 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:36:50.918029    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-99c8bd-77f6bdf9dc-tqmfm"
Oct 31 01:36:52.053704 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:36:52.053668    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:36:52.253771 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:36:52.253738    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:36:53.738601 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:36:53.738564    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:36:55.275456 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:36:55.275395    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:36:55.445600 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:36:55.445547    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:36:58.837096 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:36:58.837054    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:37:01.044869 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:37:01.044830    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:37:02.701110 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:37:02.701075    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:37:03.197158 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:37:03.197120    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:37:04.252365 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:37:04.247189    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:37:04.429796 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:37:04.429767    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:37:04.664167 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:37:04.664087    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:37:05.818724 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:37:05.818688    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:37:06.898162 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:37:06.898124    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:37:09.701532 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:37:09.701499    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:37:10.624060 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:37:10.624025    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:37:11.237879 ip-10-0-133-197 kubenswrapper[4964]: E1031 01:37:11.237836    4964 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q56x5" containerName="extractor"
Oct 31 01:37:12.015001 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:37:12.014961    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:37:15.697109 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:37:15.697072    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:37:20.593012 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:37:20.592974    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:37:25.404630 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:37:25.404596    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:37:25.905906 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:37:25.905851    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:37:26.532762 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:37:26.532709    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:37:26.977577 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:37:26.977536    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:37:29.198229 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:37:29.198189    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:37:30.207202 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:37:30.207128    4964 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-99c8bd-5dfcd8b544-q74wc"]
Oct 31 01:37:30.212618 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:37:30.212598    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-99c8bd-5dfcd8b544-q74wc"
Oct 31 01:37:30.257243 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:37:30.257215    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c1bf5a83-ee88-42cd-8d94-9502ee38bfab-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-99c8bd-5dfcd8b544-q74wc\" (UID: \"c1bf5a83-ee88-42cd-8d94-9502ee38bfab\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-99c8bd-5dfcd8b544-q74wc"
Oct 31 01:37:30.257399 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:37:30.257373    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c1bf5a83-ee88-42cd-8d94-9502ee38bfab-maas-api-tls\") pod \"maas-api-e2e-ipp-a-99c8bd-5dfcd8b544-q74wc\" (UID: \"c1bf5a83-ee88-42cd-8d94-9502ee38bfab\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-99c8bd-5dfcd8b544-q74wc"
Oct 31 01:37:30.257458 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:37:30.257438    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ps8kg\" (UniqueName: \"kubernetes.io/projected/c1bf5a83-ee88-42cd-8d94-9502ee38bfab-kube-api-access-ps8kg\") pod \"maas-api-e2e-ipp-a-99c8bd-5dfcd8b544-q74wc\" (UID: \"c1bf5a83-ee88-42cd-8d94-9502ee38bfab\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-99c8bd-5dfcd8b544-q74wc"
Oct 31 01:37:30.257559 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:37:30.257542    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c1bf5a83-ee88-42cd-8d94-9502ee38bfab-metrics-certs\") pod \"maas-api-e2e-ipp-a-99c8bd-5dfcd8b544-q74wc\" (UID: \"c1bf5a83-ee88-42cd-8d94-9502ee38bfab\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-99c8bd-5dfcd8b544-q74wc"
Oct 31 01:37:30.268921 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:37:30.268900    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-99c8bd-5dfcd8b544-q74wc"]
Oct 31 01:37:30.313432 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:37:30.313413    4964 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-99c8bd-serving-cert\""
Oct 31 01:37:30.358973 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:37:30.358944    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c1bf5a83-ee88-42cd-8d94-9502ee38bfab-metrics-certs\") pod \"maas-api-e2e-ipp-a-99c8bd-5dfcd8b544-q74wc\" (UID: \"c1bf5a83-ee88-42cd-8d94-9502ee38bfab\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-99c8bd-5dfcd8b544-q74wc"
Oct 31 01:37:30.359115 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:37:30.359084    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c1bf5a83-ee88-42cd-8d94-9502ee38bfab-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-99c8bd-5dfcd8b544-q74wc\" (UID: \"c1bf5a83-ee88-42cd-8d94-9502ee38bfab\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-99c8bd-5dfcd8b544-q74wc"
Oct 31 01:37:30.359182 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:37:30.359172    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c1bf5a83-ee88-42cd-8d94-9502ee38bfab-maas-api-tls\") pod \"maas-api-e2e-ipp-a-99c8bd-5dfcd8b544-q74wc\" (UID: \"c1bf5a83-ee88-42cd-8d94-9502ee38bfab\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-99c8bd-5dfcd8b544-q74wc"
Oct 31 01:37:30.359248 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:37:30.359230    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ps8kg\" (UniqueName: \"kubernetes.io/projected/c1bf5a83-ee88-42cd-8d94-9502ee38bfab-kube-api-access-ps8kg\") pod \"maas-api-e2e-ipp-a-99c8bd-5dfcd8b544-q74wc\" (UID: \"c1bf5a83-ee88-42cd-8d94-9502ee38bfab\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-99c8bd-5dfcd8b544-q74wc"
Oct 31 01:37:30.359551 ip-10-0-133-197 kubenswrapper[4964]: E1031 01:37:30.359529    4964 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-a-99c8bd-serving-cert: secret "maas-api-e2e-ipp-a-99c8bd-serving-cert" not found
Oct 31 01:37:30.359840 ip-10-0-133-197 kubenswrapper[4964]: E1031 01:37:30.359816    4964 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c1bf5a83-ee88-42cd-8d94-9502ee38bfab-maas-api-tls podName:c1bf5a83-ee88-42cd-8d94-9502ee38bfab nodeName:}" failed. No retries permitted until 2026-10-31 01:37:30.859795135 +0000 UTC m=+2019.336361493 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/c1bf5a83-ee88-42cd-8d94-9502ee38bfab-maas-api-tls") pod "maas-api-e2e-ipp-a-99c8bd-5dfcd8b544-q74wc" (UID: "c1bf5a83-ee88-42cd-8d94-9502ee38bfab") : secret "maas-api-e2e-ipp-a-99c8bd-serving-cert" not found
Oct 31 01:37:30.359952 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:37:30.359890    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c1bf5a83-ee88-42cd-8d94-9502ee38bfab-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-99c8bd-5dfcd8b544-q74wc\" (UID: \"c1bf5a83-ee88-42cd-8d94-9502ee38bfab\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-99c8bd-5dfcd8b544-q74wc"
Oct 31 01:37:30.361972 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:37:30.361953    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c1bf5a83-ee88-42cd-8d94-9502ee38bfab-metrics-certs\") pod \"maas-api-e2e-ipp-a-99c8bd-5dfcd8b544-q74wc\" (UID: \"c1bf5a83-ee88-42cd-8d94-9502ee38bfab\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-99c8bd-5dfcd8b544-q74wc"
Oct 31 01:37:30.416178 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:37:30.416156    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ps8kg\" (UniqueName: \"kubernetes.io/projected/c1bf5a83-ee88-42cd-8d94-9502ee38bfab-kube-api-access-ps8kg\") pod \"maas-api-e2e-ipp-a-99c8bd-5dfcd8b544-q74wc\" (UID: \"c1bf5a83-ee88-42cd-8d94-9502ee38bfab\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-99c8bd-5dfcd8b544-q74wc"
Oct 31 01:37:30.440548 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:37:30.440510    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:37:30.659149 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:37:30.659118    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:37:30.836128 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:37:30.836088    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:37:30.876331 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:37:30.876287    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c1bf5a83-ee88-42cd-8d94-9502ee38bfab-maas-api-tls\") pod \"maas-api-e2e-ipp-a-99c8bd-5dfcd8b544-q74wc\" (UID: \"c1bf5a83-ee88-42cd-8d94-9502ee38bfab\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-99c8bd-5dfcd8b544-q74wc"
Oct 31 01:37:30.878829 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:37:30.878806    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c1bf5a83-ee88-42cd-8d94-9502ee38bfab-maas-api-tls\") pod \"maas-api-e2e-ipp-a-99c8bd-5dfcd8b544-q74wc\" (UID: \"c1bf5a83-ee88-42cd-8d94-9502ee38bfab\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-99c8bd-5dfcd8b544-q74wc"
Oct 31 01:37:31.126189 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:37:31.126163    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-99c8bd-5dfcd8b544-q74wc"
Oct 31 01:37:31.162286 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:37:31.162258    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:37:31.391640 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:37:31.391560    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:37:31.506131 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:37:31.506106    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-99c8bd-5dfcd8b544-q74wc"]
Oct 31 01:37:31.508829 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:37:31.508795    4964 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1bf5a83_ee88_42cd_8d94_9502ee38bfab.slice/crio-6506e545a2415669dad20c3a67dd16524f83cbefbd6063da238ae74d24d850e3 WatchSource:0}: Error finding container 6506e545a2415669dad20c3a67dd16524f83cbefbd6063da238ae74d24d850e3: Status 404 returned error can't find the container with id 6506e545a2415669dad20c3a67dd16524f83cbefbd6063da238ae74d24d850e3
Oct 31 01:37:31.777962 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:37:31.777924    4964 prober.go:122] "Probe failed" probeType="Liveness" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-197.us-west-1.compute.internal" podUID="8dea7006d41de627c8e06bf18f518df6" containerName="haproxy" probeResult="failure" output="Get \"https://172.20.0.1:6443/version\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:37:32.271866 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:37:32.271837    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-99c8bd-5dfcd8b544-q74wc" event={"ID":"c1bf5a83-ee88-42cd-8d94-9502ee38bfab","Type":"ContainerStarted","Data":"6506e545a2415669dad20c3a67dd16524f83cbefbd6063da238ae74d24d850e3"}
Oct 31 01:37:33.280177 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:37:33.280135    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-99c8bd-5dfcd8b544-q74wc" event={"ID":"c1bf5a83-ee88-42cd-8d94-9502ee38bfab","Type":"ContainerStarted","Data":"3ad371415279d453f03782c26ec2e6dd497b1581ca74afa9164f78c47f83a80d"}
Oct 31 01:37:33.280595 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:37:33.280198    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-99c8bd-5dfcd8b544-q74wc"
Oct 31 01:37:33.416456 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:37:33.416402    4964 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-99c8bd-5dfcd8b544-q74wc" podStartSLOduration=2.508275623 podStartE2EDuration="3.416386809s" totalImagesPullingTime="908.111186ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:37:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:37:33.412813614 +0000 UTC m=+2021.889379994" watchObservedRunningTime="2026-10-31 01:37:33.416386809 +0000 UTC m=+2021.892953243"
Oct 31 01:37:33.502090 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:37:33.502059    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:37:34.386782 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:37:34.386745    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:37:34.572019 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:37:34.571980    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:37:36.000798 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:37:36.000763    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:37:39.295210 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:37:39.295181    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-99c8bd-5dfcd8b544-q74wc"
Oct 31 01:37:40.870406 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:37:40.870372    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:37:41.237951 ip-10-0-133-197 kubenswrapper[4964]: E1031 01:37:41.237915    4964 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q56x5" containerName="extractor"
Oct 31 01:37:45.614090 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:37:45.614058    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:37:45.817968 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:37:45.817932    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:37:46.012004 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:37:46.011971    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:37:50.120068 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:37:50.120036    4964 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-53af08-8495bd4fc6-dxcsr"]
Oct 31 01:37:50.124363 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:37:50.124344    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-53af08-8495bd4fc6-dxcsr"
Oct 31 01:37:50.177231 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:37:50.177201    4964 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-53af08-serving-cert\""
Oct 31 01:37:50.181902 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:37:50.181877    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-53af08-8495bd4fc6-dxcsr"]
Oct 31 01:37:50.216909 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:37:50.216883    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ceae1723-877f-4b19-9574-da266b18535f-metrics-certs\") pod \"maas-api-e2e-ipp-a-53af08-8495bd4fc6-dxcsr\" (UID: \"ceae1723-877f-4b19-9574-da266b18535f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-53af08-8495bd4fc6-dxcsr"
Oct 31 01:37:50.217105 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:37:50.217069    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8j4hx\" (UniqueName: \"kubernetes.io/projected/ceae1723-877f-4b19-9574-da266b18535f-kube-api-access-8j4hx\") pod \"maas-api-e2e-ipp-a-53af08-8495bd4fc6-dxcsr\" (UID: \"ceae1723-877f-4b19-9574-da266b18535f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-53af08-8495bd4fc6-dxcsr"
Oct 31 01:37:50.217196 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:37:50.217108    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ceae1723-877f-4b19-9574-da266b18535f-maas-api-tls\") pod \"maas-api-e2e-ipp-a-53af08-8495bd4fc6-dxcsr\" (UID: \"ceae1723-877f-4b19-9574-da266b18535f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-53af08-8495bd4fc6-dxcsr"
Oct 31 01:37:50.217269 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:37:50.217244    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ceae1723-877f-4b19-9574-da266b18535f-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-53af08-8495bd4fc6-dxcsr\" (UID: \"ceae1723-877f-4b19-9574-da266b18535f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-53af08-8495bd4fc6-dxcsr"
Oct 31 01:37:50.319122 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:37:50.319082    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ceae1723-877f-4b19-9574-da266b18535f-metrics-certs\") pod \"maas-api-e2e-ipp-a-53af08-8495bd4fc6-dxcsr\" (UID: \"ceae1723-877f-4b19-9574-da266b18535f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-53af08-8495bd4fc6-dxcsr"
Oct 31 01:37:50.319281 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:37:50.319253    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8j4hx\" (UniqueName: \"kubernetes.io/projected/ceae1723-877f-4b19-9574-da266b18535f-kube-api-access-8j4hx\") pod \"maas-api-e2e-ipp-a-53af08-8495bd4fc6-dxcsr\" (UID: \"ceae1723-877f-4b19-9574-da266b18535f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-53af08-8495bd4fc6-dxcsr"
Oct 31 01:37:50.319365 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:37:50.319282    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ceae1723-877f-4b19-9574-da266b18535f-maas-api-tls\") pod \"maas-api-e2e-ipp-a-53af08-8495bd4fc6-dxcsr\" (UID: \"ceae1723-877f-4b19-9574-da266b18535f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-53af08-8495bd4fc6-dxcsr"
Oct 31 01:37:50.319415 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:37:50.319381    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ceae1723-877f-4b19-9574-da266b18535f-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-53af08-8495bd4fc6-dxcsr\" (UID: \"ceae1723-877f-4b19-9574-da266b18535f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-53af08-8495bd4fc6-dxcsr"
Oct 31 01:37:50.319523 ip-10-0-133-197 kubenswrapper[4964]: E1031 01:37:50.319502    4964 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-a-53af08-serving-cert: secret "maas-api-e2e-ipp-a-53af08-serving-cert" not found
Oct 31 01:37:50.319583 ip-10-0-133-197 kubenswrapper[4964]: E1031 01:37:50.319572    4964 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ceae1723-877f-4b19-9574-da266b18535f-maas-api-tls podName:ceae1723-877f-4b19-9574-da266b18535f nodeName:}" failed. No retries permitted until 2026-10-31 01:37:50.819553281 +0000 UTC m=+2039.296119638 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/ceae1723-877f-4b19-9574-da266b18535f-maas-api-tls") pod "maas-api-e2e-ipp-a-53af08-8495bd4fc6-dxcsr" (UID: "ceae1723-877f-4b19-9574-da266b18535f") : secret "maas-api-e2e-ipp-a-53af08-serving-cert" not found
Oct 31 01:37:50.320091 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:37:50.320071    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ceae1723-877f-4b19-9574-da266b18535f-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-53af08-8495bd4fc6-dxcsr\" (UID: \"ceae1723-877f-4b19-9574-da266b18535f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-53af08-8495bd4fc6-dxcsr"
Oct 31 01:37:50.321686 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:37:50.321667    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ceae1723-877f-4b19-9574-da266b18535f-metrics-certs\") pod \"maas-api-e2e-ipp-a-53af08-8495bd4fc6-dxcsr\" (UID: \"ceae1723-877f-4b19-9574-da266b18535f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-53af08-8495bd4fc6-dxcsr"
Oct 31 01:37:50.353796 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:37:50.353769    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:37:50.379551 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:37:50.379468    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8j4hx\" (UniqueName: \"kubernetes.io/projected/ceae1723-877f-4b19-9574-da266b18535f-kube-api-access-8j4hx\") pod \"maas-api-e2e-ipp-a-53af08-8495bd4fc6-dxcsr\" (UID: \"ceae1723-877f-4b19-9574-da266b18535f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-53af08-8495bd4fc6-dxcsr"
Oct 31 01:37:50.588542 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:37:50.588493    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:37:50.767957 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:37:50.767925    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:37:50.828653 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:37:50.828627    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ceae1723-877f-4b19-9574-da266b18535f-maas-api-tls\") pod \"maas-api-e2e-ipp-a-53af08-8495bd4fc6-dxcsr\" (UID: \"ceae1723-877f-4b19-9574-da266b18535f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-53af08-8495bd4fc6-dxcsr"
Oct 31 01:37:50.831312 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:37:50.831285    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ceae1723-877f-4b19-9574-da266b18535f-maas-api-tls\") pod \"maas-api-e2e-ipp-a-53af08-8495bd4fc6-dxcsr\" (UID: \"ceae1723-877f-4b19-9574-da266b18535f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-53af08-8495bd4fc6-dxcsr"
Oct 31 01:37:51.038612 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:37:51.038536    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-53af08-8495bd4fc6-dxcsr"
Oct 31 01:37:51.420116 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:37:51.420084    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-53af08-8495bd4fc6-dxcsr"]
Oct 31 01:37:51.420701 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:37:51.420665    4964 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podceae1723_877f_4b19_9574_da266b18535f.slice/crio-0fa9d9c768e8a9d373c248bb8314634000679b10329d1dafc0058a1a9ce87306 WatchSource:0}: Error finding container 0fa9d9c768e8a9d373c248bb8314634000679b10329d1dafc0058a1a9ce87306: Status 404 returned error can't find the container with id 0fa9d9c768e8a9d373c248bb8314634000679b10329d1dafc0058a1a9ce87306
Oct 31 01:37:52.420859 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:37:52.420821    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-53af08-8495bd4fc6-dxcsr" event={"ID":"ceae1723-877f-4b19-9574-da266b18535f","Type":"ContainerStarted","Data":"f91a86d76b374a8b023b07afd8539bdb8b71d26715e16cc4654291328e50e2ea"}
Oct 31 01:37:52.421257 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:37:52.420883    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-53af08-8495bd4fc6-dxcsr" event={"ID":"ceae1723-877f-4b19-9574-da266b18535f","Type":"ContainerStarted","Data":"0fa9d9c768e8a9d373c248bb8314634000679b10329d1dafc0058a1a9ce87306"}
Oct 31 01:37:52.421257 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:37:52.420972    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-53af08-8495bd4fc6-dxcsr"
Oct 31 01:37:52.569715 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:37:52.569623    4964 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-53af08-8495bd4fc6-dxcsr" podStartSLOduration=1.714605643 podStartE2EDuration="2.569609821s" totalImagesPullingTime="855.004178ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:37:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:37:52.568137866 +0000 UTC m=+2041.044704245" watchObservedRunningTime="2026-10-31 01:37:52.569609821 +0000 UTC m=+2041.046176201"
Oct 31 01:37:55.575765 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:37:55.575725    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:37:58.428858 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:37:58.428817    4964 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-53af08-8495bd4fc6-dxcsr" podUID="ceae1723-877f-4b19-9574-da266b18535f" containerName="maas-api" probeResult="failure" output="Get \"https://10.132.0.132:8443/health\": dial tcp 10.132.0.132:8443: connect: connection refused"
Oct 31 01:37:59.151910 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:37:59.151873    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:37:59.634094 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:37:59.634061    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:38:01.412014 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:38:01.411967    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:38:02.411302 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:38:02.411266    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:38:02.488399 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:38:02.488365    4964 generic.go:356] "Generic (PLEG): container finished" podID="ceae1723-877f-4b19-9574-da266b18535f" containerID="f91a86d76b374a8b023b07afd8539bdb8b71d26715e16cc4654291328e50e2ea" exitCode=1
Oct 31 01:38:02.488794 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:38:02.488442    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-53af08-8495bd4fc6-dxcsr" event={"ID":"ceae1723-877f-4b19-9574-da266b18535f","Type":"ContainerDied","Data":"f91a86d76b374a8b023b07afd8539bdb8b71d26715e16cc4654291328e50e2ea"}
Oct 31 01:38:02.488902 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:38:02.488843    4964 scope.go:122] "RemoveContainer" containerID="f91a86d76b374a8b023b07afd8539bdb8b71d26715e16cc4654291328e50e2ea"
Oct 31 01:38:03.498745 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:38:03.498707    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-53af08-8495bd4fc6-dxcsr" event={"ID":"ceae1723-877f-4b19-9574-da266b18535f","Type":"ContainerStarted","Data":"7aef7c90ec66b5af747f3858074459e5a844db36139ba9c675c2cce7af7d6376"}
Oct 31 01:38:03.499215 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:38:03.499080    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-53af08-8495bd4fc6-dxcsr"
Oct 31 01:38:08.906600 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:38:08.906562    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:38:09.511621 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:38:09.511583    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-53af08-8495bd4fc6-dxcsr"
Oct 31 01:38:10.039710 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:38:10.039674    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:38:11.238642 ip-10-0-133-197 kubenswrapper[4964]: E1031 01:38:11.238600    4964 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q56x5" containerName="extractor"
Oct 31 01:38:12.348227 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:38:12.348186    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:38:13.712449 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:38:13.712416    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:38:18.517240 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:38:18.517201    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:38:23.317411 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:38:23.317295    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:38:25.410334 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:38:25.410292    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:38:27.454610 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:38:27.454567    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:38:28.088146 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:38:28.088110    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:38:29.807536 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:38:29.807502    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:38:31.346601 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:38:31.346562    4964 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-wjg8p"]
Oct 31 01:38:31.347075 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:38:31.346817    4964 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-wjg8p" podUID="8bdf16a9-47d4-429a-bd0c-d031817ff07b" containerName="istio-proxy" containerID="cri-o://c9bb7002cb60b09b72f1b4715f5375e8ff9de7e4da36c295f89be4a6edc5857a" gracePeriod=30
Oct 31 01:38:32.794908 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:38:32.794874    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:38:32.990488 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:38:32.990438    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:38:33.162173 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:38:33.162080    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:38:34.507842 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:38:34.507801    4964 patch_prober.go:29] interesting pod/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-wjg8p container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 01:38:34.508299 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:38:34.507865    4964 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-wjg8p" podUID="8bdf16a9-47d4-429a-bd0c-d031817ff07b" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 01:38:36.624066 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:38:36.623999    4964 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-wjg8p"
Oct 31 01:38:36.711378 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:38:36.711350    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8bdf16a9-47d4-429a-bd0c-d031817ff07b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8bdf16a9-47d4-429a-bd0c-d031817ff07b-istio-envoy\") pod \"8bdf16a9-47d4-429a-bd0c-d031817ff07b\" (UID: \"8bdf16a9-47d4-429a-bd0c-d031817ff07b\") "
Oct 31 01:38:36.711553 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:38:36.711448    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8bdf16a9-47d4-429a-bd0c-d031817ff07b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8bdf16a9-47d4-429a-bd0c-d031817ff07b-credential-socket\") pod \"8bdf16a9-47d4-429a-bd0c-d031817ff07b\" (UID: \"8bdf16a9-47d4-429a-bd0c-d031817ff07b\") "
Oct 31 01:38:36.711553 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:38:36.711491    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8bdf16a9-47d4-429a-bd0c-d031817ff07b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8bdf16a9-47d4-429a-bd0c-d031817ff07b-istiod-ca-cert\") pod \"8bdf16a9-47d4-429a-bd0c-d031817ff07b\" (UID: \"8bdf16a9-47d4-429a-bd0c-d031817ff07b\") "
Oct 31 01:38:36.711553 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:38:36.711525    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8bdf16a9-47d4-429a-bd0c-d031817ff07b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8bdf16a9-47d4-429a-bd0c-d031817ff07b-istio-data\") pod \"8bdf16a9-47d4-429a-bd0c-d031817ff07b\" (UID: \"8bdf16a9-47d4-429a-bd0c-d031817ff07b\") "
Oct 31 01:38:36.711725 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:38:36.711569    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8bdf16a9-47d4-429a-bd0c-d031817ff07b-kube-api-access-8n2l2\" (UniqueName: \"kubernetes.io/projected/8bdf16a9-47d4-429a-bd0c-d031817ff07b-kube-api-access-8n2l2\") pod \"8bdf16a9-47d4-429a-bd0c-d031817ff07b\" (UID: \"8bdf16a9-47d4-429a-bd0c-d031817ff07b\") "
Oct 31 01:38:36.711725 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:38:36.711627    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8bdf16a9-47d4-429a-bd0c-d031817ff07b-istio-token\" (UniqueName: \"kubernetes.io/projected/8bdf16a9-47d4-429a-bd0c-d031817ff07b-istio-token\") pod \"8bdf16a9-47d4-429a-bd0c-d031817ff07b\" (UID: \"8bdf16a9-47d4-429a-bd0c-d031817ff07b\") "
Oct 31 01:38:36.711725 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:38:36.711668    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8bdf16a9-47d4-429a-bd0c-d031817ff07b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8bdf16a9-47d4-429a-bd0c-d031817ff07b-workload-certs\") pod \"8bdf16a9-47d4-429a-bd0c-d031817ff07b\" (UID: \"8bdf16a9-47d4-429a-bd0c-d031817ff07b\") "
Oct 31 01:38:36.711725 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:38:36.711702    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8bdf16a9-47d4-429a-bd0c-d031817ff07b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8bdf16a9-47d4-429a-bd0c-d031817ff07b-istio-podinfo\") pod \"8bdf16a9-47d4-429a-bd0c-d031817ff07b\" (UID: \"8bdf16a9-47d4-429a-bd0c-d031817ff07b\") "
Oct 31 01:38:36.711917 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:38:36.711724    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8bdf16a9-47d4-429a-bd0c-d031817ff07b-credential-socket" pod "8bdf16a9-47d4-429a-bd0c-d031817ff07b" (UID: "8bdf16a9-47d4-429a-bd0c-d031817ff07b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:38:36.711917 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:38:36.711784    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8bdf16a9-47d4-429a-bd0c-d031817ff07b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8bdf16a9-47d4-429a-bd0c-d031817ff07b-workload-socket\") pod \"8bdf16a9-47d4-429a-bd0c-d031817ff07b\" (UID: \"8bdf16a9-47d4-429a-bd0c-d031817ff07b\") "
Oct 31 01:38:36.711917 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:38:36.711838    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8bdf16a9-47d4-429a-bd0c-d031817ff07b-istio-data" pod "8bdf16a9-47d4-429a-bd0c-d031817ff07b" (UID: "8bdf16a9-47d4-429a-bd0c-d031817ff07b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:38:36.712218 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:38:36.712197    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8bdf16a9-47d4-429a-bd0c-d031817ff07b-workload-certs" pod "8bdf16a9-47d4-429a-bd0c-d031817ff07b" (UID: "8bdf16a9-47d4-429a-bd0c-d031817ff07b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:38:36.712355 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:38:36.712289    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8bdf16a9-47d4-429a-bd0c-d031817ff07b-workload-socket" pod "8bdf16a9-47d4-429a-bd0c-d031817ff07b" (UID: "8bdf16a9-47d4-429a-bd0c-d031817ff07b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:38:36.712442 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:38:36.712424    4964 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8bdf16a9-47d4-429a-bd0c-d031817ff07b-credential-socket\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:38:36.712485 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:38:36.712449    4964 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8bdf16a9-47d4-429a-bd0c-d031817ff07b-istio-data\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:38:36.712485 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:38:36.712465    4964 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8bdf16a9-47d4-429a-bd0c-d031817ff07b-workload-certs\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:38:36.712691 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:38:36.712666    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8bdf16a9-47d4-429a-bd0c-d031817ff07b-istiod-ca-cert" pod "8bdf16a9-47d4-429a-bd0c-d031817ff07b" (UID: "8bdf16a9-47d4-429a-bd0c-d031817ff07b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:38:36.714261 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:38:36.714237    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8bdf16a9-47d4-429a-bd0c-d031817ff07b-istio-token" pod "8bdf16a9-47d4-429a-bd0c-d031817ff07b" (UID: "8bdf16a9-47d4-429a-bd0c-d031817ff07b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:38:36.714542 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:38:36.714522    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8bdf16a9-47d4-429a-bd0c-d031817ff07b-istio-envoy" pod "8bdf16a9-47d4-429a-bd0c-d031817ff07b" (UID: "8bdf16a9-47d4-429a-bd0c-d031817ff07b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:38:36.714715 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:38:36.714701    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8bdf16a9-47d4-429a-bd0c-d031817ff07b-istio-podinfo" pod "8bdf16a9-47d4-429a-bd0c-d031817ff07b" (UID: "8bdf16a9-47d4-429a-bd0c-d031817ff07b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 01:38:36.714877 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:38:36.714863    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8bdf16a9-47d4-429a-bd0c-d031817ff07b-kube-api-access-8n2l2" pod "8bdf16a9-47d4-429a-bd0c-d031817ff07b" (UID: "8bdf16a9-47d4-429a-bd0c-d031817ff07b"). InnerVolumeSpecName "kube-api-access-8n2l2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:38:36.766785 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:38:36.766751    4964 generic.go:356] "Generic (PLEG): container finished" podID="8bdf16a9-47d4-429a-bd0c-d031817ff07b" containerID="c9bb7002cb60b09b72f1b4715f5375e8ff9de7e4da36c295f89be4a6edc5857a" exitCode=0
Oct 31 01:38:36.767001 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:38:36.766798    4964 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-wjg8p"
Oct 31 01:38:36.767001 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:38:36.766828    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-wjg8p" event={"ID":"8bdf16a9-47d4-429a-bd0c-d031817ff07b","Type":"ContainerDied","Data":"c9bb7002cb60b09b72f1b4715f5375e8ff9de7e4da36c295f89be4a6edc5857a"}
Oct 31 01:38:36.767001 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:38:36.766881    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-wjg8p" event={"ID":"8bdf16a9-47d4-429a-bd0c-d031817ff07b","Type":"ContainerDied","Data":"80d6a9f4325aa9a4993a59595a0d6a55afd4d7d47bcc95d04a207fbd40f2e920"}
Oct 31 01:38:36.767001 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:38:36.766903    4964 scope.go:122] "RemoveContainer" containerID="c9bb7002cb60b09b72f1b4715f5375e8ff9de7e4da36c295f89be4a6edc5857a"
Oct 31 01:38:36.780075 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:38:36.780056    4964 scope.go:122] "RemoveContainer" containerID="c9bb7002cb60b09b72f1b4715f5375e8ff9de7e4da36c295f89be4a6edc5857a"
Oct 31 01:38:36.780417 ip-10-0-133-197 kubenswrapper[4964]: E1031 01:38:36.780391    4964 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c9bb7002cb60b09b72f1b4715f5375e8ff9de7e4da36c295f89be4a6edc5857a\": container with ID starting with c9bb7002cb60b09b72f1b4715f5375e8ff9de7e4da36c295f89be4a6edc5857a not found: ID does not exist" containerID="c9bb7002cb60b09b72f1b4715f5375e8ff9de7e4da36c295f89be4a6edc5857a"
Oct 31 01:38:36.780550 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:38:36.780423    4964 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c9bb7002cb60b09b72f1b4715f5375e8ff9de7e4da36c295f89be4a6edc5857a"} err="failed to get container status \"c9bb7002cb60b09b72f1b4715f5375e8ff9de7e4da36c295f89be4a6edc5857a\": rpc error: code = NotFound desc = could not find container \"c9bb7002cb60b09b72f1b4715f5375e8ff9de7e4da36c295f89be4a6edc5857a\": container with ID starting with c9bb7002cb60b09b72f1b4715f5375e8ff9de7e4da36c295f89be4a6edc5857a not found: ID does not exist"
Oct 31 01:38:36.813839 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:38:36.813803    4964 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8n2l2\" (UniqueName: \"kubernetes.io/projected/8bdf16a9-47d4-429a-bd0c-d031817ff07b-kube-api-access-8n2l2\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:38:36.813839 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:38:36.813839    4964 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8bdf16a9-47d4-429a-bd0c-d031817ff07b-istio-token\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:38:36.814023 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:38:36.813855    4964 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8bdf16a9-47d4-429a-bd0c-d031817ff07b-istio-podinfo\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:38:36.814023 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:38:36.813869    4964 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8bdf16a9-47d4-429a-bd0c-d031817ff07b-workload-socket\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:38:36.814023 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:38:36.813883    4964 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8bdf16a9-47d4-429a-bd0c-d031817ff07b-istio-envoy\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:38:36.814023 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:38:36.813896    4964 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8bdf16a9-47d4-429a-bd0c-d031817ff07b-istiod-ca-cert\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:38:36.903691 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:38:36.903620    4964 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-wjg8p"]
Oct 31 01:38:36.910980 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:38:36.910936    4964 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-wjg8p"]
Oct 31 01:38:37.777584 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:38:37.777552    4964 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8bdf16a9-47d4-429a-bd0c-d031817ff07b" path="/var/lib/kubelet/pods/8bdf16a9-47d4-429a-bd0c-d031817ff07b/volumes"
Oct 31 01:38:37.834207 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:38:37.834166    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:38:41.237050 ip-10-0-133-197 kubenswrapper[4964]: E1031 01:38:41.237018    4964 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q56x5" containerName="extractor"
Oct 31 01:38:42.714733 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:38:42.714696    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:38:43.144198 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:38:43.144166    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:38:43.348592 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:38:43.348548    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:38:43.540145 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:38:43.540109    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:38:46.153155 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:38:46.153125    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:38:47.801131 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:38:47.801093    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:38:48.031802 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:38:48.031764    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:38:48.214523 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:38:48.214436    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:38:48.391702 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:38:48.391637    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:38:48.762946 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:38:48.762911    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:38:49.190478 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:38:49.190394    4964 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-99c8bd-54b85b659f-9kck6"]
Oct 31 01:38:49.304607 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:38:49.304575    4964 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-99c8bd-77f6bdf9dc-tqmfm"]
Oct 31 01:38:49.440823 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:38:49.440736    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:38:49.627366 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:38:49.627331    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:38:49.821861 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:38:49.821819    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:38:52.073182 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:38:52.073156    4964 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-9257n_f0c0ada7-7688-4dc2-97fc-813e4d0c4ef3/console-operator/2.log"
Oct 31 01:38:52.081202 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:38:52.081175    4964 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-9257n_f0c0ada7-7688-4dc2-97fc-813e4d0c4ef3/console-operator/2.log"
Oct 31 01:38:52.620736 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:38:52.620703    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:38:54.190992 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:38:54.190945    4964 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-99c8bd-54b85b659f-9kck6" podUID="6ac8d799-a669-446b-950c-69fc4d4fb26b" containerName="payload-processing" containerID="cri-o://5e8ce5b77a5aac53b6b469b107586ccf7c6fed5c67b4012e36dc945f2b059ba3" gracePeriod=25
Oct 31 01:38:54.305059 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:38:54.305017    4964 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-99c8bd-77f6bdf9dc-tqmfm" podUID="7c87ec79-9698-470d-a9b2-f119731eb072" containerName="payload-pre-processing" containerID="cri-o://94cfa99d78de213db2dccd70495c199f2b70b23a0d7e1c601ef7e8060ad9e919" gracePeriod=25
Oct 31 01:38:54.507758 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:38:54.507442    4964 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-99c8bd-54b85b659f-9kck6"
Oct 31 01:38:54.578034 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:38:54.578003    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6ac8d799-a669-446b-950c-69fc4d4fb26b-kube-api-access-9tnrq\" (UniqueName: \"kubernetes.io/projected/6ac8d799-a669-446b-950c-69fc4d4fb26b-kube-api-access-9tnrq\") pod \"6ac8d799-a669-446b-950c-69fc4d4fb26b\" (UID: \"6ac8d799-a669-446b-950c-69fc4d4fb26b\") "
Oct 31 01:38:54.578189 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:38:54.578168    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6ac8d799-a669-446b-950c-69fc4d4fb26b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6ac8d799-a669-446b-950c-69fc4d4fb26b-plugins-config-volume\") pod \"6ac8d799-a669-446b-950c-69fc4d4fb26b\" (UID: \"6ac8d799-a669-446b-950c-69fc4d4fb26b\") "
Oct 31 01:38:54.578731 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:38:54.578650    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6ac8d799-a669-446b-950c-69fc4d4fb26b-plugins-config-volume" pod "6ac8d799-a669-446b-950c-69fc4d4fb26b" (UID: "6ac8d799-a669-446b-950c-69fc4d4fb26b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:38:54.580361 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:38:54.580332    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ac8d799-a669-446b-950c-69fc4d4fb26b-kube-api-access-9tnrq" pod "6ac8d799-a669-446b-950c-69fc4d4fb26b" (UID: "6ac8d799-a669-446b-950c-69fc4d4fb26b"). InnerVolumeSpecName "kube-api-access-9tnrq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:38:54.581109 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:38:54.581091    4964 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-99c8bd-77f6bdf9dc-tqmfm"
Oct 31 01:38:54.617451 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:38:54.617405    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:38:54.679060 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:38:54.679028    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7c87ec79-9698-470d-a9b2-f119731eb072-kube-api-access-zk98f\" (UniqueName: \"kubernetes.io/projected/7c87ec79-9698-470d-a9b2-f119731eb072-kube-api-access-zk98f\") pod \"7c87ec79-9698-470d-a9b2-f119731eb072\" (UID: \"7c87ec79-9698-470d-a9b2-f119731eb072\") "
Oct 31 01:38:54.679249 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:38:54.679176    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7c87ec79-9698-470d-a9b2-f119731eb072-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7c87ec79-9698-470d-a9b2-f119731eb072-plugins-config-volume\") pod \"7c87ec79-9698-470d-a9b2-f119731eb072\" (UID: \"7c87ec79-9698-470d-a9b2-f119731eb072\") "
Oct 31 01:38:54.679636 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:38:54.679614    4964 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9tnrq\" (UniqueName: \"kubernetes.io/projected/6ac8d799-a669-446b-950c-69fc4d4fb26b-kube-api-access-9tnrq\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:38:54.679893 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:38:54.679640    4964 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6ac8d799-a669-446b-950c-69fc4d4fb26b-plugins-config-volume\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:38:54.679893 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:38:54.679733    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7c87ec79-9698-470d-a9b2-f119731eb072-plugins-config-volume" pod "7c87ec79-9698-470d-a9b2-f119731eb072" (UID: "7c87ec79-9698-470d-a9b2-f119731eb072"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:38:54.681644 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:38:54.681617    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c87ec79-9698-470d-a9b2-f119731eb072-kube-api-access-zk98f" pod "7c87ec79-9698-470d-a9b2-f119731eb072" (UID: "7c87ec79-9698-470d-a9b2-f119731eb072"). InnerVolumeSpecName "kube-api-access-zk98f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:38:54.780705 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:38:54.780672    4964 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7c87ec79-9698-470d-a9b2-f119731eb072-plugins-config-volume\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:38:54.780956 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:38:54.780928    4964 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zk98f\" (UniqueName: \"kubernetes.io/projected/7c87ec79-9698-470d-a9b2-f119731eb072-kube-api-access-zk98f\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:38:54.849899 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:38:54.849848    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:38:54.907932 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:38:54.907890    4964 generic.go:356] "Generic (PLEG): container finished" podID="6ac8d799-a669-446b-950c-69fc4d4fb26b" containerID="5e8ce5b77a5aac53b6b469b107586ccf7c6fed5c67b4012e36dc945f2b059ba3" exitCode=0
Oct 31 01:38:54.908113 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:38:54.908063    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-99c8bd-54b85b659f-9kck6" event={"ID":"6ac8d799-a669-446b-950c-69fc4d4fb26b","Type":"ContainerDied","Data":"5e8ce5b77a5aac53b6b469b107586ccf7c6fed5c67b4012e36dc945f2b059ba3"}
Oct 31 01:38:54.908180 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:38:54.908156    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-99c8bd-54b85b659f-9kck6" event={"ID":"6ac8d799-a669-446b-950c-69fc4d4fb26b","Type":"ContainerDied","Data":"be57b1725a3e57cc6b78a30098c71f5d3ad4339bb4502de8706ca20b305f0a64"}
Oct 31 01:38:54.908229 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:38:54.908194    4964 scope.go:122] "RemoveContainer" containerID="5e8ce5b77a5aac53b6b469b107586ccf7c6fed5c67b4012e36dc945f2b059ba3"
Oct 31 01:38:54.908304 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:38:54.908280    4964 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-99c8bd-54b85b659f-9kck6"
Oct 31 01:38:54.910582 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:38:54.910552    4964 generic.go:356] "Generic (PLEG): container finished" podID="7c87ec79-9698-470d-a9b2-f119731eb072" containerID="94cfa99d78de213db2dccd70495c199f2b70b23a0d7e1c601ef7e8060ad9e919" exitCode=0
Oct 31 01:38:54.910698 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:38:54.910673    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-99c8bd-77f6bdf9dc-tqmfm" event={"ID":"7c87ec79-9698-470d-a9b2-f119731eb072","Type":"ContainerDied","Data":"94cfa99d78de213db2dccd70495c199f2b70b23a0d7e1c601ef7e8060ad9e919"}
Oct 31 01:38:54.910814 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:38:54.910738    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-99c8bd-77f6bdf9dc-tqmfm" event={"ID":"7c87ec79-9698-470d-a9b2-f119731eb072","Type":"ContainerDied","Data":"5f3ab02fe0ca9fdec5e949abdedf7d22bce20daabfeb7d382a78ba772176b940"}
Oct 31 01:38:54.910894 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:38:54.910844    4964 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-99c8bd-77f6bdf9dc-tqmfm"
Oct 31 01:38:54.936370 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:38:54.936348    4964 scope.go:122] "RemoveContainer" containerID="5e8ce5b77a5aac53b6b469b107586ccf7c6fed5c67b4012e36dc945f2b059ba3"
Oct 31 01:38:54.936690 ip-10-0-133-197 kubenswrapper[4964]: E1031 01:38:54.936670    4964 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5e8ce5b77a5aac53b6b469b107586ccf7c6fed5c67b4012e36dc945f2b059ba3\": container with ID starting with 5e8ce5b77a5aac53b6b469b107586ccf7c6fed5c67b4012e36dc945f2b059ba3 not found: ID does not exist" containerID="5e8ce5b77a5aac53b6b469b107586ccf7c6fed5c67b4012e36dc945f2b059ba3"
Oct 31 01:38:54.936772 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:38:54.936698    4964 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5e8ce5b77a5aac53b6b469b107586ccf7c6fed5c67b4012e36dc945f2b059ba3"} err="failed to get container status \"5e8ce5b77a5aac53b6b469b107586ccf7c6fed5c67b4012e36dc945f2b059ba3\": rpc error: code = NotFound desc = could not find container \"5e8ce5b77a5aac53b6b469b107586ccf7c6fed5c67b4012e36dc945f2b059ba3\": container with ID starting with 5e8ce5b77a5aac53b6b469b107586ccf7c6fed5c67b4012e36dc945f2b059ba3 not found: ID does not exist"
Oct 31 01:38:54.936772 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:38:54.936715    4964 scope.go:122] "RemoveContainer" containerID="94cfa99d78de213db2dccd70495c199f2b70b23a0d7e1c601ef7e8060ad9e919"
Oct 31 01:38:54.946497 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:38:54.946479    4964 scope.go:122] "RemoveContainer" containerID="94cfa99d78de213db2dccd70495c199f2b70b23a0d7e1c601ef7e8060ad9e919"
Oct 31 01:38:54.946724 ip-10-0-133-197 kubenswrapper[4964]: E1031 01:38:54.946703    4964 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"94cfa99d78de213db2dccd70495c199f2b70b23a0d7e1c601ef7e8060ad9e919\": container with ID starting with 94cfa99d78de213db2dccd70495c199f2b70b23a0d7e1c601ef7e8060ad9e919 not found: ID does not exist" containerID="94cfa99d78de213db2dccd70495c199f2b70b23a0d7e1c601ef7e8060ad9e919"
Oct 31 01:38:54.946791 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:38:54.946750    4964 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"94cfa99d78de213db2dccd70495c199f2b70b23a0d7e1c601ef7e8060ad9e919"} err="failed to get container status \"94cfa99d78de213db2dccd70495c199f2b70b23a0d7e1c601ef7e8060ad9e919\": rpc error: code = NotFound desc = could not find container \"94cfa99d78de213db2dccd70495c199f2b70b23a0d7e1c601ef7e8060ad9e919\": container with ID starting with 94cfa99d78de213db2dccd70495c199f2b70b23a0d7e1c601ef7e8060ad9e919 not found: ID does not exist"
Oct 31 01:38:55.107027 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:38:55.106952    4964 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-99c8bd-77f6bdf9dc-tqmfm"]
Oct 31 01:38:55.111987 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:38:55.111950    4964 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-99c8bd-77f6bdf9dc-tqmfm"]
Oct 31 01:38:55.223235 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:38:55.223210    4964 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-99c8bd-54b85b659f-9kck6"]
Oct 31 01:38:55.226661 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:38:55.226638    4964 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-99c8bd-54b85b659f-9kck6"]
Oct 31 01:38:55.691487 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:38:55.691423    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:38:55.780827 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:38:55.780799    4964 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6ac8d799-a669-446b-950c-69fc4d4fb26b" path="/var/lib/kubelet/pods/6ac8d799-a669-446b-950c-69fc4d4fb26b/volumes"
Oct 31 01:38:55.781233 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:38:55.781219    4964 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7c87ec79-9698-470d-a9b2-f119731eb072" path="/var/lib/kubelet/pods/7c87ec79-9698-470d-a9b2-f119731eb072/volumes"
Oct 31 01:38:55.788177 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:38:55.788155    4964 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w1-71bc98/tenantcfg-model-71bc98-kserve-7b7d7bc7bf-kqn2l"]
Oct 31 01:38:55.789074 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:38:55.789058    4964 cpu_manager.go:414] "containerMap: removing container" podUID="8bdf16a9-47d4-429a-bd0c-d031817ff07b" containerName="istio-proxy"
Oct 31 01:38:55.789138 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:38:55.789077    4964 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8bdf16a9-47d4-429a-bd0c-d031817ff07b" containerName="istio-proxy"
Oct 31 01:38:55.789186 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:38:55.789172    4964 cpu_manager.go:414] "containerMap: removing container" podUID="6ac8d799-a669-446b-950c-69fc4d4fb26b" containerName="payload-processing"
Oct 31 01:38:55.789186 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:38:55.789180    4964 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6ac8d799-a669-446b-950c-69fc4d4fb26b" containerName="payload-processing"
Oct 31 01:38:55.789253 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:38:55.789206    4964 cpu_manager.go:414] "containerMap: removing container" podUID="7c87ec79-9698-470d-a9b2-f119731eb072" containerName="payload-pre-processing"
Oct 31 01:38:55.789253 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:38:55.789214    4964 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7c87ec79-9698-470d-a9b2-f119731eb072" containerName="payload-pre-processing"
Oct 31 01:38:55.789369 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:38:55.789355    4964 memory_manager.go:365] "RemoveStaleState removing state" podUID="6ac8d799-a669-446b-950c-69fc4d4fb26b" containerName="payload-processing"
Oct 31 01:38:55.789410 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:38:55.789373    4964 memory_manager.go:365] "RemoveStaleState removing state" podUID="7c87ec79-9698-470d-a9b2-f119731eb072" containerName="payload-pre-processing"
Oct 31 01:38:55.789410 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:38:55.789385    4964 memory_manager.go:365] "RemoveStaleState removing state" podUID="8bdf16a9-47d4-429a-bd0c-d031817ff07b" containerName="istio-proxy"
Oct 31 01:38:55.794732 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:38:55.794715    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-71bc98/tenantcfg-model-71bc98-kserve-7b7d7bc7bf-kqn2l"
Oct 31 01:38:55.848881 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:38:55.848849    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w1-71bc98/tenantcfg-model-71bc98-kserve-7b7d7bc7bf-kqn2l"]
Oct 31 01:38:55.892781 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:38:55.892754    4964 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w1-71bc98\"/\"kube-root-ca.crt\""
Oct 31 01:38:55.892934 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:38:55.892837    4964 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w1-71bc98\"/\"default-dockercfg-t26rh\""
Oct 31 01:38:55.892934 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:38:55.892911    4964 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w1-71bc98\"/\"tenantcfg-model-71bc98-kserve-self-signed-certs\""
Oct 31 01:38:55.893267 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:38:55.893248    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lzdht\" (UniqueName: \"kubernetes.io/projected/847d367e-516e-4240-924c-3e62c532a8d6-kube-api-access-lzdht\") pod \"tenantcfg-model-71bc98-kserve-7b7d7bc7bf-kqn2l\" (UID: \"847d367e-516e-4240-924c-3e62c532a8d6\") " pod="ai-tenant-e2e-shared-b-w1-71bc98/tenantcfg-model-71bc98-kserve-7b7d7bc7bf-kqn2l"
Oct 31 01:38:55.893341 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:38:55.893289    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/847d367e-516e-4240-924c-3e62c532a8d6-home\") pod \"tenantcfg-model-71bc98-kserve-7b7d7bc7bf-kqn2l\" (UID: \"847d367e-516e-4240-924c-3e62c532a8d6\") " pod="ai-tenant-e2e-shared-b-w1-71bc98/tenantcfg-model-71bc98-kserve-7b7d7bc7bf-kqn2l"
Oct 31 01:38:55.893412 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:38:55.893396    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/847d367e-516e-4240-924c-3e62c532a8d6-tls-certs\") pod \"tenantcfg-model-71bc98-kserve-7b7d7bc7bf-kqn2l\" (UID: \"847d367e-516e-4240-924c-3e62c532a8d6\") " pod="ai-tenant-e2e-shared-b-w1-71bc98/tenantcfg-model-71bc98-kserve-7b7d7bc7bf-kqn2l"
Oct 31 01:38:55.893550 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:38:55.893537    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/847d367e-516e-4240-924c-3e62c532a8d6-model-cache\") pod \"tenantcfg-model-71bc98-kserve-7b7d7bc7bf-kqn2l\" (UID: \"847d367e-516e-4240-924c-3e62c532a8d6\") " pod="ai-tenant-e2e-shared-b-w1-71bc98/tenantcfg-model-71bc98-kserve-7b7d7bc7bf-kqn2l"
Oct 31 01:38:55.893645 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:38:55.893631    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/847d367e-516e-4240-924c-3e62c532a8d6-dshm\") pod \"tenantcfg-model-71bc98-kserve-7b7d7bc7bf-kqn2l\" (UID: \"847d367e-516e-4240-924c-3e62c532a8d6\") " pod="ai-tenant-e2e-shared-b-w1-71bc98/tenantcfg-model-71bc98-kserve-7b7d7bc7bf-kqn2l"
Oct 31 01:38:55.893742 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:38:55.893724    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/847d367e-516e-4240-924c-3e62c532a8d6-tmp-dir\") pod \"tenantcfg-model-71bc98-kserve-7b7d7bc7bf-kqn2l\" (UID: \"847d367e-516e-4240-924c-3e62c532a8d6\") " pod="ai-tenant-e2e-shared-b-w1-71bc98/tenantcfg-model-71bc98-kserve-7b7d7bc7bf-kqn2l"
Oct 31 01:38:55.915067 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:38:55.915034    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:38:55.947880 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:38:55.947806    4964 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w1-71bc98\"/\"openshift-service-ca.crt\""
Oct 31 01:38:55.994712 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:38:55.994672    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/847d367e-516e-4240-924c-3e62c532a8d6-tmp-dir\") pod \"tenantcfg-model-71bc98-kserve-7b7d7bc7bf-kqn2l\" (UID: \"847d367e-516e-4240-924c-3e62c532a8d6\") " pod="ai-tenant-e2e-shared-b-w1-71bc98/tenantcfg-model-71bc98-kserve-7b7d7bc7bf-kqn2l"
Oct 31 01:38:55.994900 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:38:55.994762    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lzdht\" (UniqueName: \"kubernetes.io/projected/847d367e-516e-4240-924c-3e62c532a8d6-kube-api-access-lzdht\") pod \"tenantcfg-model-71bc98-kserve-7b7d7bc7bf-kqn2l\" (UID: \"847d367e-516e-4240-924c-3e62c532a8d6\") " pod="ai-tenant-e2e-shared-b-w1-71bc98/tenantcfg-model-71bc98-kserve-7b7d7bc7bf-kqn2l"
Oct 31 01:38:55.994900 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:38:55.994798    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/847d367e-516e-4240-924c-3e62c532a8d6-home\") pod \"tenantcfg-model-71bc98-kserve-7b7d7bc7bf-kqn2l\" (UID: \"847d367e-516e-4240-924c-3e62c532a8d6\") " pod="ai-tenant-e2e-shared-b-w1-71bc98/tenantcfg-model-71bc98-kserve-7b7d7bc7bf-kqn2l"
Oct 31 01:38:55.994900 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:38:55.994828    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/847d367e-516e-4240-924c-3e62c532a8d6-tls-certs\") pod \"tenantcfg-model-71bc98-kserve-7b7d7bc7bf-kqn2l\" (UID: \"847d367e-516e-4240-924c-3e62c532a8d6\") " pod="ai-tenant-e2e-shared-b-w1-71bc98/tenantcfg-model-71bc98-kserve-7b7d7bc7bf-kqn2l"
Oct 31 01:38:55.995058 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:38:55.994940    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/847d367e-516e-4240-924c-3e62c532a8d6-model-cache\") pod \"tenantcfg-model-71bc98-kserve-7b7d7bc7bf-kqn2l\" (UID: \"847d367e-516e-4240-924c-3e62c532a8d6\") " pod="ai-tenant-e2e-shared-b-w1-71bc98/tenantcfg-model-71bc98-kserve-7b7d7bc7bf-kqn2l"
Oct 31 01:38:55.995058 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:38:55.995017    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/847d367e-516e-4240-924c-3e62c532a8d6-dshm\") pod \"tenantcfg-model-71bc98-kserve-7b7d7bc7bf-kqn2l\" (UID: \"847d367e-516e-4240-924c-3e62c532a8d6\") " pod="ai-tenant-e2e-shared-b-w1-71bc98/tenantcfg-model-71bc98-kserve-7b7d7bc7bf-kqn2l"
Oct 31 01:38:55.995179 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:38:55.995132    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/847d367e-516e-4240-924c-3e62c532a8d6-tmp-dir\") pod \"tenantcfg-model-71bc98-kserve-7b7d7bc7bf-kqn2l\" (UID: \"847d367e-516e-4240-924c-3e62c532a8d6\") " pod="ai-tenant-e2e-shared-b-w1-71bc98/tenantcfg-model-71bc98-kserve-7b7d7bc7bf-kqn2l"
Oct 31 01:38:55.995483 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:38:55.995463    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/847d367e-516e-4240-924c-3e62c532a8d6-model-cache\") pod \"tenantcfg-model-71bc98-kserve-7b7d7bc7bf-kqn2l\" (UID: \"847d367e-516e-4240-924c-3e62c532a8d6\") " pod="ai-tenant-e2e-shared-b-w1-71bc98/tenantcfg-model-71bc98-kserve-7b7d7bc7bf-kqn2l"
Oct 31 01:38:55.995589 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:38:55.995505    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/847d367e-516e-4240-924c-3e62c532a8d6-home\") pod \"tenantcfg-model-71bc98-kserve-7b7d7bc7bf-kqn2l\" (UID: \"847d367e-516e-4240-924c-3e62c532a8d6\") " pod="ai-tenant-e2e-shared-b-w1-71bc98/tenantcfg-model-71bc98-kserve-7b7d7bc7bf-kqn2l"
Oct 31 01:38:55.997687 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:38:55.997669    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/847d367e-516e-4240-924c-3e62c532a8d6-tls-certs\") pod \"tenantcfg-model-71bc98-kserve-7b7d7bc7bf-kqn2l\" (UID: \"847d367e-516e-4240-924c-3e62c532a8d6\") " pod="ai-tenant-e2e-shared-b-w1-71bc98/tenantcfg-model-71bc98-kserve-7b7d7bc7bf-kqn2l"
Oct 31 01:38:55.997841 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:38:55.997823    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/847d367e-516e-4240-924c-3e62c532a8d6-dshm\") pod \"tenantcfg-model-71bc98-kserve-7b7d7bc7bf-kqn2l\" (UID: \"847d367e-516e-4240-924c-3e62c532a8d6\") " pod="ai-tenant-e2e-shared-b-w1-71bc98/tenantcfg-model-71bc98-kserve-7b7d7bc7bf-kqn2l"
Oct 31 01:38:56.052605 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:38:56.052583    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lzdht\" (UniqueName: \"kubernetes.io/projected/847d367e-516e-4240-924c-3e62c532a8d6-kube-api-access-lzdht\") pod \"tenantcfg-model-71bc98-kserve-7b7d7bc7bf-kqn2l\" (UID: \"847d367e-516e-4240-924c-3e62c532a8d6\") " pod="ai-tenant-e2e-shared-b-w1-71bc98/tenantcfg-model-71bc98-kserve-7b7d7bc7bf-kqn2l"
Oct 31 01:38:56.107111 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:38:56.107085    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-71bc98/tenantcfg-model-71bc98-kserve-7b7d7bc7bf-kqn2l"
Oct 31 01:38:56.485856 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:38:56.485824    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w1-71bc98/tenantcfg-model-71bc98-kserve-7b7d7bc7bf-kqn2l"]
Oct 31 01:38:56.486704 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:38:56.486675    4964 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod847d367e_516e_4240_924c_3e62c532a8d6.slice/crio-5fc07c8937d63dd6e2b4cad8301e6fb430fad650a4b58233e16722c9688b8e2f WatchSource:0}: Error finding container 5fc07c8937d63dd6e2b4cad8301e6fb430fad650a4b58233e16722c9688b8e2f: Status 404 returned error can't find the container with id 5fc07c8937d63dd6e2b4cad8301e6fb430fad650a4b58233e16722c9688b8e2f
Oct 31 01:38:56.930437 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:38:56.930343    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-71bc98/tenantcfg-model-71bc98-kserve-7b7d7bc7bf-kqn2l" event={"ID":"847d367e-516e-4240-924c-3e62c532a8d6","Type":"ContainerStarted","Data":"18b53db5af341eb75b877294864d5a13cbd961e82a25657b7540bc930d628eb8"}
Oct 31 01:38:56.930437 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:38:56.930402    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-71bc98/tenantcfg-model-71bc98-kserve-7b7d7bc7bf-kqn2l" event={"ID":"847d367e-516e-4240-924c-3e62c532a8d6","Type":"ContainerStarted","Data":"5fc07c8937d63dd6e2b4cad8301e6fb430fad650a4b58233e16722c9688b8e2f"}
Oct 31 01:38:57.047517 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:38:57.047466    4964 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w1-71bc98/tenantcfg-model-71bc98-kserve-7b7d7bc7bf-kqn2l" podStartSLOduration=2.04745023 podStartE2EDuration="2.04745023s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:38:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:38:57.043973563 +0000 UTC m=+2105.520539940" watchObservedRunningTime="2026-10-31 01:38:57.04745023 +0000 UTC m=+2105.524016654"
Oct 31 01:38:57.107326 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:38:57.107280    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w1-71bc98/tenantcfg-model-71bc98-kserve-7b7d7bc7bf-kqn2l"
Oct 31 01:38:57.852479 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:38:57.852439    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:39:01.688681 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:39:01.688595    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:39:01.919123 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:39:01.919086    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:39:02.057312 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:39:02.057278    4964 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-8af859-794678d7b7-npqqw"]
Oct 31 01:39:02.061547 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:39:02.061531    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8af859-794678d7b7-npqqw"
Oct 31 01:39:02.121005 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:39:02.120978    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-8af859-794678d7b7-npqqw"]
Oct 31 01:39:02.163388 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:39:02.163366    4964 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-8af859\""
Oct 31 01:39:02.163536 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:39:02.163412    4964 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-8af859-dockercfg-l5sld\""
Oct 31 01:39:02.192006 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:39:02.191976    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sng6h\" (UniqueName: \"kubernetes.io/projected/8b94036e-3ae4-4026-9171-2ab95b1db68a-kube-api-access-sng6h\") pod \"payload-pre-processing-e2e-ipp-b-8af859-794678d7b7-npqqw\" (UID: \"8b94036e-3ae4-4026-9171-2ab95b1db68a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8af859-794678d7b7-npqqw"
Oct 31 01:39:02.192153 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:39:02.192106    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8b94036e-3ae4-4026-9171-2ab95b1db68a-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-8af859-794678d7b7-npqqw\" (UID: \"8b94036e-3ae4-4026-9171-2ab95b1db68a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8af859-794678d7b7-npqqw"
Oct 31 01:39:02.305350 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:39:02.299236    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sng6h\" (UniqueName: \"kubernetes.io/projected/8b94036e-3ae4-4026-9171-2ab95b1db68a-kube-api-access-sng6h\") pod \"payload-pre-processing-e2e-ipp-b-8af859-794678d7b7-npqqw\" (UID: \"8b94036e-3ae4-4026-9171-2ab95b1db68a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8af859-794678d7b7-npqqw"
Oct 31 01:39:02.305350 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:39:02.299378    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8b94036e-3ae4-4026-9171-2ab95b1db68a-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-8af859-794678d7b7-npqqw\" (UID: \"8b94036e-3ae4-4026-9171-2ab95b1db68a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8af859-794678d7b7-npqqw"
Oct 31 01:39:02.305350 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:39:02.300361    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8b94036e-3ae4-4026-9171-2ab95b1db68a-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-8af859-794678d7b7-npqqw\" (UID: \"8b94036e-3ae4-4026-9171-2ab95b1db68a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8af859-794678d7b7-npqqw"
Oct 31 01:39:02.358141 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:39:02.358054    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sng6h\" (UniqueName: \"kubernetes.io/projected/8b94036e-3ae4-4026-9171-2ab95b1db68a-kube-api-access-sng6h\") pod \"payload-pre-processing-e2e-ipp-b-8af859-794678d7b7-npqqw\" (UID: \"8b94036e-3ae4-4026-9171-2ab95b1db68a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8af859-794678d7b7-npqqw"
Oct 31 01:39:02.376916 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:39:02.376632    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8af859-794678d7b7-npqqw"
Oct 31 01:39:02.833383 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:39:02.832204    4964 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b94036e_3ae4_4026_9171_2ab95b1db68a.slice/crio-8e344044fdf3e76ebe91e0d20356847a703b27f91475ab16c835229cdae02154 WatchSource:0}: Error finding container 8e344044fdf3e76ebe91e0d20356847a703b27f91475ab16c835229cdae02154: Status 404 returned error can't find the container with id 8e344044fdf3e76ebe91e0d20356847a703b27f91475ab16c835229cdae02154
Oct 31 01:39:02.836520 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:39:02.836495    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-8af859-794678d7b7-npqqw"]
Oct 31 01:39:02.985516 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:39:02.985481    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8af859-794678d7b7-npqqw" event={"ID":"8b94036e-3ae4-4026-9171-2ab95b1db68a","Type":"ContainerStarted","Data":"fed262b4c7ca13a927cbfce65c8729cb24d4006f26d4f6697377b6095b2df5cd"}
Oct 31 01:39:02.985656 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:39:02.985537    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8af859-794678d7b7-npqqw" event={"ID":"8b94036e-3ae4-4026-9171-2ab95b1db68a","Type":"ContainerStarted","Data":"8e344044fdf3e76ebe91e0d20356847a703b27f91475ab16c835229cdae02154"}
Oct 31 01:39:02.985719 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:39:02.985702    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8af859-794678d7b7-npqqw"
Oct 31 01:39:03.292373 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:39:03.291452    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:39:05.829594 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:39:05.829563    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:39:06.032023 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:39:06.031978    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:39:06.122396 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:39:06.121721    4964 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w1-71bc98/tenantcfg-model-71bc98-kserve-7b7d7bc7bf-kqn2l"
Oct 31 01:39:06.140563 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:39:06.140532    4964 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w1-71bc98/tenantcfg-model-71bc98-kserve-7b7d7bc7bf-kqn2l"
Oct 31 01:39:06.275150 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:39:06.275093    4964 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8af859-794678d7b7-npqqw" podStartSLOduration=4.27507987 podStartE2EDuration="4.27507987s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:39:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:39:03.127144057 +0000 UTC m=+2111.603710438" watchObservedRunningTime="2026-10-31 01:39:06.27507987 +0000 UTC m=+2114.751646286"
Oct 31 01:39:07.055729 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:39:07.055702    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w1-71bc98/tenantcfg-model-71bc98-kserve-7b7d7bc7bf-kqn2l"
Oct 31 01:39:10.132203 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:39:10.131970    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:39:10.385310 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:39:10.385219    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:39:11.240162 ip-10-0-133-197 kubenswrapper[4964]: E1031 01:39:11.240120    4964 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q56x5" containerName="extractor"
Oct 31 01:39:12.083569 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:39:12.083499    4964 generic.go:356] "Generic (PLEG): container finished" podID="778c73f2-648f-4909-a1e9-a685d1d9a2c7" containerID="537884ca81ca127b04c754bf17d7fe77bcbae562cc0dcd4adf9033d91990d433" exitCode=0
Oct 31 01:39:12.083744 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:39:12.083564    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-53af08-openshift-default-556ff7748f-4nqv7" event={"ID":"778c73f2-648f-4909-a1e9-a685d1d9a2c7","Type":"ContainerDied","Data":"537884ca81ca127b04c754bf17d7fe77bcbae562cc0dcd4adf9033d91990d433"}
Oct 31 01:39:12.083972 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:39:12.083953    4964 scope.go:122] "RemoveContainer" containerID="537884ca81ca127b04c754bf17d7fe77bcbae562cc0dcd4adf9033d91990d433"
Oct 31 01:39:12.085515 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:39:12.085486    4964 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 01:39:12.085620 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:39:12.085556    4964 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 01:39:12.085620 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:39:12.085588    4964 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 01:39:12.690691 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:39:12.690653    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:39:13.109349 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:39:13.105671    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-53af08-openshift-default-556ff7748f-4nqv7" event={"ID":"778c73f2-648f-4909-a1e9-a685d1d9a2c7","Type":"ContainerStarted","Data":"b79a849b557c24c90da3d9ff4869534df181b56f2702e63570fa562aa0df24e6"}
Oct 31 01:39:13.511342 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:39:13.510703    4964 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-53af08-openshift-default-556ff7748f-4nqv7"
Oct 31 01:39:13.511342 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:39:13.510831    4964 patch_prober.go:29] interesting pod/e2e-ipp-a-53af08-openshift-default-556ff7748f-4nqv7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused" start-of-body=
Oct 31 01:39:13.511342 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:39:13.510876    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-53af08-openshift-default-556ff7748f-4nqv7" podUID="778c73f2-648f-4909-a1e9-a685d1d9a2c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused"
Oct 31 01:39:14.021811 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:39:14.021782    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8af859-794678d7b7-npqqw"
Oct 31 01:39:14.515345 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:39:14.514653    4964 patch_prober.go:29] interesting pod/e2e-ipp-a-53af08-openshift-default-556ff7748f-4nqv7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused" start-of-body=
Oct 31 01:39:14.515345 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:39:14.514710    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-53af08-openshift-default-556ff7748f-4nqv7" podUID="778c73f2-648f-4909-a1e9-a685d1d9a2c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused"
Oct 31 01:39:15.180095 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:39:15.180059    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:39:15.358702 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:39:15.358666    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:39:15.510341 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:39:15.509747    4964 patch_prober.go:29] interesting pod/e2e-ipp-a-53af08-openshift-default-556ff7748f-4nqv7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused" start-of-body=
Oct 31 01:39:15.510341 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:39:15.509810    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-53af08-openshift-default-556ff7748f-4nqv7" podUID="778c73f2-648f-4909-a1e9-a685d1d9a2c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused"
Oct 31 01:39:15.531018 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:39:15.530978    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:39:16.512142 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:39:16.511704    4964 patch_prober.go:29] interesting pod/e2e-ipp-a-53af08-openshift-default-556ff7748f-4nqv7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused" start-of-body=
Oct 31 01:39:16.512142 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:39:16.511764    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-53af08-openshift-default-556ff7748f-4nqv7" podUID="778c73f2-648f-4909-a1e9-a685d1d9a2c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused"
Oct 31 01:39:16.734805 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:39:16.734773    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:39:17.506722 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:39:17.506685    4964 patch_prober.go:29] interesting pod/e2e-ipp-a-53af08-openshift-default-556ff7748f-4nqv7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused" start-of-body=
Oct 31 01:39:17.507029 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:39:17.506998    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-53af08-openshift-default-556ff7748f-4nqv7" podUID="778c73f2-648f-4909-a1e9-a685d1d9a2c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused"
Oct 31 01:39:18.506259 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:39:18.506216    4964 patch_prober.go:29] interesting pod/e2e-ipp-a-53af08-openshift-default-556ff7748f-4nqv7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused" start-of-body=
Oct 31 01:39:18.506702 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:39:18.506292    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-53af08-openshift-default-556ff7748f-4nqv7" podUID="778c73f2-648f-4909-a1e9-a685d1d9a2c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused"
Oct 31 01:39:19.511363 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:39:19.511331    4964 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-53af08-openshift-default-556ff7748f-4nqv7"
Oct 31 01:39:19.511854 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:39:19.511839    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-53af08-openshift-default-556ff7748f-4nqv7"
Oct 31 01:39:19.512505 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:39:19.512481    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-53af08-openshift-default-556ff7748f-4nqv7"
Oct 31 01:39:21.507466 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:39:21.507431    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:39:26.307886 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:39:26.307832    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:39:31.059050 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:39:31.059014    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:39:35.779081 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:39:35.779044    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:39:40.520467 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:39:40.520430    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:39:41.237234 ip-10-0-133-197 kubenswrapper[4964]: E1031 01:39:41.237198    4964 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q56x5" containerName="extractor"
Oct 31 01:39:45.506591 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:39:45.506548    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:39:49.267822 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:39:49.267777    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:39:49.782709 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:39:49.782659    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:39:52.341848 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:39:52.341810    4964 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-df5d0b-6f9767f49c-gxnsd"]
Oct 31 01:39:52.346899 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:39:52.346874    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-df5d0b-6f9767f49c-gxnsd"
Oct 31 01:39:52.398721 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:39:52.398695    4964 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-df5d0b-serving-cert\""
Oct 31 01:39:52.402378 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:39:52.402357    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/70860802-a6eb-4d36-a11d-77f9c006a93a-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-df5d0b-6f9767f49c-gxnsd\" (UID: \"70860802-a6eb-4d36-a11d-77f9c006a93a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-df5d0b-6f9767f49c-gxnsd"
Oct 31 01:39:52.402484 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:39:52.402470    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/70860802-a6eb-4d36-a11d-77f9c006a93a-metrics-certs\") pod \"maas-api-e2e-ipp-b-df5d0b-6f9767f49c-gxnsd\" (UID: \"70860802-a6eb-4d36-a11d-77f9c006a93a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-df5d0b-6f9767f49c-gxnsd"
Oct 31 01:39:52.402583 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:39:52.402570    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/70860802-a6eb-4d36-a11d-77f9c006a93a-maas-api-tls\") pod \"maas-api-e2e-ipp-b-df5d0b-6f9767f49c-gxnsd\" (UID: \"70860802-a6eb-4d36-a11d-77f9c006a93a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-df5d0b-6f9767f49c-gxnsd"
Oct 31 01:39:52.402625 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:39:52.402594    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hbjjs\" (UniqueName: \"kubernetes.io/projected/70860802-a6eb-4d36-a11d-77f9c006a93a-kube-api-access-hbjjs\") pod \"maas-api-e2e-ipp-b-df5d0b-6f9767f49c-gxnsd\" (UID: \"70860802-a6eb-4d36-a11d-77f9c006a93a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-df5d0b-6f9767f49c-gxnsd"
Oct 31 01:39:52.453170 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:39:52.453138    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-df5d0b-6f9767f49c-gxnsd"]
Oct 31 01:39:52.503426 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:39:52.503393    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/70860802-a6eb-4d36-a11d-77f9c006a93a-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-df5d0b-6f9767f49c-gxnsd\" (UID: \"70860802-a6eb-4d36-a11d-77f9c006a93a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-df5d0b-6f9767f49c-gxnsd"
Oct 31 01:39:52.503587 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:39:52.503482    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/70860802-a6eb-4d36-a11d-77f9c006a93a-metrics-certs\") pod \"maas-api-e2e-ipp-b-df5d0b-6f9767f49c-gxnsd\" (UID: \"70860802-a6eb-4d36-a11d-77f9c006a93a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-df5d0b-6f9767f49c-gxnsd"
Oct 31 01:39:52.503587 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:39:52.503542    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/70860802-a6eb-4d36-a11d-77f9c006a93a-maas-api-tls\") pod \"maas-api-e2e-ipp-b-df5d0b-6f9767f49c-gxnsd\" (UID: \"70860802-a6eb-4d36-a11d-77f9c006a93a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-df5d0b-6f9767f49c-gxnsd"
Oct 31 01:39:52.503587 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:39:52.503562    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hbjjs\" (UniqueName: \"kubernetes.io/projected/70860802-a6eb-4d36-a11d-77f9c006a93a-kube-api-access-hbjjs\") pod \"maas-api-e2e-ipp-b-df5d0b-6f9767f49c-gxnsd\" (UID: \"70860802-a6eb-4d36-a11d-77f9c006a93a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-df5d0b-6f9767f49c-gxnsd"
Oct 31 01:39:52.503763 ip-10-0-133-197 kubenswrapper[4964]: E1031 01:39:52.503664    4964 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-b-df5d0b-serving-cert: secret "maas-api-e2e-ipp-b-df5d0b-serving-cert" not found
Oct 31 01:39:52.503763 ip-10-0-133-197 kubenswrapper[4964]: E1031 01:39:52.503738    4964 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/70860802-a6eb-4d36-a11d-77f9c006a93a-maas-api-tls podName:70860802-a6eb-4d36-a11d-77f9c006a93a nodeName:}" failed. No retries permitted until 2026-10-31 01:39:53.003715526 +0000 UTC m=+2161.480281908 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/70860802-a6eb-4d36-a11d-77f9c006a93a-maas-api-tls") pod "maas-api-e2e-ipp-b-df5d0b-6f9767f49c-gxnsd" (UID: "70860802-a6eb-4d36-a11d-77f9c006a93a") : secret "maas-api-e2e-ipp-b-df5d0b-serving-cert" not found
Oct 31 01:39:52.504135 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:39:52.504113    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/70860802-a6eb-4d36-a11d-77f9c006a93a-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-df5d0b-6f9767f49c-gxnsd\" (UID: \"70860802-a6eb-4d36-a11d-77f9c006a93a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-df5d0b-6f9767f49c-gxnsd"
Oct 31 01:39:52.506525 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:39:52.506504    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/70860802-a6eb-4d36-a11d-77f9c006a93a-metrics-certs\") pod \"maas-api-e2e-ipp-b-df5d0b-6f9767f49c-gxnsd\" (UID: \"70860802-a6eb-4d36-a11d-77f9c006a93a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-df5d0b-6f9767f49c-gxnsd"
Oct 31 01:39:52.562592 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:39:52.562556    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hbjjs\" (UniqueName: \"kubernetes.io/projected/70860802-a6eb-4d36-a11d-77f9c006a93a-kube-api-access-hbjjs\") pod \"maas-api-e2e-ipp-b-df5d0b-6f9767f49c-gxnsd\" (UID: \"70860802-a6eb-4d36-a11d-77f9c006a93a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-df5d0b-6f9767f49c-gxnsd"
Oct 31 01:39:52.577822 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:39:52.577794    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:39:52.805150 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:39:52.805115    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:39:53.013051 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:39:53.013020    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/70860802-a6eb-4d36-a11d-77f9c006a93a-maas-api-tls\") pod \"maas-api-e2e-ipp-b-df5d0b-6f9767f49c-gxnsd\" (UID: \"70860802-a6eb-4d36-a11d-77f9c006a93a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-df5d0b-6f9767f49c-gxnsd"
Oct 31 01:39:53.015770 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:39:53.015741    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/70860802-a6eb-4d36-a11d-77f9c006a93a-maas-api-tls\") pod \"maas-api-e2e-ipp-b-df5d0b-6f9767f49c-gxnsd\" (UID: \"70860802-a6eb-4d36-a11d-77f9c006a93a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-df5d0b-6f9767f49c-gxnsd"
Oct 31 01:39:53.260453 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:39:53.260423    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-df5d0b-6f9767f49c-gxnsd"
Oct 31 01:39:53.641209 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:39:53.641118    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-df5d0b-6f9767f49c-gxnsd"]
Oct 31 01:39:53.646582 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:39:53.646550    4964 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70860802_a6eb_4d36_a11d_77f9c006a93a.slice/crio-b7cc86bffe62dbb4354f34dc345d91fe762b7e0f15768894cc6fa9f468831c50 WatchSource:0}: Error finding container b7cc86bffe62dbb4354f34dc345d91fe762b7e0f15768894cc6fa9f468831c50: Status 404 returned error can't find the container with id b7cc86bffe62dbb4354f34dc345d91fe762b7e0f15768894cc6fa9f468831c50
Oct 31 01:39:53.648378 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:39:53.648357    4964 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 31 01:39:54.400449 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:39:54.400418    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-df5d0b-6f9767f49c-gxnsd" event={"ID":"70860802-a6eb-4d36-a11d-77f9c006a93a","Type":"ContainerStarted","Data":"b7cc86bffe62dbb4354f34dc345d91fe762b7e0f15768894cc6fa9f468831c50"}
Oct 31 01:39:55.409747 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:39:55.409701    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-df5d0b-6f9767f49c-gxnsd" event={"ID":"70860802-a6eb-4d36-a11d-77f9c006a93a","Type":"ContainerStarted","Data":"a5871846f0d6622ac926109861ae68e725c79d0dc8292fa2e3b809d2be05b5b3"}
Oct 31 01:39:55.410214 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:39:55.409776    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-df5d0b-6f9767f49c-gxnsd"
Oct 31 01:39:55.529131 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:39:55.529078    4964 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-df5d0b-6f9767f49c-gxnsd" podStartSLOduration=2.5351364 podStartE2EDuration="3.529063725s" totalImagesPullingTime="993.927325ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:39:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:39:55.525628306 +0000 UTC m=+2164.002194686" watchObservedRunningTime="2026-10-31 01:39:55.529063725 +0000 UTC m=+2164.005630104"
Oct 31 01:39:56.237997 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:39:56.237965    4964 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-cdtt2"]
Oct 31 01:39:56.238194 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:39:56.238156    4964 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-cdtt2" podUID="ea794d28-3a8e-4262-bdaa-734f3510cfbe" containerName="istio-proxy" containerID="cri-o://4900b06e3bcba7ad06cfc24b420b0e3221df977167ed690370636bcae546bee2" gracePeriod=30
Oct 31 01:39:57.812181 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:39:57.812143    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:39:58.706198 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:39:58.706163    4964 patch_prober.go:29] interesting pod/e2e-worker-w1-baacee-openshift-default-57f845d5bf-cdtt2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 01:39:58.706513 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:39:58.706221    4964 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-cdtt2" podUID="ea794d28-3a8e-4262-bdaa-734f3510cfbe" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 01:39:58.814335 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:39:58.814288    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:40:00.266754 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:40:00.266712    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:40:00.446381 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:40:00.446340    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:40:01.422482 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:40:01.422455    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-df5d0b-6f9767f49c-gxnsd"
Oct 31 01:40:01.455641 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:40:01.455605    4964 generic.go:356] "Generic (PLEG): container finished" podID="ea794d28-3a8e-4262-bdaa-734f3510cfbe" containerID="4900b06e3bcba7ad06cfc24b420b0e3221df977167ed690370636bcae546bee2" exitCode=0
Oct 31 01:40:01.455811 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:40:01.455652    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-cdtt2" event={"ID":"ea794d28-3a8e-4262-bdaa-734f3510cfbe","Type":"ContainerDied","Data":"4900b06e3bcba7ad06cfc24b420b0e3221df977167ed690370636bcae546bee2"}
Oct 31 01:40:01.605878 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:40:01.605847    4964 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-cdtt2"
Oct 31 01:40:01.732158 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:40:01.732117    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea794d28-3a8e-4262-bdaa-734f3510cfbe-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ea794d28-3a8e-4262-bdaa-734f3510cfbe-istio-data\") pod \"ea794d28-3a8e-4262-bdaa-734f3510cfbe\" (UID: \"ea794d28-3a8e-4262-bdaa-734f3510cfbe\") "
Oct 31 01:40:01.732395 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:40:01.732195    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea794d28-3a8e-4262-bdaa-734f3510cfbe-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ea794d28-3a8e-4262-bdaa-734f3510cfbe-workload-certs\") pod \"ea794d28-3a8e-4262-bdaa-734f3510cfbe\" (UID: \"ea794d28-3a8e-4262-bdaa-734f3510cfbe\") "
Oct 31 01:40:01.732395 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:40:01.732249    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea794d28-3a8e-4262-bdaa-734f3510cfbe-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea794d28-3a8e-4262-bdaa-734f3510cfbe-workload-socket\") pod \"ea794d28-3a8e-4262-bdaa-734f3510cfbe\" (UID: \"ea794d28-3a8e-4262-bdaa-734f3510cfbe\") "
Oct 31 01:40:01.732395 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:40:01.732302    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea794d28-3a8e-4262-bdaa-734f3510cfbe-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea794d28-3a8e-4262-bdaa-734f3510cfbe-credential-socket\") pod \"ea794d28-3a8e-4262-bdaa-734f3510cfbe\" (UID: \"ea794d28-3a8e-4262-bdaa-734f3510cfbe\") "
Oct 31 01:40:01.732395 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:40:01.732374    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ea794d28-3a8e-4262-bdaa-734f3510cfbe-istio-token\" (UniqueName: \"kubernetes.io/projected/ea794d28-3a8e-4262-bdaa-734f3510cfbe-istio-token\") pod \"ea794d28-3a8e-4262-bdaa-734f3510cfbe\" (UID: \"ea794d28-3a8e-4262-bdaa-734f3510cfbe\") "
Oct 31 01:40:01.732655 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:40:01.732408    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ea794d28-3a8e-4262-bdaa-734f3510cfbe-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ea794d28-3a8e-4262-bdaa-734f3510cfbe-istiod-ca-cert\") pod \"ea794d28-3a8e-4262-bdaa-734f3510cfbe\" (UID: \"ea794d28-3a8e-4262-bdaa-734f3510cfbe\") "
Oct 31 01:40:01.732655 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:40:01.732502    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea794d28-3a8e-4262-bdaa-734f3510cfbe-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ea794d28-3a8e-4262-bdaa-734f3510cfbe-istio-envoy\") pod \"ea794d28-3a8e-4262-bdaa-734f3510cfbe\" (UID: \"ea794d28-3a8e-4262-bdaa-734f3510cfbe\") "
Oct 31 01:40:01.732655 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:40:01.732533    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea794d28-3a8e-4262-bdaa-734f3510cfbe-istio-data" pod "ea794d28-3a8e-4262-bdaa-734f3510cfbe" (UID: "ea794d28-3a8e-4262-bdaa-734f3510cfbe"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:40:01.732824 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:40:01.732653    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ea794d28-3a8e-4262-bdaa-734f3510cfbe-kube-api-access-68kzr\" (UniqueName: \"kubernetes.io/projected/ea794d28-3a8e-4262-bdaa-734f3510cfbe-kube-api-access-68kzr\") pod \"ea794d28-3a8e-4262-bdaa-734f3510cfbe\" (UID: \"ea794d28-3a8e-4262-bdaa-734f3510cfbe\") "
Oct 31 01:40:01.732824 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:40:01.732683    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ea794d28-3a8e-4262-bdaa-734f3510cfbe-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ea794d28-3a8e-4262-bdaa-734f3510cfbe-istio-podinfo\") pod \"ea794d28-3a8e-4262-bdaa-734f3510cfbe\" (UID: \"ea794d28-3a8e-4262-bdaa-734f3510cfbe\") "
Oct 31 01:40:01.732824 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:40:01.732774    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea794d28-3a8e-4262-bdaa-734f3510cfbe-credential-socket" pod "ea794d28-3a8e-4262-bdaa-734f3510cfbe" (UID: "ea794d28-3a8e-4262-bdaa-734f3510cfbe"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:40:01.733844 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:40:01.733024    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea794d28-3a8e-4262-bdaa-734f3510cfbe-workload-certs" pod "ea794d28-3a8e-4262-bdaa-734f3510cfbe" (UID: "ea794d28-3a8e-4262-bdaa-734f3510cfbe"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:40:01.733844 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:40:01.733815    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea794d28-3a8e-4262-bdaa-734f3510cfbe-workload-socket" pod "ea794d28-3a8e-4262-bdaa-734f3510cfbe" (UID: "ea794d28-3a8e-4262-bdaa-734f3510cfbe"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:40:01.734631 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:40:01.734609    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ea794d28-3a8e-4262-bdaa-734f3510cfbe-istiod-ca-cert" pod "ea794d28-3a8e-4262-bdaa-734f3510cfbe" (UID: "ea794d28-3a8e-4262-bdaa-734f3510cfbe"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:40:01.736465 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:40:01.736445    4964 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ea794d28-3a8e-4262-bdaa-734f3510cfbe-istio-data\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:40:01.736599 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:40:01.736585    4964 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ea794d28-3a8e-4262-bdaa-734f3510cfbe-workload-certs\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:40:01.736703 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:40:01.736689    4964 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea794d28-3a8e-4262-bdaa-734f3510cfbe-credential-socket\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:40:01.741629 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:40:01.741583    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea794d28-3a8e-4262-bdaa-734f3510cfbe-istio-envoy" pod "ea794d28-3a8e-4262-bdaa-734f3510cfbe" (UID: "ea794d28-3a8e-4262-bdaa-734f3510cfbe"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:40:01.744435 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:40:01.744406    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea794d28-3a8e-4262-bdaa-734f3510cfbe-kube-api-access-68kzr" pod "ea794d28-3a8e-4262-bdaa-734f3510cfbe" (UID: "ea794d28-3a8e-4262-bdaa-734f3510cfbe"). InnerVolumeSpecName "kube-api-access-68kzr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:40:01.747383 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:40:01.747355    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea794d28-3a8e-4262-bdaa-734f3510cfbe-istio-token" pod "ea794d28-3a8e-4262-bdaa-734f3510cfbe" (UID: "ea794d28-3a8e-4262-bdaa-734f3510cfbe"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:40:01.748507 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:40:01.748420    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ea794d28-3a8e-4262-bdaa-734f3510cfbe-istio-podinfo" pod "ea794d28-3a8e-4262-bdaa-734f3510cfbe" (UID: "ea794d28-3a8e-4262-bdaa-734f3510cfbe"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 01:40:01.838226 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:40:01.838192    4964 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-68kzr\" (UniqueName: \"kubernetes.io/projected/ea794d28-3a8e-4262-bdaa-734f3510cfbe-kube-api-access-68kzr\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:40:01.838226 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:40:01.838227    4964 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ea794d28-3a8e-4262-bdaa-734f3510cfbe-istio-podinfo\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:40:01.838453 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:40:01.838241    4964 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea794d28-3a8e-4262-bdaa-734f3510cfbe-workload-socket\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:40:01.838453 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:40:01.838253    4964 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ea794d28-3a8e-4262-bdaa-734f3510cfbe-istio-token\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:40:01.838453 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:40:01.838265    4964 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ea794d28-3a8e-4262-bdaa-734f3510cfbe-istiod-ca-cert\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:40:01.838453 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:40:01.838277    4964 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ea794d28-3a8e-4262-bdaa-734f3510cfbe-istio-envoy\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:40:02.060958 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:40:02.060864    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:40:02.463429 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:40:02.463331    4964 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-cdtt2"
Oct 31 01:40:02.463429 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:40:02.463347    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-cdtt2" event={"ID":"ea794d28-3a8e-4262-bdaa-734f3510cfbe","Type":"ContainerDied","Data":"2af817e99adf4134d43937626b2a054cff12c21db7d84b7f887a07d570ef6d05"}
Oct 31 01:40:02.463939 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:40:02.463436    4964 scope.go:122] "RemoveContainer" containerID="4900b06e3bcba7ad06cfc24b420b0e3221df977167ed690370636bcae546bee2"
Oct 31 01:40:02.676710 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:40:02.676669    4964 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-cdtt2"]
Oct 31 01:40:02.699922 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:40:02.699881    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:40:02.727865 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:40:02.727826    4964 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-cdtt2"]
Oct 31 01:40:03.777871 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:40:03.777840    4964 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ea794d28-3a8e-4262-bdaa-734f3510cfbe" path="/var/lib/kubelet/pods/ea794d28-3a8e-4262-bdaa-734f3510cfbe/volumes"
Oct 31 01:40:07.107075 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:40:07.107038    4964 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-8af859-6b6b774599-r4ql2"]
Oct 31 01:40:07.107727 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:40:07.107710    4964 cpu_manager.go:414] "containerMap: removing container" podUID="ea794d28-3a8e-4262-bdaa-734f3510cfbe" containerName="istio-proxy"
Oct 31 01:40:07.107727 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:40:07.107727    4964 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ea794d28-3a8e-4262-bdaa-734f3510cfbe" containerName="istio-proxy"
Oct 31 01:40:07.107874 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:40:07.107862    4964 memory_manager.go:365] "RemoveStaleState removing state" podUID="ea794d28-3a8e-4262-bdaa-734f3510cfbe" containerName="istio-proxy"
Oct 31 01:40:07.112631 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:40:07.112613    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8af859-6b6b774599-r4ql2"
Oct 31 01:40:07.204205 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:40:07.204157    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7wlrn\" (UniqueName: \"kubernetes.io/projected/cf5d4bc6-4111-4c96-8ab9-27a461e3e000-kube-api-access-7wlrn\") pod \"maas-api-e2e-ipp-b-8af859-6b6b774599-r4ql2\" (UID: \"cf5d4bc6-4111-4c96-8ab9-27a461e3e000\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8af859-6b6b774599-r4ql2"
Oct 31 01:40:07.204418 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:40:07.204285    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cf5d4bc6-4111-4c96-8ab9-27a461e3e000-maas-api-tls\") pod \"maas-api-e2e-ipp-b-8af859-6b6b774599-r4ql2\" (UID: \"cf5d4bc6-4111-4c96-8ab9-27a461e3e000\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8af859-6b6b774599-r4ql2"
Oct 31 01:40:07.204502 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:40:07.204419    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cf5d4bc6-4111-4c96-8ab9-27a461e3e000-metrics-certs\") pod \"maas-api-e2e-ipp-b-8af859-6b6b774599-r4ql2\" (UID: \"cf5d4bc6-4111-4c96-8ab9-27a461e3e000\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8af859-6b6b774599-r4ql2"
Oct 31 01:40:07.204502 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:40:07.204483    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cf5d4bc6-4111-4c96-8ab9-27a461e3e000-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-8af859-6b6b774599-r4ql2\" (UID: \"cf5d4bc6-4111-4c96-8ab9-27a461e3e000\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8af859-6b6b774599-r4ql2"
Oct 31 01:40:07.214758 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:40:07.214733    4964 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-8af859-serving-cert\""
Oct 31 01:40:07.222735 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:40:07.222704    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-8af859-6b6b774599-r4ql2"]
Oct 31 01:40:07.305390 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:40:07.305357    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cf5d4bc6-4111-4c96-8ab9-27a461e3e000-maas-api-tls\") pod \"maas-api-e2e-ipp-b-8af859-6b6b774599-r4ql2\" (UID: \"cf5d4bc6-4111-4c96-8ab9-27a461e3e000\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8af859-6b6b774599-r4ql2"
Oct 31 01:40:07.305560 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:40:07.305474    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cf5d4bc6-4111-4c96-8ab9-27a461e3e000-metrics-certs\") pod \"maas-api-e2e-ipp-b-8af859-6b6b774599-r4ql2\" (UID: \"cf5d4bc6-4111-4c96-8ab9-27a461e3e000\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8af859-6b6b774599-r4ql2"
Oct 31 01:40:07.305560 ip-10-0-133-197 kubenswrapper[4964]: E1031 01:40:07.305488    4964 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-b-8af859-serving-cert: secret "maas-api-e2e-ipp-b-8af859-serving-cert" not found
Oct 31 01:40:07.305560 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:40:07.305538    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cf5d4bc6-4111-4c96-8ab9-27a461e3e000-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-8af859-6b6b774599-r4ql2\" (UID: \"cf5d4bc6-4111-4c96-8ab9-27a461e3e000\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8af859-6b6b774599-r4ql2"
Oct 31 01:40:07.305560 ip-10-0-133-197 kubenswrapper[4964]: E1031 01:40:07.305554    4964 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cf5d4bc6-4111-4c96-8ab9-27a461e3e000-maas-api-tls podName:cf5d4bc6-4111-4c96-8ab9-27a461e3e000 nodeName:}" failed. No retries permitted until 2026-10-31 01:40:07.805537747 +0000 UTC m=+2176.282104128 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/cf5d4bc6-4111-4c96-8ab9-27a461e3e000-maas-api-tls") pod "maas-api-e2e-ipp-b-8af859-6b6b774599-r4ql2" (UID: "cf5d4bc6-4111-4c96-8ab9-27a461e3e000") : secret "maas-api-e2e-ipp-b-8af859-serving-cert" not found
Oct 31 01:40:07.305787 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:40:07.305681    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7wlrn\" (UniqueName: \"kubernetes.io/projected/cf5d4bc6-4111-4c96-8ab9-27a461e3e000-kube-api-access-7wlrn\") pod \"maas-api-e2e-ipp-b-8af859-6b6b774599-r4ql2\" (UID: \"cf5d4bc6-4111-4c96-8ab9-27a461e3e000\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8af859-6b6b774599-r4ql2"
Oct 31 01:40:07.306259 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:40:07.306235    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cf5d4bc6-4111-4c96-8ab9-27a461e3e000-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-8af859-6b6b774599-r4ql2\" (UID: \"cf5d4bc6-4111-4c96-8ab9-27a461e3e000\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8af859-6b6b774599-r4ql2"
Oct 31 01:40:07.308217 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:40:07.308192    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cf5d4bc6-4111-4c96-8ab9-27a461e3e000-metrics-certs\") pod \"maas-api-e2e-ipp-b-8af859-6b6b774599-r4ql2\" (UID: \"cf5d4bc6-4111-4c96-8ab9-27a461e3e000\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8af859-6b6b774599-r4ql2"
Oct 31 01:40:07.344627 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:40:07.344590    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:40:07.368196 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:40:07.368122    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7wlrn\" (UniqueName: \"kubernetes.io/projected/cf5d4bc6-4111-4c96-8ab9-27a461e3e000-kube-api-access-7wlrn\") pod \"maas-api-e2e-ipp-b-8af859-6b6b774599-r4ql2\" (UID: \"cf5d4bc6-4111-4c96-8ab9-27a461e3e000\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8af859-6b6b774599-r4ql2"
Oct 31 01:40:07.548856 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:40:07.548815    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:40:07.813703 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:40:07.813668    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cf5d4bc6-4111-4c96-8ab9-27a461e3e000-maas-api-tls\") pod \"maas-api-e2e-ipp-b-8af859-6b6b774599-r4ql2\" (UID: \"cf5d4bc6-4111-4c96-8ab9-27a461e3e000\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8af859-6b6b774599-r4ql2"
Oct 31 01:40:07.816843 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:40:07.816817    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cf5d4bc6-4111-4c96-8ab9-27a461e3e000-maas-api-tls\") pod \"maas-api-e2e-ipp-b-8af859-6b6b774599-r4ql2\" (UID: \"cf5d4bc6-4111-4c96-8ab9-27a461e3e000\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8af859-6b6b774599-r4ql2"
Oct 31 01:40:08.026561 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:40:08.026524    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8af859-6b6b774599-r4ql2"
Oct 31 01:40:08.405757 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:40:08.405722    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-8af859-6b6b774599-r4ql2"]
Oct 31 01:40:08.407109 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:40:08.407084    4964 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf5d4bc6_4111_4c96_8ab9_27a461e3e000.slice/crio-3d87ef629a92740b10008ddd5cb305b5ab841ad13c863a8bfddf3ca53fe6d011 WatchSource:0}: Error finding container 3d87ef629a92740b10008ddd5cb305b5ab841ad13c863a8bfddf3ca53fe6d011: Status 404 returned error can't find the container with id 3d87ef629a92740b10008ddd5cb305b5ab841ad13c863a8bfddf3ca53fe6d011
Oct 31 01:40:08.509929 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:40:08.509899    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8af859-6b6b774599-r4ql2" event={"ID":"cf5d4bc6-4111-4c96-8ab9-27a461e3e000","Type":"ContainerStarted","Data":"3d87ef629a92740b10008ddd5cb305b5ab841ad13c863a8bfddf3ca53fe6d011"}
Oct 31 01:40:09.523365 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:40:09.523263    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8af859-6b6b774599-r4ql2" event={"ID":"cf5d4bc6-4111-4c96-8ab9-27a461e3e000","Type":"ContainerStarted","Data":"3a9b3304ac4a85060fbfae35964d3977f0e18b23291e66dfc5117dce2979cd41"}
Oct 31 01:40:09.523754 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:40:09.523385    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8af859-6b6b774599-r4ql2"
Oct 31 01:40:09.641227 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:40:09.641162    4964 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8af859-6b6b774599-r4ql2" podStartSLOduration=1.816582176 podStartE2EDuration="2.6411452s" totalImagesPullingTime="824.563024ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:40:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:40:09.638023728 +0000 UTC m=+2178.114590107" watchObservedRunningTime="2026-10-31 01:40:09.6411452 +0000 UTC m=+2178.117711582"
Oct 31 01:40:10.716776 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:40:10.716739    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:40:11.274787 ip-10-0-133-197 kubenswrapper[4964]: E1031 01:40:11.274748    4964 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q56x5" containerName="extractor"
Oct 31 01:40:12.311216 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:40:12.311162    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:40:13.021764 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:40:13.021727    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:40:15.537163 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:40:15.537134    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8af859-6b6b774599-r4ql2"
Oct 31 01:40:17.384149 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:40:17.384118    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:40:20.922785 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:40:20.922738    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:40:21.416164 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:40:21.416124    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:40:24.196545 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:40:24.196511    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:40:30.799880 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:40:30.799791    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:40:37.218656 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:40:37.218617    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:40:41.286240 ip-10-0-133-197 kubenswrapper[4964]: E1031 01:40:41.286185    4964 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q56x5" containerName="extractor"
Oct 31 01:40:41.998116 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:40:41.998080    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:40:46.778190 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:40:46.778159    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:40:48.064652 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:40:48.064616    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:40:48.587439 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:40:48.587409    4964 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-b2d49b-openshift-default-d7d8667b7-69jgz"]
Oct 31 01:40:48.591585 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:40:48.591565    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-b2d49b-openshift-default-d7d8667b7-69jgz"
Oct 31 01:40:48.646412 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:40:48.646383    4964 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-b2d49b-openshift-default-dockercfg-l7tpw\""
Oct 31 01:40:48.655332 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:40:48.655276    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2mxkl\" (UniqueName: \"kubernetes.io/projected/710b2ea2-bffd-45b2-b0fc-0e5a90defed1-kube-api-access-2mxkl\") pod \"e2e-ipp-b-b2d49b-openshift-default-d7d8667b7-69jgz\" (UID: \"710b2ea2-bffd-45b2-b0fc-0e5a90defed1\") " pod="openshift-ingress/e2e-ipp-b-b2d49b-openshift-default-d7d8667b7-69jgz"
Oct 31 01:40:48.655700 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:40:48.655676    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/710b2ea2-bffd-45b2-b0fc-0e5a90defed1-workload-socket\") pod \"e2e-ipp-b-b2d49b-openshift-default-d7d8667b7-69jgz\" (UID: \"710b2ea2-bffd-45b2-b0fc-0e5a90defed1\") " pod="openshift-ingress/e2e-ipp-b-b2d49b-openshift-default-d7d8667b7-69jgz"
Oct 31 01:40:48.712501 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:40:48.711122    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-b2d49b-openshift-default-d7d8667b7-69jgz"]
Oct 31 01:40:48.735448 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:40:48.733759    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:40:48.735844 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:40:48.735823    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/710b2ea2-bffd-45b2-b0fc-0e5a90defed1-workload-certs\") pod \"e2e-ipp-b-b2d49b-openshift-default-d7d8667b7-69jgz\" (UID: \"710b2ea2-bffd-45b2-b0fc-0e5a90defed1\") " pod="openshift-ingress/e2e-ipp-b-b2d49b-openshift-default-d7d8667b7-69jgz"
Oct 31 01:40:48.751276 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:40:48.751243    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/710b2ea2-bffd-45b2-b0fc-0e5a90defed1-istio-podinfo\") pod \"e2e-ipp-b-b2d49b-openshift-default-d7d8667b7-69jgz\" (UID: \"710b2ea2-bffd-45b2-b0fc-0e5a90defed1\") " pod="openshift-ingress/e2e-ipp-b-b2d49b-openshift-default-d7d8667b7-69jgz"
Oct 31 01:40:48.751746 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:40:48.751717    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/710b2ea2-bffd-45b2-b0fc-0e5a90defed1-istio-data\") pod \"e2e-ipp-b-b2d49b-openshift-default-d7d8667b7-69jgz\" (UID: \"710b2ea2-bffd-45b2-b0fc-0e5a90defed1\") " pod="openshift-ingress/e2e-ipp-b-b2d49b-openshift-default-d7d8667b7-69jgz"
Oct 31 01:40:48.751943 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:40:48.751920    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/710b2ea2-bffd-45b2-b0fc-0e5a90defed1-istio-token\") pod \"e2e-ipp-b-b2d49b-openshift-default-d7d8667b7-69jgz\" (UID: \"710b2ea2-bffd-45b2-b0fc-0e5a90defed1\") " pod="openshift-ingress/e2e-ipp-b-b2d49b-openshift-default-d7d8667b7-69jgz"
Oct 31 01:40:48.752193 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:40:48.752171    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/710b2ea2-bffd-45b2-b0fc-0e5a90defed1-istio-envoy\") pod \"e2e-ipp-b-b2d49b-openshift-default-d7d8667b7-69jgz\" (UID: \"710b2ea2-bffd-45b2-b0fc-0e5a90defed1\") " pod="openshift-ingress/e2e-ipp-b-b2d49b-openshift-default-d7d8667b7-69jgz"
Oct 31 01:40:48.752446 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:40:48.752422    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/710b2ea2-bffd-45b2-b0fc-0e5a90defed1-istiod-ca-cert\") pod \"e2e-ipp-b-b2d49b-openshift-default-d7d8667b7-69jgz\" (UID: \"710b2ea2-bffd-45b2-b0fc-0e5a90defed1\") " pod="openshift-ingress/e2e-ipp-b-b2d49b-openshift-default-d7d8667b7-69jgz"
Oct 31 01:40:48.752879 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:40:48.752827    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/710b2ea2-bffd-45b2-b0fc-0e5a90defed1-credential-socket\") pod \"e2e-ipp-b-b2d49b-openshift-default-d7d8667b7-69jgz\" (UID: \"710b2ea2-bffd-45b2-b0fc-0e5a90defed1\") " pod="openshift-ingress/e2e-ipp-b-b2d49b-openshift-default-d7d8667b7-69jgz"
Oct 31 01:40:48.854759 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:40:48.854668    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/710b2ea2-bffd-45b2-b0fc-0e5a90defed1-istiod-ca-cert\") pod \"e2e-ipp-b-b2d49b-openshift-default-d7d8667b7-69jgz\" (UID: \"710b2ea2-bffd-45b2-b0fc-0e5a90defed1\") " pod="openshift-ingress/e2e-ipp-b-b2d49b-openshift-default-d7d8667b7-69jgz"
Oct 31 01:40:48.854759 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:40:48.854717    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/710b2ea2-bffd-45b2-b0fc-0e5a90defed1-credential-socket\") pod \"e2e-ipp-b-b2d49b-openshift-default-d7d8667b7-69jgz\" (UID: \"710b2ea2-bffd-45b2-b0fc-0e5a90defed1\") " pod="openshift-ingress/e2e-ipp-b-b2d49b-openshift-default-d7d8667b7-69jgz"
Oct 31 01:40:48.854966 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:40:48.854891    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2mxkl\" (UniqueName: \"kubernetes.io/projected/710b2ea2-bffd-45b2-b0fc-0e5a90defed1-kube-api-access-2mxkl\") pod \"e2e-ipp-b-b2d49b-openshift-default-d7d8667b7-69jgz\" (UID: \"710b2ea2-bffd-45b2-b0fc-0e5a90defed1\") " pod="openshift-ingress/e2e-ipp-b-b2d49b-openshift-default-d7d8667b7-69jgz"
Oct 31 01:40:48.855029 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:40:48.855001    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/710b2ea2-bffd-45b2-b0fc-0e5a90defed1-workload-socket\") pod \"e2e-ipp-b-b2d49b-openshift-default-d7d8667b7-69jgz\" (UID: \"710b2ea2-bffd-45b2-b0fc-0e5a90defed1\") " pod="openshift-ingress/e2e-ipp-b-b2d49b-openshift-default-d7d8667b7-69jgz"
Oct 31 01:40:48.855029 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:40:48.855021    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/710b2ea2-bffd-45b2-b0fc-0e5a90defed1-credential-socket\") pod \"e2e-ipp-b-b2d49b-openshift-default-d7d8667b7-69jgz\" (UID: \"710b2ea2-bffd-45b2-b0fc-0e5a90defed1\") " pod="openshift-ingress/e2e-ipp-b-b2d49b-openshift-default-d7d8667b7-69jgz"
Oct 31 01:40:48.855100 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:40:48.855071    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/710b2ea2-bffd-45b2-b0fc-0e5a90defed1-workload-certs\") pod \"e2e-ipp-b-b2d49b-openshift-default-d7d8667b7-69jgz\" (UID: \"710b2ea2-bffd-45b2-b0fc-0e5a90defed1\") " pod="openshift-ingress/e2e-ipp-b-b2d49b-openshift-default-d7d8667b7-69jgz"
Oct 31 01:40:48.855158 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:40:48.855144    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/710b2ea2-bffd-45b2-b0fc-0e5a90defed1-istio-podinfo\") pod \"e2e-ipp-b-b2d49b-openshift-default-d7d8667b7-69jgz\" (UID: \"710b2ea2-bffd-45b2-b0fc-0e5a90defed1\") " pod="openshift-ingress/e2e-ipp-b-b2d49b-openshift-default-d7d8667b7-69jgz"
Oct 31 01:40:48.855195 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:40:48.855172    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/710b2ea2-bffd-45b2-b0fc-0e5a90defed1-istio-data\") pod \"e2e-ipp-b-b2d49b-openshift-default-d7d8667b7-69jgz\" (UID: \"710b2ea2-bffd-45b2-b0fc-0e5a90defed1\") " pod="openshift-ingress/e2e-ipp-b-b2d49b-openshift-default-d7d8667b7-69jgz"
Oct 31 01:40:48.855270 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:40:48.855196    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/710b2ea2-bffd-45b2-b0fc-0e5a90defed1-istio-token\") pod \"e2e-ipp-b-b2d49b-openshift-default-d7d8667b7-69jgz\" (UID: \"710b2ea2-bffd-45b2-b0fc-0e5a90defed1\") " pod="openshift-ingress/e2e-ipp-b-b2d49b-openshift-default-d7d8667b7-69jgz"
Oct 31 01:40:48.855337 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:40:48.855303    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/710b2ea2-bffd-45b2-b0fc-0e5a90defed1-istio-envoy\") pod \"e2e-ipp-b-b2d49b-openshift-default-d7d8667b7-69jgz\" (UID: \"710b2ea2-bffd-45b2-b0fc-0e5a90defed1\") " pod="openshift-ingress/e2e-ipp-b-b2d49b-openshift-default-d7d8667b7-69jgz"
Oct 31 01:40:48.855473 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:40:48.855450    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/710b2ea2-bffd-45b2-b0fc-0e5a90defed1-istiod-ca-cert\") pod \"e2e-ipp-b-b2d49b-openshift-default-d7d8667b7-69jgz\" (UID: \"710b2ea2-bffd-45b2-b0fc-0e5a90defed1\") " pod="openshift-ingress/e2e-ipp-b-b2d49b-openshift-default-d7d8667b7-69jgz"
Oct 31 01:40:48.855572 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:40:48.855545    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/710b2ea2-bffd-45b2-b0fc-0e5a90defed1-workload-certs\") pod \"e2e-ipp-b-b2d49b-openshift-default-d7d8667b7-69jgz\" (UID: \"710b2ea2-bffd-45b2-b0fc-0e5a90defed1\") " pod="openshift-ingress/e2e-ipp-b-b2d49b-openshift-default-d7d8667b7-69jgz"
Oct 31 01:40:48.855782 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:40:48.855757    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/710b2ea2-bffd-45b2-b0fc-0e5a90defed1-istio-data\") pod \"e2e-ipp-b-b2d49b-openshift-default-d7d8667b7-69jgz\" (UID: \"710b2ea2-bffd-45b2-b0fc-0e5a90defed1\") " pod="openshift-ingress/e2e-ipp-b-b2d49b-openshift-default-d7d8667b7-69jgz"
Oct 31 01:40:48.855937 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:40:48.855875    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/710b2ea2-bffd-45b2-b0fc-0e5a90defed1-workload-socket\") pod \"e2e-ipp-b-b2d49b-openshift-default-d7d8667b7-69jgz\" (UID: \"710b2ea2-bffd-45b2-b0fc-0e5a90defed1\") " pod="openshift-ingress/e2e-ipp-b-b2d49b-openshift-default-d7d8667b7-69jgz"
Oct 31 01:40:48.858148 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:40:48.858126    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/710b2ea2-bffd-45b2-b0fc-0e5a90defed1-istio-podinfo\") pod \"e2e-ipp-b-b2d49b-openshift-default-d7d8667b7-69jgz\" (UID: \"710b2ea2-bffd-45b2-b0fc-0e5a90defed1\") " pod="openshift-ingress/e2e-ipp-b-b2d49b-openshift-default-d7d8667b7-69jgz"
Oct 31 01:40:48.858239 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:40:48.858195    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/710b2ea2-bffd-45b2-b0fc-0e5a90defed1-istio-envoy\") pod \"e2e-ipp-b-b2d49b-openshift-default-d7d8667b7-69jgz\" (UID: \"710b2ea2-bffd-45b2-b0fc-0e5a90defed1\") " pod="openshift-ingress/e2e-ipp-b-b2d49b-openshift-default-d7d8667b7-69jgz"
Oct 31 01:40:48.912212 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:40:48.912176    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2mxkl\" (UniqueName: \"kubernetes.io/projected/710b2ea2-bffd-45b2-b0fc-0e5a90defed1-kube-api-access-2mxkl\") pod \"e2e-ipp-b-b2d49b-openshift-default-d7d8667b7-69jgz\" (UID: \"710b2ea2-bffd-45b2-b0fc-0e5a90defed1\") " pod="openshift-ingress/e2e-ipp-b-b2d49b-openshift-default-d7d8667b7-69jgz"
Oct 31 01:40:48.912519 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:40:48.912494    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/710b2ea2-bffd-45b2-b0fc-0e5a90defed1-istio-token\") pod \"e2e-ipp-b-b2d49b-openshift-default-d7d8667b7-69jgz\" (UID: \"710b2ea2-bffd-45b2-b0fc-0e5a90defed1\") " pod="openshift-ingress/e2e-ipp-b-b2d49b-openshift-default-d7d8667b7-69jgz"
Oct 31 01:40:48.975499 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:40:48.975456    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:40:49.206741 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:40:49.206630    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-b2d49b-openshift-default-d7d8667b7-69jgz"
Oct 31 01:40:49.647873 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:40:49.647832    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-b2d49b-openshift-default-d7d8667b7-69jgz"]
Oct 31 01:40:49.686993 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:40:49.686964    4964 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod710b2ea2_bffd_45b2_b0fc_0e5a90defed1.slice/crio-767bc2238c8567081847853833d623827f63f6bdf23efeb09e280f03c63877ca WatchSource:0}: Error finding container 767bc2238c8567081847853833d623827f63f6bdf23efeb09e280f03c63877ca: Status 404 returned error can't find the container with id 767bc2238c8567081847853833d623827f63f6bdf23efeb09e280f03c63877ca
Oct 31 01:40:49.689575 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:40:49.689544    4964 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 01:40:49.689671 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:40:49.689608    4964 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 01:40:49.689671 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:40:49.689637    4964 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 01:40:49.821949 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:40:49.821914    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-b2d49b-openshift-default-d7d8667b7-69jgz" event={"ID":"710b2ea2-bffd-45b2-b0fc-0e5a90defed1","Type":"ContainerStarted","Data":"f974e97c15eabf759974ed24a834e0f80d2dc04c22051c94739c2369d33357fb"}
Oct 31 01:40:49.822118 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:40:49.821978    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-b2d49b-openshift-default-d7d8667b7-69jgz" event={"ID":"710b2ea2-bffd-45b2-b0fc-0e5a90defed1","Type":"ContainerStarted","Data":"767bc2238c8567081847853833d623827f63f6bdf23efeb09e280f03c63877ca"}
Oct 31 01:40:49.940676 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:40:49.940573    4964 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-b2d49b-openshift-default-d7d8667b7-69jgz" podStartSLOduration=1.940558183 podStartE2EDuration="1.940558183s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:40:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:40:49.937700693 +0000 UTC m=+2218.414267073" watchObservedRunningTime="2026-10-31 01:40:49.940558183 +0000 UTC m=+2218.417124564"
Oct 31 01:40:50.207175 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:40:50.207090    4964 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-b2d49b-openshift-default-d7d8667b7-69jgz"
Oct 31 01:40:50.212964 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:40:50.212940    4964 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-b2d49b-openshift-default-d7d8667b7-69jgz"
Oct 31 01:40:50.312709 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:40:50.312676    4964 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-8af859-openshift-default-59576db9cf-6mc8p"]
Oct 31 01:40:50.316917 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:40:50.316900    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-8af859-openshift-default-59576db9cf-6mc8p"
Oct 31 01:40:50.368600 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:40:50.368568    4964 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-8af859-openshift-default-dockercfg-2ft7r\""
Oct 31 01:40:50.374092 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:40:50.374065    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/940b399a-de68-4440-a629-df5ec46dc796-credential-socket\") pod \"e2e-ipp-b-8af859-openshift-default-59576db9cf-6mc8p\" (UID: \"940b399a-de68-4440-a629-df5ec46dc796\") " pod="openshift-ingress/e2e-ipp-b-8af859-openshift-default-59576db9cf-6mc8p"
Oct 31 01:40:50.374243 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:40:50.374133    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/940b399a-de68-4440-a629-df5ec46dc796-workload-socket\") pod \"e2e-ipp-b-8af859-openshift-default-59576db9cf-6mc8p\" (UID: \"940b399a-de68-4440-a629-df5ec46dc796\") " pod="openshift-ingress/e2e-ipp-b-8af859-openshift-default-59576db9cf-6mc8p"
Oct 31 01:40:50.374333 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:40:50.374253    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/940b399a-de68-4440-a629-df5ec46dc796-istiod-ca-cert\") pod \"e2e-ipp-b-8af859-openshift-default-59576db9cf-6mc8p\" (UID: \"940b399a-de68-4440-a629-df5ec46dc796\") " pod="openshift-ingress/e2e-ipp-b-8af859-openshift-default-59576db9cf-6mc8p"
Oct 31 01:40:50.374406 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:40:50.374359    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/940b399a-de68-4440-a629-df5ec46dc796-istio-token\") pod \"e2e-ipp-b-8af859-openshift-default-59576db9cf-6mc8p\" (UID: \"940b399a-de68-4440-a629-df5ec46dc796\") " pod="openshift-ingress/e2e-ipp-b-8af859-openshift-default-59576db9cf-6mc8p"
Oct 31 01:40:50.374452 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:40:50.374400    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gk2bl\" (UniqueName: \"kubernetes.io/projected/940b399a-de68-4440-a629-df5ec46dc796-kube-api-access-gk2bl\") pod \"e2e-ipp-b-8af859-openshift-default-59576db9cf-6mc8p\" (UID: \"940b399a-de68-4440-a629-df5ec46dc796\") " pod="openshift-ingress/e2e-ipp-b-8af859-openshift-default-59576db9cf-6mc8p"
Oct 31 01:40:50.374490 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:40:50.374450    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/940b399a-de68-4440-a629-df5ec46dc796-workload-certs\") pod \"e2e-ipp-b-8af859-openshift-default-59576db9cf-6mc8p\" (UID: \"940b399a-de68-4440-a629-df5ec46dc796\") " pod="openshift-ingress/e2e-ipp-b-8af859-openshift-default-59576db9cf-6mc8p"
Oct 31 01:40:50.374490 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:40:50.374475    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/940b399a-de68-4440-a629-df5ec46dc796-istio-envoy\") pod \"e2e-ipp-b-8af859-openshift-default-59576db9cf-6mc8p\" (UID: \"940b399a-de68-4440-a629-df5ec46dc796\") " pod="openshift-ingress/e2e-ipp-b-8af859-openshift-default-59576db9cf-6mc8p"
Oct 31 01:40:50.374610 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:40:50.374596    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/940b399a-de68-4440-a629-df5ec46dc796-istio-data\") pod \"e2e-ipp-b-8af859-openshift-default-59576db9cf-6mc8p\" (UID: \"940b399a-de68-4440-a629-df5ec46dc796\") " pod="openshift-ingress/e2e-ipp-b-8af859-openshift-default-59576db9cf-6mc8p"
Oct 31 01:40:50.374674 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:40:50.374658    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/940b399a-de68-4440-a629-df5ec46dc796-istio-podinfo\") pod \"e2e-ipp-b-8af859-openshift-default-59576db9cf-6mc8p\" (UID: \"940b399a-de68-4440-a629-df5ec46dc796\") " pod="openshift-ingress/e2e-ipp-b-8af859-openshift-default-59576db9cf-6mc8p"
Oct 31 01:40:50.418910 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:40:50.418873    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-8af859-openshift-default-59576db9cf-6mc8p"]
Oct 31 01:40:50.476383 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:40:50.476347    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/940b399a-de68-4440-a629-df5ec46dc796-istiod-ca-cert\") pod \"e2e-ipp-b-8af859-openshift-default-59576db9cf-6mc8p\" (UID: \"940b399a-de68-4440-a629-df5ec46dc796\") " pod="openshift-ingress/e2e-ipp-b-8af859-openshift-default-59576db9cf-6mc8p"
Oct 31 01:40:50.476561 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:40:50.476410    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/940b399a-de68-4440-a629-df5ec46dc796-istio-token\") pod \"e2e-ipp-b-8af859-openshift-default-59576db9cf-6mc8p\" (UID: \"940b399a-de68-4440-a629-df5ec46dc796\") " pod="openshift-ingress/e2e-ipp-b-8af859-openshift-default-59576db9cf-6mc8p"
Oct 31 01:40:50.476561 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:40:50.476450    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gk2bl\" (UniqueName: \"kubernetes.io/projected/940b399a-de68-4440-a629-df5ec46dc796-kube-api-access-gk2bl\") pod \"e2e-ipp-b-8af859-openshift-default-59576db9cf-6mc8p\" (UID: \"940b399a-de68-4440-a629-df5ec46dc796\") " pod="openshift-ingress/e2e-ipp-b-8af859-openshift-default-59576db9cf-6mc8p"
Oct 31 01:40:50.476561 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:40:50.476478    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/940b399a-de68-4440-a629-df5ec46dc796-workload-certs\") pod \"e2e-ipp-b-8af859-openshift-default-59576db9cf-6mc8p\" (UID: \"940b399a-de68-4440-a629-df5ec46dc796\") " pod="openshift-ingress/e2e-ipp-b-8af859-openshift-default-59576db9cf-6mc8p"
Oct 31 01:40:50.476561 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:40:50.476498    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/940b399a-de68-4440-a629-df5ec46dc796-istio-envoy\") pod \"e2e-ipp-b-8af859-openshift-default-59576db9cf-6mc8p\" (UID: \"940b399a-de68-4440-a629-df5ec46dc796\") " pod="openshift-ingress/e2e-ipp-b-8af859-openshift-default-59576db9cf-6mc8p"
Oct 31 01:40:50.476803 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:40:50.476657    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/940b399a-de68-4440-a629-df5ec46dc796-istio-data\") pod \"e2e-ipp-b-8af859-openshift-default-59576db9cf-6mc8p\" (UID: \"940b399a-de68-4440-a629-df5ec46dc796\") " pod="openshift-ingress/e2e-ipp-b-8af859-openshift-default-59576db9cf-6mc8p"
Oct 31 01:40:50.476803 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:40:50.476715    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/940b399a-de68-4440-a629-df5ec46dc796-istio-podinfo\") pod \"e2e-ipp-b-8af859-openshift-default-59576db9cf-6mc8p\" (UID: \"940b399a-de68-4440-a629-df5ec46dc796\") " pod="openshift-ingress/e2e-ipp-b-8af859-openshift-default-59576db9cf-6mc8p"
Oct 31 01:40:50.476924 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:40:50.476818    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/940b399a-de68-4440-a629-df5ec46dc796-credential-socket\") pod \"e2e-ipp-b-8af859-openshift-default-59576db9cf-6mc8p\" (UID: \"940b399a-de68-4440-a629-df5ec46dc796\") " pod="openshift-ingress/e2e-ipp-b-8af859-openshift-default-59576db9cf-6mc8p"
Oct 31 01:40:50.476924 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:40:50.476885    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/940b399a-de68-4440-a629-df5ec46dc796-workload-certs\") pod \"e2e-ipp-b-8af859-openshift-default-59576db9cf-6mc8p\" (UID: \"940b399a-de68-4440-a629-df5ec46dc796\") " pod="openshift-ingress/e2e-ipp-b-8af859-openshift-default-59576db9cf-6mc8p"
Oct 31 01:40:50.477027 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:40:50.476937    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/940b399a-de68-4440-a629-df5ec46dc796-workload-socket\") pod \"e2e-ipp-b-8af859-openshift-default-59576db9cf-6mc8p\" (UID: \"940b399a-de68-4440-a629-df5ec46dc796\") " pod="openshift-ingress/e2e-ipp-b-8af859-openshift-default-59576db9cf-6mc8p"
Oct 31 01:40:50.477088 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:40:50.477022    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/940b399a-de68-4440-a629-df5ec46dc796-istio-data\") pod \"e2e-ipp-b-8af859-openshift-default-59576db9cf-6mc8p\" (UID: \"940b399a-de68-4440-a629-df5ec46dc796\") " pod="openshift-ingress/e2e-ipp-b-8af859-openshift-default-59576db9cf-6mc8p"
Oct 31 01:40:50.477088 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:40:50.477050    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/940b399a-de68-4440-a629-df5ec46dc796-istiod-ca-cert\") pod \"e2e-ipp-b-8af859-openshift-default-59576db9cf-6mc8p\" (UID: \"940b399a-de68-4440-a629-df5ec46dc796\") " pod="openshift-ingress/e2e-ipp-b-8af859-openshift-default-59576db9cf-6mc8p"
Oct 31 01:40:50.477209 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:40:50.477177    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/940b399a-de68-4440-a629-df5ec46dc796-credential-socket\") pod \"e2e-ipp-b-8af859-openshift-default-59576db9cf-6mc8p\" (UID: \"940b399a-de68-4440-a629-df5ec46dc796\") " pod="openshift-ingress/e2e-ipp-b-8af859-openshift-default-59576db9cf-6mc8p"
Oct 31 01:40:50.477515 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:40:50.477489    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/940b399a-de68-4440-a629-df5ec46dc796-workload-socket\") pod \"e2e-ipp-b-8af859-openshift-default-59576db9cf-6mc8p\" (UID: \"940b399a-de68-4440-a629-df5ec46dc796\") " pod="openshift-ingress/e2e-ipp-b-8af859-openshift-default-59576db9cf-6mc8p"
Oct 31 01:40:50.478944 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:40:50.478924    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/940b399a-de68-4440-a629-df5ec46dc796-istio-envoy\") pod \"e2e-ipp-b-8af859-openshift-default-59576db9cf-6mc8p\" (UID: \"940b399a-de68-4440-a629-df5ec46dc796\") " pod="openshift-ingress/e2e-ipp-b-8af859-openshift-default-59576db9cf-6mc8p"
Oct 31 01:40:50.479390 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:40:50.479371    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/940b399a-de68-4440-a629-df5ec46dc796-istio-podinfo\") pod \"e2e-ipp-b-8af859-openshift-default-59576db9cf-6mc8p\" (UID: \"940b399a-de68-4440-a629-df5ec46dc796\") " pod="openshift-ingress/e2e-ipp-b-8af859-openshift-default-59576db9cf-6mc8p"
Oct 31 01:40:50.543365 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:40:50.538459    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gk2bl\" (UniqueName: \"kubernetes.io/projected/940b399a-de68-4440-a629-df5ec46dc796-kube-api-access-gk2bl\") pod \"e2e-ipp-b-8af859-openshift-default-59576db9cf-6mc8p\" (UID: \"940b399a-de68-4440-a629-df5ec46dc796\") " pod="openshift-ingress/e2e-ipp-b-8af859-openshift-default-59576db9cf-6mc8p"
Oct 31 01:40:50.543365 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:40:50.539000    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/940b399a-de68-4440-a629-df5ec46dc796-istio-token\") pod \"e2e-ipp-b-8af859-openshift-default-59576db9cf-6mc8p\" (UID: \"940b399a-de68-4440-a629-df5ec46dc796\") " pod="openshift-ingress/e2e-ipp-b-8af859-openshift-default-59576db9cf-6mc8p"
Oct 31 01:40:50.632127 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:40:50.632095    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-8af859-openshift-default-59576db9cf-6mc8p"
Oct 31 01:40:50.828733 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:40:50.828643    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-b2d49b-openshift-default-d7d8667b7-69jgz"
Oct 31 01:40:50.829756 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:40:50.829734    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-b2d49b-openshift-default-d7d8667b7-69jgz"
Oct 31 01:40:51.071667 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:40:51.071637    4964 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940b399a_de68_4440_a629_df5ec46dc796.slice/crio-a967229cff894f28998520d8979bc953d3ad15fc4ef579bbb0fa77e6c205b545 WatchSource:0}: Error finding container a967229cff894f28998520d8979bc953d3ad15fc4ef579bbb0fa77e6c205b545: Status 404 returned error can't find the container with id a967229cff894f28998520d8979bc953d3ad15fc4ef579bbb0fa77e6c205b545
Oct 31 01:40:51.072327 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:40:51.072281    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-8af859-openshift-default-59576db9cf-6mc8p"]
Oct 31 01:40:51.074880 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:40:51.074843    4964 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 01:40:51.074972 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:40:51.074926    4964 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 01:40:51.075033 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:40:51.074971    4964 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 01:40:51.579363 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:40:51.579207    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:40:51.838003 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:40:51.837899    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-8af859-openshift-default-59576db9cf-6mc8p" event={"ID":"940b399a-de68-4440-a629-df5ec46dc796","Type":"ContainerStarted","Data":"f0641e882dcbb25e307f2b537f6e081248b46cedf5b4439852ec3519bddd3642"}
Oct 31 01:40:51.838003 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:40:51.837975    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-8af859-openshift-default-59576db9cf-6mc8p" event={"ID":"940b399a-de68-4440-a629-df5ec46dc796","Type":"ContainerStarted","Data":"a967229cff894f28998520d8979bc953d3ad15fc4ef579bbb0fa77e6c205b545"}
Oct 31 01:40:51.960081 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:40:51.960019    4964 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-8af859-openshift-default-59576db9cf-6mc8p" podStartSLOduration=1.960003583 podStartE2EDuration="1.960003583s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:40:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:40:51.959098161 +0000 UTC m=+2220.435664573" watchObservedRunningTime="2026-10-31 01:40:51.960003583 +0000 UTC m=+2220.436569969"
Oct 31 01:40:52.632310 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:40:52.632272    4964 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-8af859-openshift-default-59576db9cf-6mc8p"
Oct 31 01:40:52.634025 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:40:52.633994    4964 patch_prober.go:29] interesting pod/e2e-ipp-b-8af859-openshift-default-59576db9cf-6mc8p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.138:15021/healthz/ready\": dial tcp 10.132.0.138:15021: connect: connection refused" start-of-body=
Oct 31 01:40:52.634169 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:40:52.634047    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-8af859-openshift-default-59576db9cf-6mc8p" podUID="940b399a-de68-4440-a629-df5ec46dc796" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.138:15021/healthz/ready\": dial tcp 10.132.0.138:15021: connect: connection refused"
Oct 31 01:40:53.632924 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:40:53.632888    4964 patch_prober.go:29] interesting pod/e2e-ipp-b-8af859-openshift-default-59576db9cf-6mc8p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.138:15021/healthz/ready\": dial tcp 10.132.0.138:15021: connect: connection refused" start-of-body=
Oct 31 01:40:53.633447 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:40:53.632955    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-8af859-openshift-default-59576db9cf-6mc8p" podUID="940b399a-de68-4440-a629-df5ec46dc796" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.138:15021/healthz/ready\": dial tcp 10.132.0.138:15021: connect: connection refused"
Oct 31 01:40:54.632676 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:40:54.632644    4964 patch_prober.go:29] interesting pod/e2e-ipp-b-8af859-openshift-default-59576db9cf-6mc8p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.138:15021/healthz/ready\": dial tcp 10.132.0.138:15021: connect: connection refused" start-of-body=
Oct 31 01:40:54.632869 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:40:54.632709    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-8af859-openshift-default-59576db9cf-6mc8p" podUID="940b399a-de68-4440-a629-df5ec46dc796" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.138:15021/healthz/ready\": dial tcp 10.132.0.138:15021: connect: connection refused"
Oct 31 01:40:55.633345 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:40:55.633295    4964 patch_prober.go:29] interesting pod/e2e-ipp-b-8af859-openshift-default-59576db9cf-6mc8p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.138:15021/healthz/ready\": dial tcp 10.132.0.138:15021: connect: connection refused" start-of-body=
Oct 31 01:40:55.633829 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:40:55.633375    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-8af859-openshift-default-59576db9cf-6mc8p" podUID="940b399a-de68-4440-a629-df5ec46dc796" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.138:15021/healthz/ready\": dial tcp 10.132.0.138:15021: connect: connection refused"
Oct 31 01:40:56.450952 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:40:56.450914    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:40:56.633371 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:40:56.633310    4964 patch_prober.go:29] interesting pod/e2e-ipp-b-8af859-openshift-default-59576db9cf-6mc8p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.138:15021/healthz/ready\": dial tcp 10.132.0.138:15021: connect: connection refused" start-of-body=
Oct 31 01:40:56.633783 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:40:56.633409    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-8af859-openshift-default-59576db9cf-6mc8p" podUID="940b399a-de68-4440-a629-df5ec46dc796" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.138:15021/healthz/ready\": dial tcp 10.132.0.138:15021: connect: connection refused"
Oct 31 01:40:57.637759 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:40:57.637721    4964 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-8af859-openshift-default-59576db9cf-6mc8p"
Oct 31 01:40:57.638236 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:40:57.638106    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-8af859-openshift-default-59576db9cf-6mc8p"
Oct 31 01:40:57.638766 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:40:57.638746    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-8af859-openshift-default-59576db9cf-6mc8p"
Oct 31 01:41:01.443351 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:41:01.443277    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:41:02.409060 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:41:02.409017    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:41:03.077799 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:41:03.077762    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:41:03.268788 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:41:03.268759    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:41:03.442905 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:41:03.442809    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:41:05.209123 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:41:05.209090    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:41:05.732839 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:41:05.732803    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:41:06.318875 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:41:06.318841    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:41:06.500437 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:41:06.500397    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:41:06.847350 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:41:06.844190    4964 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-b2d49b-b7b76c997-5vm2r"]
Oct 31 01:41:06.873137 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:41:06.873110    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-b2d49b-b7b76c997-5vm2r"
Oct 31 01:41:06.901692 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:41:06.901656    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-b2d49b-b7b76c997-5vm2r"]
Oct 31 01:41:06.946391 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:41:06.946356    4964 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-b2d49b\""
Oct 31 01:41:06.946547 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:41:06.946399    4964 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-b2d49b-dockercfg-pcrv8\""
Oct 31 01:41:07.044717 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:41:07.044684    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4gwdx\" (UniqueName: \"kubernetes.io/projected/0cc66d3e-3edd-4399-a970-5ab7d0f6f415-kube-api-access-4gwdx\") pod \"payload-processing-e2e-ipp-b-b2d49b-b7b76c997-5vm2r\" (UID: \"0cc66d3e-3edd-4399-a970-5ab7d0f6f415\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-b2d49b-b7b76c997-5vm2r"
Oct 31 01:41:07.044904 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:41:07.044795    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0cc66d3e-3edd-4399-a970-5ab7d0f6f415-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-b2d49b-b7b76c997-5vm2r\" (UID: \"0cc66d3e-3edd-4399-a970-5ab7d0f6f415\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-b2d49b-b7b76c997-5vm2r"
Oct 31 01:41:07.146423 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:41:07.146338    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4gwdx\" (UniqueName: \"kubernetes.io/projected/0cc66d3e-3edd-4399-a970-5ab7d0f6f415-kube-api-access-4gwdx\") pod \"payload-processing-e2e-ipp-b-b2d49b-b7b76c997-5vm2r\" (UID: \"0cc66d3e-3edd-4399-a970-5ab7d0f6f415\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-b2d49b-b7b76c997-5vm2r"
Oct 31 01:41:07.146772 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:41:07.146745    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0cc66d3e-3edd-4399-a970-5ab7d0f6f415-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-b2d49b-b7b76c997-5vm2r\" (UID: \"0cc66d3e-3edd-4399-a970-5ab7d0f6f415\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-b2d49b-b7b76c997-5vm2r"
Oct 31 01:41:07.147792 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:41:07.147743    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0cc66d3e-3edd-4399-a970-5ab7d0f6f415-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-b2d49b-b7b76c997-5vm2r\" (UID: \"0cc66d3e-3edd-4399-a970-5ab7d0f6f415\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-b2d49b-b7b76c997-5vm2r"
Oct 31 01:41:07.204208 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:41:07.204175    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4gwdx\" (UniqueName: \"kubernetes.io/projected/0cc66d3e-3edd-4399-a970-5ab7d0f6f415-kube-api-access-4gwdx\") pod \"payload-processing-e2e-ipp-b-b2d49b-b7b76c997-5vm2r\" (UID: \"0cc66d3e-3edd-4399-a970-5ab7d0f6f415\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-b2d49b-b7b76c997-5vm2r"
Oct 31 01:41:07.491503 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:41:07.491472    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-b2d49b-b7b76c997-5vm2r"
Oct 31 01:41:07.868082 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:41:07.868059    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-b2d49b-b7b76c997-5vm2r"]
Oct 31 01:41:07.869009 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:41:07.868983    4964 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0cc66d3e_3edd_4399_a970_5ab7d0f6f415.slice/crio-29a4442d76a7bedac51dc99b65d332f4179be0a417dc76d44c7416ddfd2a0ac3 WatchSource:0}: Error finding container 29a4442d76a7bedac51dc99b65d332f4179be0a417dc76d44c7416ddfd2a0ac3: Status 404 returned error can't find the container with id 29a4442d76a7bedac51dc99b65d332f4179be0a417dc76d44c7416ddfd2a0ac3
Oct 31 01:41:07.974633 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:41:07.974604    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-b2d49b-b7b76c997-5vm2r" event={"ID":"0cc66d3e-3edd-4399-a970-5ab7d0f6f415","Type":"ContainerStarted","Data":"175a113a54711c378c346221f4d7d2b2f0ce109832211f2a402da22a3ee2850a"}
Oct 31 01:41:07.974785 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:41:07.974667    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-b2d49b-b7b76c997-5vm2r"
Oct 31 01:41:07.974785 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:41:07.974683    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-b2d49b-b7b76c997-5vm2r" event={"ID":"0cc66d3e-3edd-4399-a970-5ab7d0f6f415","Type":"ContainerStarted","Data":"29a4442d76a7bedac51dc99b65d332f4179be0a417dc76d44c7416ddfd2a0ac3"}
Oct 31 01:41:08.091448 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:41:08.091387    4964 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-b2d49b-b7b76c997-5vm2r" podStartSLOduration=2.09137275 podStartE2EDuration="2.09137275s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:41:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:41:08.088121953 +0000 UTC m=+2236.564688332" watchObservedRunningTime="2026-10-31 01:41:08.09137275 +0000 UTC m=+2236.567939130"
Oct 31 01:41:08.908398 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:41:08.908366    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:41:11.055537 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:41:11.055503    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:41:11.239529 ip-10-0-133-197 kubenswrapper[4964]: E1031 01:41:11.239497    4964 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q56x5" containerName="extractor"
Oct 31 01:41:13.367774 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:41:13.367739    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:41:15.604714 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:41:15.604682    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:41:18.987037 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:41:18.987006    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-b2d49b-b7b76c997-5vm2r"
Oct 31 01:41:20.407145 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:41:20.407105    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:41:25.523878 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:41:25.523839    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:41:26.363899 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:41:26.363866    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:41:26.634764 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:41:26.634661    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:41:26.833081 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:41:26.833048    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:41:30.469122 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:41:30.468595    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:41:35.228026 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:41:35.227990    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:41:35.668796 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:41:35.668711    4964 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-8af859-openshift-default-59576db9cf-cnbq6"]
Oct 31 01:41:35.674238 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:41:35.674219    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-8af859-openshift-default-59576db9cf-cnbq6"
Oct 31 01:41:35.764032 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:41:35.763993    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/503330d0-a5f9-470b-8fd6-3ad2e4cf4698-istio-podinfo\") pod \"e2e-ipp-b-8af859-openshift-default-59576db9cf-cnbq6\" (UID: \"503330d0-a5f9-470b-8fd6-3ad2e4cf4698\") " pod="openshift-ingress/e2e-ipp-b-8af859-openshift-default-59576db9cf-cnbq6"
Oct 31 01:41:35.764205 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:41:35.764037    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/503330d0-a5f9-470b-8fd6-3ad2e4cf4698-istio-data\") pod \"e2e-ipp-b-8af859-openshift-default-59576db9cf-cnbq6\" (UID: \"503330d0-a5f9-470b-8fd6-3ad2e4cf4698\") " pod="openshift-ingress/e2e-ipp-b-8af859-openshift-default-59576db9cf-cnbq6"
Oct 31 01:41:35.764205 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:41:35.764159    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/503330d0-a5f9-470b-8fd6-3ad2e4cf4698-workload-certs\") pod \"e2e-ipp-b-8af859-openshift-default-59576db9cf-cnbq6\" (UID: \"503330d0-a5f9-470b-8fd6-3ad2e4cf4698\") " pod="openshift-ingress/e2e-ipp-b-8af859-openshift-default-59576db9cf-cnbq6"
Oct 31 01:41:35.764285 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:41:35.764224    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/503330d0-a5f9-470b-8fd6-3ad2e4cf4698-istiod-ca-cert\") pod \"e2e-ipp-b-8af859-openshift-default-59576db9cf-cnbq6\" (UID: \"503330d0-a5f9-470b-8fd6-3ad2e4cf4698\") " pod="openshift-ingress/e2e-ipp-b-8af859-openshift-default-59576db9cf-cnbq6"
Oct 31 01:41:35.764285 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:41:35.764278    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jwl4b\" (UniqueName: \"kubernetes.io/projected/503330d0-a5f9-470b-8fd6-3ad2e4cf4698-kube-api-access-jwl4b\") pod \"e2e-ipp-b-8af859-openshift-default-59576db9cf-cnbq6\" (UID: \"503330d0-a5f9-470b-8fd6-3ad2e4cf4698\") " pod="openshift-ingress/e2e-ipp-b-8af859-openshift-default-59576db9cf-cnbq6"
Oct 31 01:41:35.764426 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:41:35.764403    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/503330d0-a5f9-470b-8fd6-3ad2e4cf4698-istio-envoy\") pod \"e2e-ipp-b-8af859-openshift-default-59576db9cf-cnbq6\" (UID: \"503330d0-a5f9-470b-8fd6-3ad2e4cf4698\") " pod="openshift-ingress/e2e-ipp-b-8af859-openshift-default-59576db9cf-cnbq6"
Oct 31 01:41:35.764485 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:41:35.764473    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/503330d0-a5f9-470b-8fd6-3ad2e4cf4698-istio-token\") pod \"e2e-ipp-b-8af859-openshift-default-59576db9cf-cnbq6\" (UID: \"503330d0-a5f9-470b-8fd6-3ad2e4cf4698\") " pod="openshift-ingress/e2e-ipp-b-8af859-openshift-default-59576db9cf-cnbq6"
Oct 31 01:41:35.764585 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:41:35.764563    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/503330d0-a5f9-470b-8fd6-3ad2e4cf4698-credential-socket\") pod \"e2e-ipp-b-8af859-openshift-default-59576db9cf-cnbq6\" (UID: \"503330d0-a5f9-470b-8fd6-3ad2e4cf4698\") " pod="openshift-ingress/e2e-ipp-b-8af859-openshift-default-59576db9cf-cnbq6"
Oct 31 01:41:35.764651 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:41:35.764607    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/503330d0-a5f9-470b-8fd6-3ad2e4cf4698-workload-socket\") pod \"e2e-ipp-b-8af859-openshift-default-59576db9cf-cnbq6\" (UID: \"503330d0-a5f9-470b-8fd6-3ad2e4cf4698\") " pod="openshift-ingress/e2e-ipp-b-8af859-openshift-default-59576db9cf-cnbq6"
Oct 31 01:41:35.781075 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:41:35.781044    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-8af859-openshift-default-59576db9cf-cnbq6"]
Oct 31 01:41:35.865882 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:41:35.865843    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/503330d0-a5f9-470b-8fd6-3ad2e4cf4698-workload-certs\") pod \"e2e-ipp-b-8af859-openshift-default-59576db9cf-cnbq6\" (UID: \"503330d0-a5f9-470b-8fd6-3ad2e4cf4698\") " pod="openshift-ingress/e2e-ipp-b-8af859-openshift-default-59576db9cf-cnbq6"
Oct 31 01:41:35.866085 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:41:35.865932    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/503330d0-a5f9-470b-8fd6-3ad2e4cf4698-istiod-ca-cert\") pod \"e2e-ipp-b-8af859-openshift-default-59576db9cf-cnbq6\" (UID: \"503330d0-a5f9-470b-8fd6-3ad2e4cf4698\") " pod="openshift-ingress/e2e-ipp-b-8af859-openshift-default-59576db9cf-cnbq6"
Oct 31 01:41:35.866163 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:41:35.866113    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jwl4b\" (UniqueName: \"kubernetes.io/projected/503330d0-a5f9-470b-8fd6-3ad2e4cf4698-kube-api-access-jwl4b\") pod \"e2e-ipp-b-8af859-openshift-default-59576db9cf-cnbq6\" (UID: \"503330d0-a5f9-470b-8fd6-3ad2e4cf4698\") " pod="openshift-ingress/e2e-ipp-b-8af859-openshift-default-59576db9cf-cnbq6"
Oct 31 01:41:35.866228 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:41:35.866199    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/503330d0-a5f9-470b-8fd6-3ad2e4cf4698-istio-envoy\") pod \"e2e-ipp-b-8af859-openshift-default-59576db9cf-cnbq6\" (UID: \"503330d0-a5f9-470b-8fd6-3ad2e4cf4698\") " pod="openshift-ingress/e2e-ipp-b-8af859-openshift-default-59576db9cf-cnbq6"
Oct 31 01:41:35.866284 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:41:35.866261    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/503330d0-a5f9-470b-8fd6-3ad2e4cf4698-workload-certs\") pod \"e2e-ipp-b-8af859-openshift-default-59576db9cf-cnbq6\" (UID: \"503330d0-a5f9-470b-8fd6-3ad2e4cf4698\") " pod="openshift-ingress/e2e-ipp-b-8af859-openshift-default-59576db9cf-cnbq6"
Oct 31 01:41:35.866367 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:41:35.866355    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/503330d0-a5f9-470b-8fd6-3ad2e4cf4698-istio-token\") pod \"e2e-ipp-b-8af859-openshift-default-59576db9cf-cnbq6\" (UID: \"503330d0-a5f9-470b-8fd6-3ad2e4cf4698\") " pod="openshift-ingress/e2e-ipp-b-8af859-openshift-default-59576db9cf-cnbq6"
Oct 31 01:41:35.866430 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:41:35.866406    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/503330d0-a5f9-470b-8fd6-3ad2e4cf4698-credential-socket\") pod \"e2e-ipp-b-8af859-openshift-default-59576db9cf-cnbq6\" (UID: \"503330d0-a5f9-470b-8fd6-3ad2e4cf4698\") " pod="openshift-ingress/e2e-ipp-b-8af859-openshift-default-59576db9cf-cnbq6"
Oct 31 01:41:35.866497 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:41:35.866445    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/503330d0-a5f9-470b-8fd6-3ad2e4cf4698-workload-socket\") pod \"e2e-ipp-b-8af859-openshift-default-59576db9cf-cnbq6\" (UID: \"503330d0-a5f9-470b-8fd6-3ad2e4cf4698\") " pod="openshift-ingress/e2e-ipp-b-8af859-openshift-default-59576db9cf-cnbq6"
Oct 31 01:41:35.866560 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:41:35.866529    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/503330d0-a5f9-470b-8fd6-3ad2e4cf4698-istio-podinfo\") pod \"e2e-ipp-b-8af859-openshift-default-59576db9cf-cnbq6\" (UID: \"503330d0-a5f9-470b-8fd6-3ad2e4cf4698\") " pod="openshift-ingress/e2e-ipp-b-8af859-openshift-default-59576db9cf-cnbq6"
Oct 31 01:41:35.866620 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:41:35.866571    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/503330d0-a5f9-470b-8fd6-3ad2e4cf4698-istio-data\") pod \"e2e-ipp-b-8af859-openshift-default-59576db9cf-cnbq6\" (UID: \"503330d0-a5f9-470b-8fd6-3ad2e4cf4698\") " pod="openshift-ingress/e2e-ipp-b-8af859-openshift-default-59576db9cf-cnbq6"
Oct 31 01:41:35.866928 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:41:35.866905    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/503330d0-a5f9-470b-8fd6-3ad2e4cf4698-istio-data\") pod \"e2e-ipp-b-8af859-openshift-default-59576db9cf-cnbq6\" (UID: \"503330d0-a5f9-470b-8fd6-3ad2e4cf4698\") " pod="openshift-ingress/e2e-ipp-b-8af859-openshift-default-59576db9cf-cnbq6"
Oct 31 01:41:35.867099 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:41:35.866944    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/503330d0-a5f9-470b-8fd6-3ad2e4cf4698-workload-socket\") pod \"e2e-ipp-b-8af859-openshift-default-59576db9cf-cnbq6\" (UID: \"503330d0-a5f9-470b-8fd6-3ad2e4cf4698\") " pod="openshift-ingress/e2e-ipp-b-8af859-openshift-default-59576db9cf-cnbq6"
Oct 31 01:41:35.867204 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:41:35.867007    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/503330d0-a5f9-470b-8fd6-3ad2e4cf4698-istiod-ca-cert\") pod \"e2e-ipp-b-8af859-openshift-default-59576db9cf-cnbq6\" (UID: \"503330d0-a5f9-470b-8fd6-3ad2e4cf4698\") " pod="openshift-ingress/e2e-ipp-b-8af859-openshift-default-59576db9cf-cnbq6"
Oct 31 01:41:35.867204 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:41:35.867107    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/503330d0-a5f9-470b-8fd6-3ad2e4cf4698-credential-socket\") pod \"e2e-ipp-b-8af859-openshift-default-59576db9cf-cnbq6\" (UID: \"503330d0-a5f9-470b-8fd6-3ad2e4cf4698\") " pod="openshift-ingress/e2e-ipp-b-8af859-openshift-default-59576db9cf-cnbq6"
Oct 31 01:41:35.868701 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:41:35.868681    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/503330d0-a5f9-470b-8fd6-3ad2e4cf4698-istio-envoy\") pod \"e2e-ipp-b-8af859-openshift-default-59576db9cf-cnbq6\" (UID: \"503330d0-a5f9-470b-8fd6-3ad2e4cf4698\") " pod="openshift-ingress/e2e-ipp-b-8af859-openshift-default-59576db9cf-cnbq6"
Oct 31 01:41:35.869533 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:41:35.869514    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/503330d0-a5f9-470b-8fd6-3ad2e4cf4698-istio-podinfo\") pod \"e2e-ipp-b-8af859-openshift-default-59576db9cf-cnbq6\" (UID: \"503330d0-a5f9-470b-8fd6-3ad2e4cf4698\") " pod="openshift-ingress/e2e-ipp-b-8af859-openshift-default-59576db9cf-cnbq6"
Oct 31 01:41:35.923012 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:41:35.922936    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jwl4b\" (UniqueName: \"kubernetes.io/projected/503330d0-a5f9-470b-8fd6-3ad2e4cf4698-kube-api-access-jwl4b\") pod \"e2e-ipp-b-8af859-openshift-default-59576db9cf-cnbq6\" (UID: \"503330d0-a5f9-470b-8fd6-3ad2e4cf4698\") " pod="openshift-ingress/e2e-ipp-b-8af859-openshift-default-59576db9cf-cnbq6"
Oct 31 01:41:35.923012 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:41:35.922940    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/503330d0-a5f9-470b-8fd6-3ad2e4cf4698-istio-token\") pod \"e2e-ipp-b-8af859-openshift-default-59576db9cf-cnbq6\" (UID: \"503330d0-a5f9-470b-8fd6-3ad2e4cf4698\") " pod="openshift-ingress/e2e-ipp-b-8af859-openshift-default-59576db9cf-cnbq6"
Oct 31 01:41:35.992205 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:41:35.992168    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-8af859-openshift-default-59576db9cf-cnbq6"
Oct 31 01:41:36.427481 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:41:36.427456    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-8af859-openshift-default-59576db9cf-cnbq6"]
Oct 31 01:41:36.429198 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:41:36.429173    4964 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod503330d0_a5f9_470b_8fd6_3ad2e4cf4698.slice/crio-dce1a7a834697b4311c1447c80e88dd00fe5131ae86329c3ae85e6df1db415e4 WatchSource:0}: Error finding container dce1a7a834697b4311c1447c80e88dd00fe5131ae86329c3ae85e6df1db415e4: Status 404 returned error can't find the container with id dce1a7a834697b4311c1447c80e88dd00fe5131ae86329c3ae85e6df1db415e4
Oct 31 01:41:36.432654 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:41:36.432620    4964 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 01:41:36.432721 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:41:36.432702    4964 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 01:41:36.432760 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:41:36.432746    4964 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 01:41:37.188772 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:41:37.188734    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-8af859-openshift-default-59576db9cf-cnbq6" event={"ID":"503330d0-a5f9-470b-8fd6-3ad2e4cf4698","Type":"ContainerStarted","Data":"c05faf4dfa15809705f5312ca188b291bfb39dbe3e0d6722c252ad6fd4bef3b6"}
Oct 31 01:41:37.188969 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:41:37.188806    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-8af859-openshift-default-59576db9cf-cnbq6" event={"ID":"503330d0-a5f9-470b-8fd6-3ad2e4cf4698","Type":"ContainerStarted","Data":"dce1a7a834697b4311c1447c80e88dd00fe5131ae86329c3ae85e6df1db415e4"}
Oct 31 01:41:37.333176 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:41:37.333142    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:41:37.379799 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:41:37.371750    4964 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-8af859-openshift-default-59576db9cf-cnbq6" podStartSLOduration=2.37173224 podStartE2EDuration="2.37173224s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:41:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:41:37.370567483 +0000 UTC m=+2265.847133876" watchObservedRunningTime="2026-10-31 01:41:37.37173224 +0000 UTC m=+2265.848298620"
Oct 31 01:41:37.994366 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:41:37.992647    4964 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-8af859-openshift-default-59576db9cf-cnbq6"
Oct 31 01:41:37.998823 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:41:37.996449    4964 patch_prober.go:29] interesting pod/e2e-ipp-b-8af859-openshift-default-59576db9cf-cnbq6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused" start-of-body=
Oct 31 01:41:37.998823 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:41:37.996505    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-8af859-openshift-default-59576db9cf-cnbq6" podUID="503330d0-a5f9-470b-8fd6-3ad2e4cf4698" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused"
Oct 31 01:41:39.003342 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:41:38.994725    4964 patch_prober.go:29] interesting pod/e2e-ipp-b-8af859-openshift-default-59576db9cf-cnbq6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused" start-of-body=
Oct 31 01:41:39.003342 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:41:38.994791    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-8af859-openshift-default-59576db9cf-cnbq6" podUID="503330d0-a5f9-470b-8fd6-3ad2e4cf4698" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused"
Oct 31 01:41:39.359034 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:41:39.358951    4964 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-8af859-794678d7b7-npqqw"]
Oct 31 01:41:39.650178 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:41:39.650098    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:41:39.834577 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:41:39.834539    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:41:39.992714 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:41:39.992676    4964 patch_prober.go:29] interesting pod/e2e-ipp-b-8af859-openshift-default-59576db9cf-cnbq6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused" start-of-body=
Oct 31 01:41:39.993005 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:41:39.992968    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-8af859-openshift-default-59576db9cf-cnbq6" podUID="503330d0-a5f9-470b-8fd6-3ad2e4cf4698" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused"
Oct 31 01:41:40.575190 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:41:40.575153    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:41:40.994769 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:41:40.994730    4964 patch_prober.go:29] interesting pod/e2e-ipp-b-8af859-openshift-default-59576db9cf-cnbq6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused" start-of-body=
Oct 31 01:41:40.995094 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:41:40.995062    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-8af859-openshift-default-59576db9cf-cnbq6" podUID="503330d0-a5f9-470b-8fd6-3ad2e4cf4698" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused"
Oct 31 01:41:41.245907 ip-10-0-133-197 kubenswrapper[4964]: E1031 01:41:41.245826    4964 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q56x5" containerName="extractor"
Oct 31 01:41:41.996800 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:41:41.996763    4964 patch_prober.go:29] interesting pod/e2e-ipp-b-8af859-openshift-default-59576db9cf-cnbq6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused" start-of-body=
Oct 31 01:41:41.997444 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:41:41.997404    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-8af859-openshift-default-59576db9cf-cnbq6" podUID="503330d0-a5f9-470b-8fd6-3ad2e4cf4698" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused"
Oct 31 01:41:42.997778 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:41:42.997741    4964 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-8af859-openshift-default-59576db9cf-cnbq6"
Oct 31 01:41:42.998241 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:41:42.998084    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-8af859-openshift-default-59576db9cf-cnbq6"
Oct 31 01:41:42.998781 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:41:42.998762    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-8af859-openshift-default-59576db9cf-cnbq6"
Oct 31 01:41:44.359484 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:41:44.359445    4964 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8af859-794678d7b7-npqqw" podUID="8b94036e-3ae4-4026-9171-2ab95b1db68a" containerName="payload-pre-processing" containerID="cri-o://fed262b4c7ca13a927cbfce65c8729cb24d4006f26d4f6697377b6095b2df5cd" gracePeriod=25
Oct 31 01:41:44.676881 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:41:44.676841    4964 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8af859-794678d7b7-npqqw"
Oct 31 01:41:44.744064 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:41:44.744022    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8b94036e-3ae4-4026-9171-2ab95b1db68a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8b94036e-3ae4-4026-9171-2ab95b1db68a-plugins-config-volume\") pod \"8b94036e-3ae4-4026-9171-2ab95b1db68a\" (UID: \"8b94036e-3ae4-4026-9171-2ab95b1db68a\") "
Oct 31 01:41:44.744252 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:41:44.744090    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8b94036e-3ae4-4026-9171-2ab95b1db68a-kube-api-access-sng6h\" (UniqueName: \"kubernetes.io/projected/8b94036e-3ae4-4026-9171-2ab95b1db68a-kube-api-access-sng6h\") pod \"8b94036e-3ae4-4026-9171-2ab95b1db68a\" (UID: \"8b94036e-3ae4-4026-9171-2ab95b1db68a\") "
Oct 31 01:41:44.744442 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:41:44.744416    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8b94036e-3ae4-4026-9171-2ab95b1db68a-plugins-config-volume" pod "8b94036e-3ae4-4026-9171-2ab95b1db68a" (UID: "8b94036e-3ae4-4026-9171-2ab95b1db68a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:41:44.744658 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:41:44.744642    4964 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8b94036e-3ae4-4026-9171-2ab95b1db68a-plugins-config-volume\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:41:44.746402 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:41:44.746367    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b94036e-3ae4-4026-9171-2ab95b1db68a-kube-api-access-sng6h" pod "8b94036e-3ae4-4026-9171-2ab95b1db68a" (UID: "8b94036e-3ae4-4026-9171-2ab95b1db68a"). InnerVolumeSpecName "kube-api-access-sng6h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:41:44.846106 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:41:44.846066    4964 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sng6h\" (UniqueName: \"kubernetes.io/projected/8b94036e-3ae4-4026-9171-2ab95b1db68a-kube-api-access-sng6h\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:41:45.273483 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:41:45.273445    4964 generic.go:356] "Generic (PLEG): container finished" podID="8b94036e-3ae4-4026-9171-2ab95b1db68a" containerID="fed262b4c7ca13a927cbfce65c8729cb24d4006f26d4f6697377b6095b2df5cd" exitCode=0
Oct 31 01:41:45.273687 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:41:45.273494    4964 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8af859-794678d7b7-npqqw"
Oct 31 01:41:45.273687 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:41:45.273533    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8af859-794678d7b7-npqqw" event={"ID":"8b94036e-3ae4-4026-9171-2ab95b1db68a","Type":"ContainerDied","Data":"fed262b4c7ca13a927cbfce65c8729cb24d4006f26d4f6697377b6095b2df5cd"}
Oct 31 01:41:45.273687 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:41:45.273587    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8af859-794678d7b7-npqqw" event={"ID":"8b94036e-3ae4-4026-9171-2ab95b1db68a","Type":"ContainerDied","Data":"8e344044fdf3e76ebe91e0d20356847a703b27f91475ab16c835229cdae02154"}
Oct 31 01:41:45.273687 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:41:45.273613    4964 scope.go:122] "RemoveContainer" containerID="fed262b4c7ca13a927cbfce65c8729cb24d4006f26d4f6697377b6095b2df5cd"
Oct 31 01:41:45.285244 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:41:45.285223    4964 scope.go:122] "RemoveContainer" containerID="fed262b4c7ca13a927cbfce65c8729cb24d4006f26d4f6697377b6095b2df5cd"
Oct 31 01:41:45.285637 ip-10-0-133-197 kubenswrapper[4964]: E1031 01:41:45.285615    4964 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fed262b4c7ca13a927cbfce65c8729cb24d4006f26d4f6697377b6095b2df5cd\": container with ID starting with fed262b4c7ca13a927cbfce65c8729cb24d4006f26d4f6697377b6095b2df5cd not found: ID does not exist" containerID="fed262b4c7ca13a927cbfce65c8729cb24d4006f26d4f6697377b6095b2df5cd"
Oct 31 01:41:45.285727 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:41:45.285647    4964 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fed262b4c7ca13a927cbfce65c8729cb24d4006f26d4f6697377b6095b2df5cd"} err="failed to get container status \"fed262b4c7ca13a927cbfce65c8729cb24d4006f26d4f6697377b6095b2df5cd\": rpc error: code = NotFound desc = could not find container \"fed262b4c7ca13a927cbfce65c8729cb24d4006f26d4f6697377b6095b2df5cd\": container with ID starting with fed262b4c7ca13a927cbfce65c8729cb24d4006f26d4f6697377b6095b2df5cd not found: ID does not exist"
Oct 31 01:41:45.439010 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:41:45.438979    4964 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-8af859-794678d7b7-npqqw"]
Oct 31 01:41:45.487015 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:41:45.486975    4964 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-8af859-794678d7b7-npqqw"]
Oct 31 01:41:45.778881 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:41:45.778834    4964 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8b94036e-3ae4-4026-9171-2ab95b1db68a" path="/var/lib/kubelet/pods/8b94036e-3ae4-4026-9171-2ab95b1db68a/volumes"
Oct 31 01:41:47.803061 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:41:47.803013    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:41:54.424412 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:41:54.424360    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:42:00.938022 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:42:00.937923    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:42:05.698327 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:42:05.698271    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:42:10.807412 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:42:10.807375    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:42:11.237089 ip-10-0-133-197 kubenswrapper[4964]: E1031 01:42:11.237056    4964 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q56x5" containerName="extractor"
Oct 31 01:42:11.392545 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:42:11.392513    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:42:12.750082 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:42:12.750043    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:42:13.696837 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:42:13.696781    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:42:15.528920 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:42:15.528863    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:42:18.106405 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:42:18.106365    4964 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-k5bkg"]
Oct 31 01:42:18.106829 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:42:18.106636    4964 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-k5bkg" podUID="197445f1-d262-48ee-87c5-7354132a69ba" containerName="istio-proxy" containerID="cri-o://502ae917f77852b871ed373eec53cd1752926cb6ecf3861f531da60d67d10253" gracePeriod=30
Oct 31 01:42:20.267809 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:42:20.267762    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:42:23.373922 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:42:23.373866    4964 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-k5bkg"
Oct 31 01:42:23.453017 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:42:23.452987    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/197445f1-d262-48ee-87c5-7354132a69ba-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/197445f1-d262-48ee-87c5-7354132a69ba-credential-socket\") pod \"197445f1-d262-48ee-87c5-7354132a69ba\" (UID: \"197445f1-d262-48ee-87c5-7354132a69ba\") "
Oct 31 01:42:23.453131 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:42:23.453059    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/197445f1-d262-48ee-87c5-7354132a69ba-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/197445f1-d262-48ee-87c5-7354132a69ba-workload-certs\") pod \"197445f1-d262-48ee-87c5-7354132a69ba\" (UID: \"197445f1-d262-48ee-87c5-7354132a69ba\") "
Oct 31 01:42:23.453131 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:42:23.453088    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/197445f1-d262-48ee-87c5-7354132a69ba-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/197445f1-d262-48ee-87c5-7354132a69ba-istio-data\") pod \"197445f1-d262-48ee-87c5-7354132a69ba\" (UID: \"197445f1-d262-48ee-87c5-7354132a69ba\") "
Oct 31 01:42:23.453207 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:42:23.453148    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/197445f1-d262-48ee-87c5-7354132a69ba-istio-token\" (UniqueName: \"kubernetes.io/projected/197445f1-d262-48ee-87c5-7354132a69ba-istio-token\") pod \"197445f1-d262-48ee-87c5-7354132a69ba\" (UID: \"197445f1-d262-48ee-87c5-7354132a69ba\") "
Oct 31 01:42:23.453207 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:42:23.453183    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/197445f1-d262-48ee-87c5-7354132a69ba-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/197445f1-d262-48ee-87c5-7354132a69ba-workload-socket\") pod \"197445f1-d262-48ee-87c5-7354132a69ba\" (UID: \"197445f1-d262-48ee-87c5-7354132a69ba\") "
Oct 31 01:42:23.453361 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:42:23.453225    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/197445f1-d262-48ee-87c5-7354132a69ba-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/197445f1-d262-48ee-87c5-7354132a69ba-istio-podinfo\") pod \"197445f1-d262-48ee-87c5-7354132a69ba\" (UID: \"197445f1-d262-48ee-87c5-7354132a69ba\") "
Oct 31 01:42:23.453361 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:42:23.453236    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/197445f1-d262-48ee-87c5-7354132a69ba-credential-socket" pod "197445f1-d262-48ee-87c5-7354132a69ba" (UID: "197445f1-d262-48ee-87c5-7354132a69ba"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:42:23.453477 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:42:23.453373    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/197445f1-d262-48ee-87c5-7354132a69ba-kube-api-access-7855f\" (UniqueName: \"kubernetes.io/projected/197445f1-d262-48ee-87c5-7354132a69ba-kube-api-access-7855f\") pod \"197445f1-d262-48ee-87c5-7354132a69ba\" (UID: \"197445f1-d262-48ee-87c5-7354132a69ba\") "
Oct 31 01:42:23.453477 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:42:23.453425    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/197445f1-d262-48ee-87c5-7354132a69ba-workload-certs" pod "197445f1-d262-48ee-87c5-7354132a69ba" (UID: "197445f1-d262-48ee-87c5-7354132a69ba"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:42:23.453477 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:42:23.453466    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/197445f1-d262-48ee-87c5-7354132a69ba-istio-data" pod "197445f1-d262-48ee-87c5-7354132a69ba" (UID: "197445f1-d262-48ee-87c5-7354132a69ba"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:42:23.453638 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:42:23.453484    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/197445f1-d262-48ee-87c5-7354132a69ba-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/197445f1-d262-48ee-87c5-7354132a69ba-istiod-ca-cert\") pod \"197445f1-d262-48ee-87c5-7354132a69ba\" (UID: \"197445f1-d262-48ee-87c5-7354132a69ba\") "
Oct 31 01:42:23.453638 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:42:23.453587    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/197445f1-d262-48ee-87c5-7354132a69ba-workload-socket" pod "197445f1-d262-48ee-87c5-7354132a69ba" (UID: "197445f1-d262-48ee-87c5-7354132a69ba"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:42:23.453739 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:42:23.453657    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/197445f1-d262-48ee-87c5-7354132a69ba-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/197445f1-d262-48ee-87c5-7354132a69ba-istio-envoy\") pod \"197445f1-d262-48ee-87c5-7354132a69ba\" (UID: \"197445f1-d262-48ee-87c5-7354132a69ba\") "
Oct 31 01:42:23.453833 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:42:23.453809    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/197445f1-d262-48ee-87c5-7354132a69ba-istiod-ca-cert" pod "197445f1-d262-48ee-87c5-7354132a69ba" (UID: "197445f1-d262-48ee-87c5-7354132a69ba"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:42:23.454286 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:42:23.454163    4964 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/197445f1-d262-48ee-87c5-7354132a69ba-credential-socket\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:42:23.454286 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:42:23.454188    4964 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/197445f1-d262-48ee-87c5-7354132a69ba-workload-certs\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:42:23.454286 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:42:23.454203    4964 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/197445f1-d262-48ee-87c5-7354132a69ba-istio-data\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:42:23.454286 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:42:23.454212    4964 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/197445f1-d262-48ee-87c5-7354132a69ba-workload-socket\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:42:23.454286 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:42:23.454220    4964 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/197445f1-d262-48ee-87c5-7354132a69ba-istiod-ca-cert\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:42:23.455680 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:42:23.455657    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/197445f1-d262-48ee-87c5-7354132a69ba-kube-api-access-7855f" pod "197445f1-d262-48ee-87c5-7354132a69ba" (UID: "197445f1-d262-48ee-87c5-7354132a69ba"). InnerVolumeSpecName "kube-api-access-7855f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:42:23.455790 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:42:23.455768    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/197445f1-d262-48ee-87c5-7354132a69ba-istio-envoy" pod "197445f1-d262-48ee-87c5-7354132a69ba" (UID: "197445f1-d262-48ee-87c5-7354132a69ba"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:42:23.456077 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:42:23.456051    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/197445f1-d262-48ee-87c5-7354132a69ba-istio-podinfo" pod "197445f1-d262-48ee-87c5-7354132a69ba" (UID: "197445f1-d262-48ee-87c5-7354132a69ba"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 01:42:23.456077 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:42:23.456064    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/197445f1-d262-48ee-87c5-7354132a69ba-istio-token" pod "197445f1-d262-48ee-87c5-7354132a69ba" (UID: "197445f1-d262-48ee-87c5-7354132a69ba"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:42:23.555906 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:42:23.555880    4964 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/197445f1-d262-48ee-87c5-7354132a69ba-istio-podinfo\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:42:23.555906 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:42:23.555910    4964 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7855f\" (UniqueName: \"kubernetes.io/projected/197445f1-d262-48ee-87c5-7354132a69ba-kube-api-access-7855f\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:42:23.556088 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:42:23.555925    4964 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/197445f1-d262-48ee-87c5-7354132a69ba-istio-envoy\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:42:23.556088 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:42:23.555936    4964 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/197445f1-d262-48ee-87c5-7354132a69ba-istio-token\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:42:23.567148 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:42:23.567115    4964 generic.go:356] "Generic (PLEG): container finished" podID="197445f1-d262-48ee-87c5-7354132a69ba" containerID="502ae917f77852b871ed373eec53cd1752926cb6ecf3861f531da60d67d10253" exitCode=0
Oct 31 01:42:23.567297 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:42:23.567163    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-k5bkg" event={"ID":"197445f1-d262-48ee-87c5-7354132a69ba","Type":"ContainerDied","Data":"502ae917f77852b871ed373eec53cd1752926cb6ecf3861f531da60d67d10253"}
Oct 31 01:42:23.567297 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:42:23.567233    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-k5bkg" event={"ID":"197445f1-d262-48ee-87c5-7354132a69ba","Type":"ContainerDied","Data":"2fca55b360fee11bcd5f8395c67c48c9ee548c315168b50474f524f46a6446f7"}
Oct 31 01:42:23.567297 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:42:23.567174    4964 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-k5bkg"
Oct 31 01:42:23.567482 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:42:23.567264    4964 scope.go:122] "RemoveContainer" containerID="502ae917f77852b871ed373eec53cd1752926cb6ecf3861f531da60d67d10253"
Oct 31 01:42:23.582833 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:42:23.582814    4964 scope.go:122] "RemoveContainer" containerID="2da91d4df5685d2274c1062cb98ca34fa297d1f9e78c8e9e4ed6e4662122a1c4"
Oct 31 01:42:23.595948 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:42:23.595929    4964 scope.go:122] "RemoveContainer" containerID="502ae917f77852b871ed373eec53cd1752926cb6ecf3861f531da60d67d10253"
Oct 31 01:42:23.596215 ip-10-0-133-197 kubenswrapper[4964]: E1031 01:42:23.596199    4964 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"502ae917f77852b871ed373eec53cd1752926cb6ecf3861f531da60d67d10253\": container with ID starting with 502ae917f77852b871ed373eec53cd1752926cb6ecf3861f531da60d67d10253 not found: ID does not exist" containerID="502ae917f77852b871ed373eec53cd1752926cb6ecf3861f531da60d67d10253"
Oct 31 01:42:23.596264 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:42:23.596224    4964 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"502ae917f77852b871ed373eec53cd1752926cb6ecf3861f531da60d67d10253"} err="failed to get container status \"502ae917f77852b871ed373eec53cd1752926cb6ecf3861f531da60d67d10253\": rpc error: code = NotFound desc = could not find container \"502ae917f77852b871ed373eec53cd1752926cb6ecf3861f531da60d67d10253\": container with ID starting with 502ae917f77852b871ed373eec53cd1752926cb6ecf3861f531da60d67d10253 not found: ID does not exist"
Oct 31 01:42:23.596264 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:42:23.596241    4964 scope.go:122] "RemoveContainer" containerID="2da91d4df5685d2274c1062cb98ca34fa297d1f9e78c8e9e4ed6e4662122a1c4"
Oct 31 01:42:23.596501 ip-10-0-133-197 kubenswrapper[4964]: E1031 01:42:23.596483    4964 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2da91d4df5685d2274c1062cb98ca34fa297d1f9e78c8e9e4ed6e4662122a1c4\": container with ID starting with 2da91d4df5685d2274c1062cb98ca34fa297d1f9e78c8e9e4ed6e4662122a1c4 not found: ID does not exist" containerID="2da91d4df5685d2274c1062cb98ca34fa297d1f9e78c8e9e4ed6e4662122a1c4"
Oct 31 01:42:23.596544 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:42:23.596508    4964 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2da91d4df5685d2274c1062cb98ca34fa297d1f9e78c8e9e4ed6e4662122a1c4"} err="failed to get container status \"2da91d4df5685d2274c1062cb98ca34fa297d1f9e78c8e9e4ed6e4662122a1c4\": rpc error: code = NotFound desc = could not find container \"2da91d4df5685d2274c1062cb98ca34fa297d1f9e78c8e9e4ed6e4662122a1c4\": container with ID starting with 2da91d4df5685d2274c1062cb98ca34fa297d1f9e78c8e9e4ed6e4662122a1c4 not found: ID does not exist"
Oct 31 01:42:23.734295 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:42:23.734247    4964 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-k5bkg"]
Oct 31 01:42:23.738918 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:42:23.738888    4964 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-k5bkg"]
Oct 31 01:42:23.777985 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:42:23.777955    4964 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="197445f1-d262-48ee-87c5-7354132a69ba" path="/var/lib/kubelet/pods/197445f1-d262-48ee-87c5-7354132a69ba/volumes"
Oct 31 01:42:24.005510 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:42:24.005430    4964 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-jd7bn"]
Oct 31 01:42:24.005687 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:42:24.005622    4964 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-jd7bn" podUID="2bf44719-ff6b-4ada-9fe7-a6f466992512" containerName="istio-proxy" containerID="cri-o://5d9493ca37bee237a56b1d5009b41f338585064db868ef1f43c6a72b70aa582a" gracePeriod=30
Oct 31 01:42:25.198555 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:42:25.198525    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:42:25.561110 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:42:25.561081    4964 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-z5xgw"]
Oct 31 01:42:25.561334 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:42:25.561287    4964 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-z5xgw" podUID="b8249620-2e20-4e80-a8b8-7d8e566f8973" containerName="istio-proxy" containerID="cri-o://7827d6024a864c5b388e226d2fcf57413ad5424cfeddd516ddaa4b8d1e73be9d" gracePeriod=30
Oct 31 01:42:29.300665 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:42:29.300641    4964 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-jd7bn"
Oct 31 01:42:29.327389 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:42:29.327364    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2bf44719-ff6b-4ada-9fe7-a6f466992512-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2bf44719-ff6b-4ada-9fe7-a6f466992512-workload-certs\") pod \"2bf44719-ff6b-4ada-9fe7-a6f466992512\" (UID: \"2bf44719-ff6b-4ada-9fe7-a6f466992512\") "
Oct 31 01:42:29.327555 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:42:29.327449    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2bf44719-ff6b-4ada-9fe7-a6f466992512-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2bf44719-ff6b-4ada-9fe7-a6f466992512-istio-data\") pod \"2bf44719-ff6b-4ada-9fe7-a6f466992512\" (UID: \"2bf44719-ff6b-4ada-9fe7-a6f466992512\") "
Oct 31 01:42:29.327555 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:42:29.327507    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2bf44719-ff6b-4ada-9fe7-a6f466992512-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2bf44719-ff6b-4ada-9fe7-a6f466992512-credential-socket\") pod \"2bf44719-ff6b-4ada-9fe7-a6f466992512\" (UID: \"2bf44719-ff6b-4ada-9fe7-a6f466992512\") "
Oct 31 01:42:29.327555 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:42:29.327537    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2bf44719-ff6b-4ada-9fe7-a6f466992512-kube-api-access-8qtd9\" (UniqueName: \"kubernetes.io/projected/2bf44719-ff6b-4ada-9fe7-a6f466992512-kube-api-access-8qtd9\") pod \"2bf44719-ff6b-4ada-9fe7-a6f466992512\" (UID: \"2bf44719-ff6b-4ada-9fe7-a6f466992512\") "
Oct 31 01:42:29.327726 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:42:29.327581    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2bf44719-ff6b-4ada-9fe7-a6f466992512-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2bf44719-ff6b-4ada-9fe7-a6f466992512-istio-podinfo\") pod \"2bf44719-ff6b-4ada-9fe7-a6f466992512\" (UID: \"2bf44719-ff6b-4ada-9fe7-a6f466992512\") "
Oct 31 01:42:29.327726 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:42:29.327680    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2bf44719-ff6b-4ada-9fe7-a6f466992512-workload-certs" pod "2bf44719-ff6b-4ada-9fe7-a6f466992512" (UID: "2bf44719-ff6b-4ada-9fe7-a6f466992512"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:42:29.327726 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:42:29.327696    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2bf44719-ff6b-4ada-9fe7-a6f466992512-istio-data" pod "2bf44719-ff6b-4ada-9fe7-a6f466992512" (UID: "2bf44719-ff6b-4ada-9fe7-a6f466992512"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:42:29.327888 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:42:29.327736    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2bf44719-ff6b-4ada-9fe7-a6f466992512-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2bf44719-ff6b-4ada-9fe7-a6f466992512-workload-socket\") pod \"2bf44719-ff6b-4ada-9fe7-a6f466992512\" (UID: \"2bf44719-ff6b-4ada-9fe7-a6f466992512\") "
Oct 31 01:42:29.327888 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:42:29.327821    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2bf44719-ff6b-4ada-9fe7-a6f466992512-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2bf44719-ff6b-4ada-9fe7-a6f466992512-istiod-ca-cert\") pod \"2bf44719-ff6b-4ada-9fe7-a6f466992512\" (UID: \"2bf44719-ff6b-4ada-9fe7-a6f466992512\") "
Oct 31 01:42:29.327888 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:42:29.327823    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2bf44719-ff6b-4ada-9fe7-a6f466992512-credential-socket" pod "2bf44719-ff6b-4ada-9fe7-a6f466992512" (UID: "2bf44719-ff6b-4ada-9fe7-a6f466992512"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:42:29.327888 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:42:29.327873    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2bf44719-ff6b-4ada-9fe7-a6f466992512-istio-token\" (UniqueName: \"kubernetes.io/projected/2bf44719-ff6b-4ada-9fe7-a6f466992512-istio-token\") pod \"2bf44719-ff6b-4ada-9fe7-a6f466992512\" (UID: \"2bf44719-ff6b-4ada-9fe7-a6f466992512\") "
Oct 31 01:42:29.328091 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:42:29.327914    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2bf44719-ff6b-4ada-9fe7-a6f466992512-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2bf44719-ff6b-4ada-9fe7-a6f466992512-istio-envoy\") pod \"2bf44719-ff6b-4ada-9fe7-a6f466992512\" (UID: \"2bf44719-ff6b-4ada-9fe7-a6f466992512\") "
Oct 31 01:42:29.328091 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:42:29.327934    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2bf44719-ff6b-4ada-9fe7-a6f466992512-workload-socket" pod "2bf44719-ff6b-4ada-9fe7-a6f466992512" (UID: "2bf44719-ff6b-4ada-9fe7-a6f466992512"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:42:29.328441 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:42:29.328416    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2bf44719-ff6b-4ada-9fe7-a6f466992512-istiod-ca-cert" pod "2bf44719-ff6b-4ada-9fe7-a6f466992512" (UID: "2bf44719-ff6b-4ada-9fe7-a6f466992512"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:42:29.328827 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:42:29.328802    4964 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2bf44719-ff6b-4ada-9fe7-a6f466992512-workload-socket\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:42:29.328935 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:42:29.328834    4964 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2bf44719-ff6b-4ada-9fe7-a6f466992512-istiod-ca-cert\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:42:29.328935 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:42:29.328851    4964 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2bf44719-ff6b-4ada-9fe7-a6f466992512-workload-certs\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:42:29.328935 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:42:29.328864    4964 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2bf44719-ff6b-4ada-9fe7-a6f466992512-istio-data\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:42:29.328935 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:42:29.328878    4964 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2bf44719-ff6b-4ada-9fe7-a6f466992512-credential-socket\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:42:29.329970 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:42:29.329947    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2bf44719-ff6b-4ada-9fe7-a6f466992512-kube-api-access-8qtd9" pod "2bf44719-ff6b-4ada-9fe7-a6f466992512" (UID: "2bf44719-ff6b-4ada-9fe7-a6f466992512"). InnerVolumeSpecName "kube-api-access-8qtd9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:42:29.330116 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:42:29.330093    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2bf44719-ff6b-4ada-9fe7-a6f466992512-istio-podinfo" pod "2bf44719-ff6b-4ada-9fe7-a6f466992512" (UID: "2bf44719-ff6b-4ada-9fe7-a6f466992512"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 01:42:29.330442 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:42:29.330419    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2bf44719-ff6b-4ada-9fe7-a6f466992512-istio-token" pod "2bf44719-ff6b-4ada-9fe7-a6f466992512" (UID: "2bf44719-ff6b-4ada-9fe7-a6f466992512"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:42:29.330649 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:42:29.330634    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2bf44719-ff6b-4ada-9fe7-a6f466992512-istio-envoy" pod "2bf44719-ff6b-4ada-9fe7-a6f466992512" (UID: "2bf44719-ff6b-4ada-9fe7-a6f466992512"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:42:29.429794 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:42:29.429770    4964 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8qtd9\" (UniqueName: \"kubernetes.io/projected/2bf44719-ff6b-4ada-9fe7-a6f466992512-kube-api-access-8qtd9\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:42:29.429794 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:42:29.429792    4964 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2bf44719-ff6b-4ada-9fe7-a6f466992512-istio-podinfo\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:42:29.429945 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:42:29.429803    4964 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2bf44719-ff6b-4ada-9fe7-a6f466992512-istio-token\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:42:29.429945 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:42:29.429813    4964 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2bf44719-ff6b-4ada-9fe7-a6f466992512-istio-envoy\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:42:29.613557 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:42:29.613472    4964 generic.go:356] "Generic (PLEG): container finished" podID="2bf44719-ff6b-4ada-9fe7-a6f466992512" containerID="5d9493ca37bee237a56b1d5009b41f338585064db868ef1f43c6a72b70aa582a" exitCode=0
Oct 31 01:42:29.613557 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:42:29.613526    4964 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-jd7bn"
Oct 31 01:42:29.613747 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:42:29.613556    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-jd7bn" event={"ID":"2bf44719-ff6b-4ada-9fe7-a6f466992512","Type":"ContainerDied","Data":"5d9493ca37bee237a56b1d5009b41f338585064db868ef1f43c6a72b70aa582a"}
Oct 31 01:42:29.613747 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:42:29.613614    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-jd7bn" event={"ID":"2bf44719-ff6b-4ada-9fe7-a6f466992512","Type":"ContainerDied","Data":"9ac92f8e8af1ae59ddc7d88c96d1deab689c1d28c5c9bc9fc83ee4bac76f843b"}
Oct 31 01:42:29.613747 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:42:29.613645    4964 scope.go:122] "RemoveContainer" containerID="5d9493ca37bee237a56b1d5009b41f338585064db868ef1f43c6a72b70aa582a"
Oct 31 01:42:29.625530 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:42:29.625497    4964 scope.go:122] "RemoveContainer" containerID="5d9493ca37bee237a56b1d5009b41f338585064db868ef1f43c6a72b70aa582a"
Oct 31 01:42:29.625887 ip-10-0-133-197 kubenswrapper[4964]: E1031 01:42:29.625857    4964 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5d9493ca37bee237a56b1d5009b41f338585064db868ef1f43c6a72b70aa582a\": container with ID starting with 5d9493ca37bee237a56b1d5009b41f338585064db868ef1f43c6a72b70aa582a not found: ID does not exist" containerID="5d9493ca37bee237a56b1d5009b41f338585064db868ef1f43c6a72b70aa582a"
Oct 31 01:42:29.625972 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:42:29.625896    4964 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5d9493ca37bee237a56b1d5009b41f338585064db868ef1f43c6a72b70aa582a"} err="failed to get container status \"5d9493ca37bee237a56b1d5009b41f338585064db868ef1f43c6a72b70aa582a\": rpc error: code = NotFound desc = could not find container \"5d9493ca37bee237a56b1d5009b41f338585064db868ef1f43c6a72b70aa582a\": container with ID starting with 5d9493ca37bee237a56b1d5009b41f338585064db868ef1f43c6a72b70aa582a not found: ID does not exist"
Oct 31 01:42:29.777774 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:42:29.777741    4964 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-jd7bn"]
Oct 31 01:42:29.780452 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:42:29.780426    4964 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-jd7bn"]
Oct 31 01:42:30.858676 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:42:30.858654    4964 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-z5xgw"
Oct 31 01:42:30.947953 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:42:30.947926    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b8249620-2e20-4e80-a8b8-7d8e566f8973-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b8249620-2e20-4e80-a8b8-7d8e566f8973-istiod-ca-cert\") pod \"b8249620-2e20-4e80-a8b8-7d8e566f8973\" (UID: \"b8249620-2e20-4e80-a8b8-7d8e566f8973\") "
Oct 31 01:42:30.948139 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:42:30.948045    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b8249620-2e20-4e80-a8b8-7d8e566f8973-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b8249620-2e20-4e80-a8b8-7d8e566f8973-workload-certs\") pod \"b8249620-2e20-4e80-a8b8-7d8e566f8973\" (UID: \"b8249620-2e20-4e80-a8b8-7d8e566f8973\") "
Oct 31 01:42:30.948139 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:42:30.948088    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b8249620-2e20-4e80-a8b8-7d8e566f8973-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b8249620-2e20-4e80-a8b8-7d8e566f8973-istio-podinfo\") pod \"b8249620-2e20-4e80-a8b8-7d8e566f8973\" (UID: \"b8249620-2e20-4e80-a8b8-7d8e566f8973\") "
Oct 31 01:42:30.948139 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:42:30.948130    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b8249620-2e20-4e80-a8b8-7d8e566f8973-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8249620-2e20-4e80-a8b8-7d8e566f8973-workload-socket\") pod \"b8249620-2e20-4e80-a8b8-7d8e566f8973\" (UID: \"b8249620-2e20-4e80-a8b8-7d8e566f8973\") "
Oct 31 01:42:30.948308 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:42:30.948254    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b8249620-2e20-4e80-a8b8-7d8e566f8973-kube-api-access-f4qzp\" (UniqueName: \"kubernetes.io/projected/b8249620-2e20-4e80-a8b8-7d8e566f8973-kube-api-access-f4qzp\") pod \"b8249620-2e20-4e80-a8b8-7d8e566f8973\" (UID: \"b8249620-2e20-4e80-a8b8-7d8e566f8973\") "
Oct 31 01:42:30.948308 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:42:30.948265    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b8249620-2e20-4e80-a8b8-7d8e566f8973-istiod-ca-cert" pod "b8249620-2e20-4e80-a8b8-7d8e566f8973" (UID: "b8249620-2e20-4e80-a8b8-7d8e566f8973"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:42:30.948468 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:42:30.948285    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b8249620-2e20-4e80-a8b8-7d8e566f8973-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8249620-2e20-4e80-a8b8-7d8e566f8973-credential-socket\") pod \"b8249620-2e20-4e80-a8b8-7d8e566f8973\" (UID: \"b8249620-2e20-4e80-a8b8-7d8e566f8973\") "
Oct 31 01:42:30.948532 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:42:30.948350    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8249620-2e20-4e80-a8b8-7d8e566f8973-workload-certs" pod "b8249620-2e20-4e80-a8b8-7d8e566f8973" (UID: "b8249620-2e20-4e80-a8b8-7d8e566f8973"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:42:30.948532 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:42:30.948460    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b8249620-2e20-4e80-a8b8-7d8e566f8973-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b8249620-2e20-4e80-a8b8-7d8e566f8973-istio-envoy\") pod \"b8249620-2e20-4e80-a8b8-7d8e566f8973\" (UID: \"b8249620-2e20-4e80-a8b8-7d8e566f8973\") "
Oct 31 01:42:30.948620 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:42:30.948536    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8249620-2e20-4e80-a8b8-7d8e566f8973-workload-socket" pod "b8249620-2e20-4e80-a8b8-7d8e566f8973" (UID: "b8249620-2e20-4e80-a8b8-7d8e566f8973"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:42:30.948620 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:42:30.948539    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8249620-2e20-4e80-a8b8-7d8e566f8973-credential-socket" pod "b8249620-2e20-4e80-a8b8-7d8e566f8973" (UID: "b8249620-2e20-4e80-a8b8-7d8e566f8973"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:42:30.948620 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:42:30.948591    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b8249620-2e20-4e80-a8b8-7d8e566f8973-istio-token\" (UniqueName: \"kubernetes.io/projected/b8249620-2e20-4e80-a8b8-7d8e566f8973-istio-token\") pod \"b8249620-2e20-4e80-a8b8-7d8e566f8973\" (UID: \"b8249620-2e20-4e80-a8b8-7d8e566f8973\") "
Oct 31 01:42:30.948772 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:42:30.948682    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b8249620-2e20-4e80-a8b8-7d8e566f8973-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b8249620-2e20-4e80-a8b8-7d8e566f8973-istio-data\") pod \"b8249620-2e20-4e80-a8b8-7d8e566f8973\" (UID: \"b8249620-2e20-4e80-a8b8-7d8e566f8973\") "
Oct 31 01:42:30.949044 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:42:30.948992    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8249620-2e20-4e80-a8b8-7d8e566f8973-istio-data" pod "b8249620-2e20-4e80-a8b8-7d8e566f8973" (UID: "b8249620-2e20-4e80-a8b8-7d8e566f8973"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:42:30.949663 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:42:30.949568    4964 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b8249620-2e20-4e80-a8b8-7d8e566f8973-workload-certs\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:42:30.949663 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:42:30.949592    4964 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8249620-2e20-4e80-a8b8-7d8e566f8973-workload-socket\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:42:30.949663 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:42:30.949608    4964 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8249620-2e20-4e80-a8b8-7d8e566f8973-credential-socket\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:42:30.949663 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:42:30.949624    4964 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b8249620-2e20-4e80-a8b8-7d8e566f8973-istio-data\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:42:30.949663 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:42:30.949641    4964 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b8249620-2e20-4e80-a8b8-7d8e566f8973-istiod-ca-cert\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:42:30.950777 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:42:30.950756    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b8249620-2e20-4e80-a8b8-7d8e566f8973-istio-podinfo" pod "b8249620-2e20-4e80-a8b8-7d8e566f8973" (UID: "b8249620-2e20-4e80-a8b8-7d8e566f8973"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 01:42:30.951064 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:42:30.951045    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8249620-2e20-4e80-a8b8-7d8e566f8973-istio-envoy" pod "b8249620-2e20-4e80-a8b8-7d8e566f8973" (UID: "b8249620-2e20-4e80-a8b8-7d8e566f8973"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:42:30.951186 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:42:30.951171    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b8249620-2e20-4e80-a8b8-7d8e566f8973-kube-api-access-f4qzp" pod "b8249620-2e20-4e80-a8b8-7d8e566f8973" (UID: "b8249620-2e20-4e80-a8b8-7d8e566f8973"). InnerVolumeSpecName "kube-api-access-f4qzp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:42:30.951240 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:42:30.951184    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b8249620-2e20-4e80-a8b8-7d8e566f8973-istio-token" pod "b8249620-2e20-4e80-a8b8-7d8e566f8973" (UID: "b8249620-2e20-4e80-a8b8-7d8e566f8973"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:42:31.050893 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:42:31.050860    4964 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b8249620-2e20-4e80-a8b8-7d8e566f8973-istio-podinfo\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:42:31.050893 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:42:31.050890    4964 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f4qzp\" (UniqueName: \"kubernetes.io/projected/b8249620-2e20-4e80-a8b8-7d8e566f8973-kube-api-access-f4qzp\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:42:31.051076 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:42:31.050907    4964 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b8249620-2e20-4e80-a8b8-7d8e566f8973-istio-envoy\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:42:31.051076 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:42:31.050919    4964 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b8249620-2e20-4e80-a8b8-7d8e566f8973-istio-token\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:42:31.541329 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:42:31.541269    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:42:31.633698 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:42:31.633661    4964 generic.go:356] "Generic (PLEG): container finished" podID="b8249620-2e20-4e80-a8b8-7d8e566f8973" containerID="7827d6024a864c5b388e226d2fcf57413ad5424cfeddd516ddaa4b8d1e73be9d" exitCode=0
Oct 31 01:42:31.633897 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:42:31.633733    4964 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-z5xgw"
Oct 31 01:42:31.633897 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:42:31.633743    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-z5xgw" event={"ID":"b8249620-2e20-4e80-a8b8-7d8e566f8973","Type":"ContainerDied","Data":"7827d6024a864c5b388e226d2fcf57413ad5424cfeddd516ddaa4b8d1e73be9d"}
Oct 31 01:42:31.633897 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:42:31.633849    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-z5xgw" event={"ID":"b8249620-2e20-4e80-a8b8-7d8e566f8973","Type":"ContainerDied","Data":"eb82087df74af2e96066f2973438d8ef40647db391c64ebc1aa64de4a875299b"}
Oct 31 01:42:31.633897 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:42:31.633865    4964 scope.go:122] "RemoveContainer" containerID="7827d6024a864c5b388e226d2fcf57413ad5424cfeddd516ddaa4b8d1e73be9d"
Oct 31 01:42:31.646250 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:42:31.646231    4964 scope.go:122] "RemoveContainer" containerID="fb10c38a790062d0c788cfe72ba32e556f4b7975316f7e9459bd3b5ada405c17"
Oct 31 01:42:31.655537 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:42:31.655517    4964 scope.go:122] "RemoveContainer" containerID="7827d6024a864c5b388e226d2fcf57413ad5424cfeddd516ddaa4b8d1e73be9d"
Oct 31 01:42:31.655806 ip-10-0-133-197 kubenswrapper[4964]: E1031 01:42:31.655784    4964 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7827d6024a864c5b388e226d2fcf57413ad5424cfeddd516ddaa4b8d1e73be9d\": container with ID starting with 7827d6024a864c5b388e226d2fcf57413ad5424cfeddd516ddaa4b8d1e73be9d not found: ID does not exist" containerID="7827d6024a864c5b388e226d2fcf57413ad5424cfeddd516ddaa4b8d1e73be9d"
Oct 31 01:42:31.655873 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:42:31.655817    4964 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7827d6024a864c5b388e226d2fcf57413ad5424cfeddd516ddaa4b8d1e73be9d"} err="failed to get container status \"7827d6024a864c5b388e226d2fcf57413ad5424cfeddd516ddaa4b8d1e73be9d\": rpc error: code = NotFound desc = could not find container \"7827d6024a864c5b388e226d2fcf57413ad5424cfeddd516ddaa4b8d1e73be9d\": container with ID starting with 7827d6024a864c5b388e226d2fcf57413ad5424cfeddd516ddaa4b8d1e73be9d not found: ID does not exist"
Oct 31 01:42:31.655873 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:42:31.655840    4964 scope.go:122] "RemoveContainer" containerID="fb10c38a790062d0c788cfe72ba32e556f4b7975316f7e9459bd3b5ada405c17"
Oct 31 01:42:31.656106 ip-10-0-133-197 kubenswrapper[4964]: E1031 01:42:31.656089    4964 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fb10c38a790062d0c788cfe72ba32e556f4b7975316f7e9459bd3b5ada405c17\": container with ID starting with fb10c38a790062d0c788cfe72ba32e556f4b7975316f7e9459bd3b5ada405c17 not found: ID does not exist" containerID="fb10c38a790062d0c788cfe72ba32e556f4b7975316f7e9459bd3b5ada405c17"
Oct 31 01:42:31.656157 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:42:31.656116    4964 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fb10c38a790062d0c788cfe72ba32e556f4b7975316f7e9459bd3b5ada405c17"} err="failed to get container status \"fb10c38a790062d0c788cfe72ba32e556f4b7975316f7e9459bd3b5ada405c17\": rpc error: code = NotFound desc = could not find container \"fb10c38a790062d0c788cfe72ba32e556f4b7975316f7e9459bd3b5ada405c17\": container with ID starting with fb10c38a790062d0c788cfe72ba32e556f4b7975316f7e9459bd3b5ada405c17 not found: ID does not exist"
Oct 31 01:42:31.770216 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:42:31.770185    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:42:31.779168 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:42:31.779116    4964 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2bf44719-ff6b-4ada-9fe7-a6f466992512" path="/var/lib/kubelet/pods/2bf44719-ff6b-4ada-9fe7-a6f466992512/volumes"
Oct 31 01:42:31.799057 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:42:31.798969    4964 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-z5xgw"]
Oct 31 01:42:31.847664 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:42:31.847633    4964 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-z5xgw"]
Oct 31 01:42:31.956138 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:42:31.956096    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:42:33.780470 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:42:33.780428    4964 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b8249620-2e20-4e80-a8b8-7d8e566f8973" path="/var/lib/kubelet/pods/b8249620-2e20-4e80-a8b8-7d8e566f8973/volumes"
Oct 31 01:42:34.985453 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:42:34.985413    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:42:35.658875 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:42:35.658831    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:42:35.787843 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:42:35.787811    4964 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-99c8bd-5dfcd8b544-q74wc"]
Oct 31 01:42:35.788036 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:42:35.787999    4964 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-99c8bd-5dfcd8b544-q74wc" podUID="c1bf5a83-ee88-42cd-8d94-9502ee38bfab" containerName="maas-api" containerID="cri-o://3ad371415279d453f03782c26ec2e6dd497b1581ca74afa9164f78c47f83a80d" gracePeriod=30
Oct 31 01:42:35.930757 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:42:35.930676    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:42:36.094664 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:42:36.094638    4964 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-99c8bd-5dfcd8b544-q74wc"
Oct 31 01:42:36.212424 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:42:36.211382    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c1bf5a83-ee88-42cd-8d94-9502ee38bfab-kube-api-access-ps8kg\" (UniqueName: \"kubernetes.io/projected/c1bf5a83-ee88-42cd-8d94-9502ee38bfab-kube-api-access-ps8kg\") pod \"c1bf5a83-ee88-42cd-8d94-9502ee38bfab\" (UID: \"c1bf5a83-ee88-42cd-8d94-9502ee38bfab\") "
Oct 31 01:42:36.212424 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:42:36.211551    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c1bf5a83-ee88-42cd-8d94-9502ee38bfab-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c1bf5a83-ee88-42cd-8d94-9502ee38bfab-maas-api-tls\") pod \"c1bf5a83-ee88-42cd-8d94-9502ee38bfab\" (UID: \"c1bf5a83-ee88-42cd-8d94-9502ee38bfab\") "
Oct 31 01:42:36.212424 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:42:36.211683    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c1bf5a83-ee88-42cd-8d94-9502ee38bfab-metrics-certs\" (UniqueName: \"kubernetes.io/secret/c1bf5a83-ee88-42cd-8d94-9502ee38bfab-metrics-certs\") pod \"c1bf5a83-ee88-42cd-8d94-9502ee38bfab\" (UID: \"c1bf5a83-ee88-42cd-8d94-9502ee38bfab\") "
Oct 31 01:42:36.212424 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:42:36.211733    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c1bf5a83-ee88-42cd-8d94-9502ee38bfab-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c1bf5a83-ee88-42cd-8d94-9502ee38bfab-openshift-service-ca\") pod \"c1bf5a83-ee88-42cd-8d94-9502ee38bfab\" (UID: \"c1bf5a83-ee88-42cd-8d94-9502ee38bfab\") "
Oct 31 01:42:36.212772 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:42:36.212596    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c1bf5a83-ee88-42cd-8d94-9502ee38bfab-openshift-service-ca" pod "c1bf5a83-ee88-42cd-8d94-9502ee38bfab" (UID: "c1bf5a83-ee88-42cd-8d94-9502ee38bfab"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:42:36.219363 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:42:36.215063    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c1bf5a83-ee88-42cd-8d94-9502ee38bfab-metrics-certs" pod "c1bf5a83-ee88-42cd-8d94-9502ee38bfab" (UID: "c1bf5a83-ee88-42cd-8d94-9502ee38bfab"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:42:36.219363 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:42:36.217394    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c1bf5a83-ee88-42cd-8d94-9502ee38bfab-maas-api-tls" pod "c1bf5a83-ee88-42cd-8d94-9502ee38bfab" (UID: "c1bf5a83-ee88-42cd-8d94-9502ee38bfab"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:42:36.219363 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:42:36.218857    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c1bf5a83-ee88-42cd-8d94-9502ee38bfab-kube-api-access-ps8kg" pod "c1bf5a83-ee88-42cd-8d94-9502ee38bfab" (UID: "c1bf5a83-ee88-42cd-8d94-9502ee38bfab"). InnerVolumeSpecName "kube-api-access-ps8kg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:42:36.313816 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:42:36.313778    4964 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ps8kg\" (UniqueName: \"kubernetes.io/projected/c1bf5a83-ee88-42cd-8d94-9502ee38bfab-kube-api-access-ps8kg\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:42:36.313816 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:42:36.313815    4964 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c1bf5a83-ee88-42cd-8d94-9502ee38bfab-maas-api-tls\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:42:36.314021 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:42:36.313830    4964 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c1bf5a83-ee88-42cd-8d94-9502ee38bfab-metrics-certs\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:42:36.314021 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:42:36.313847    4964 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c1bf5a83-ee88-42cd-8d94-9502ee38bfab-openshift-service-ca\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:42:36.534311 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:42:36.534272    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:42:36.671501 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:42:36.671467    4964 generic.go:356] "Generic (PLEG): container finished" podID="c1bf5a83-ee88-42cd-8d94-9502ee38bfab" containerID="3ad371415279d453f03782c26ec2e6dd497b1581ca74afa9164f78c47f83a80d" exitCode=0
Oct 31 01:42:36.671671 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:42:36.671514    4964 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-99c8bd-5dfcd8b544-q74wc"
Oct 31 01:42:36.671671 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:42:36.671546    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-99c8bd-5dfcd8b544-q74wc" event={"ID":"c1bf5a83-ee88-42cd-8d94-9502ee38bfab","Type":"ContainerDied","Data":"3ad371415279d453f03782c26ec2e6dd497b1581ca74afa9164f78c47f83a80d"}
Oct 31 01:42:36.671671 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:42:36.671609    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-99c8bd-5dfcd8b544-q74wc" event={"ID":"c1bf5a83-ee88-42cd-8d94-9502ee38bfab","Type":"ContainerDied","Data":"6506e545a2415669dad20c3a67dd16524f83cbefbd6063da238ae74d24d850e3"}
Oct 31 01:42:36.671671 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:42:36.671632    4964 scope.go:122] "RemoveContainer" containerID="3ad371415279d453f03782c26ec2e6dd497b1581ca74afa9164f78c47f83a80d"
Oct 31 01:42:36.683043 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:42:36.683022    4964 scope.go:122] "RemoveContainer" containerID="3ad371415279d453f03782c26ec2e6dd497b1581ca74afa9164f78c47f83a80d"
Oct 31 01:42:36.683345 ip-10-0-133-197 kubenswrapper[4964]: E1031 01:42:36.683303    4964 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3ad371415279d453f03782c26ec2e6dd497b1581ca74afa9164f78c47f83a80d\": container with ID starting with 3ad371415279d453f03782c26ec2e6dd497b1581ca74afa9164f78c47f83a80d not found: ID does not exist" containerID="3ad371415279d453f03782c26ec2e6dd497b1581ca74afa9164f78c47f83a80d"
Oct 31 01:42:36.683405 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:42:36.683355    4964 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3ad371415279d453f03782c26ec2e6dd497b1581ca74afa9164f78c47f83a80d"} err="failed to get container status \"3ad371415279d453f03782c26ec2e6dd497b1581ca74afa9164f78c47f83a80d\": rpc error: code = NotFound desc = could not find container \"3ad371415279d453f03782c26ec2e6dd497b1581ca74afa9164f78c47f83a80d\": container with ID starting with 3ad371415279d453f03782c26ec2e6dd497b1581ca74afa9164f78c47f83a80d not found: ID does not exist"
Oct 31 01:42:36.742524 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:42:36.742491    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:42:36.854612 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:42:36.850394    4964 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-99c8bd-5dfcd8b544-q74wc"]
Oct 31 01:42:36.854612 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:42:36.852222    4964 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-99c8bd-5dfcd8b544-q74wc"]
Oct 31 01:42:37.581743 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:42:37.581708    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:42:37.777579 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:42:37.777550    4964 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c1bf5a83-ee88-42cd-8d94-9502ee38bfab" path="/var/lib/kubelet/pods/c1bf5a83-ee88-42cd-8d94-9502ee38bfab/volumes"
Oct 31 01:42:37.859284 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:42:37.858444    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:42:38.067291 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:42:38.067251    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:42:38.610815 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:42:38.610777    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:42:41.239761 ip-10-0-133-197 kubenswrapper[4964]: E1031 01:42:41.239727    4964 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q56x5" containerName="extractor"
Oct 31 01:42:42.012518 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:42:42.012479    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:42:42.531969 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:42:42.531935    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:42:44.611504 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:42:44.611467    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:42:44.850889 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:42:44.850859    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:42:45.048339 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:42:45.047862    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:42:45.327847 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:42:45.327747    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:42:50.521542 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:42:50.521508    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:42:55.812896 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:42:55.812860    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:42:59.336903 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:42:59.336866    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:42:59.829146 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:42:59.829086    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:43:02.522877 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:43:02.522747    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:43:02.901307 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:43:02.901208    4964 generic.go:356] "Generic (PLEG): container finished" podID="710b2ea2-bffd-45b2-b0fc-0e5a90defed1" containerID="f974e97c15eabf759974ed24a834e0f80d2dc04c22051c94739c2369d33357fb" exitCode=0
Oct 31 01:43:02.901307 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:43:02.901237    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-b2d49b-openshift-default-d7d8667b7-69jgz" event={"ID":"710b2ea2-bffd-45b2-b0fc-0e5a90defed1","Type":"ContainerDied","Data":"f974e97c15eabf759974ed24a834e0f80d2dc04c22051c94739c2369d33357fb"}
Oct 31 01:43:02.901775 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:43:02.901755    4964 scope.go:122] "RemoveContainer" containerID="f974e97c15eabf759974ed24a834e0f80d2dc04c22051c94739c2369d33357fb"
Oct 31 01:43:02.903646 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:43:02.903616    4964 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 01:43:02.903767 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:43:02.903698    4964 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 01:43:02.903767 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:43:02.903749    4964 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 01:43:03.923425 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:43:03.923062    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-b2d49b-openshift-default-d7d8667b7-69jgz" event={"ID":"710b2ea2-bffd-45b2-b0fc-0e5a90defed1","Type":"ContainerStarted","Data":"572ce9e7a635c5c4063309cc43dbe0dc8a8d201675a8db0c67b5c4e6237af0d0"}
Oct 31 01:43:04.212817 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:43:04.212674    4964 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-b2d49b-openshift-default-d7d8667b7-69jgz"
Oct 31 01:43:04.217791 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:43:04.217456    4964 patch_prober.go:29] interesting pod/e2e-ipp-b-b2d49b-openshift-default-d7d8667b7-69jgz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" start-of-body=
Oct 31 01:43:04.217791 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:43:04.217517    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-b2d49b-openshift-default-d7d8667b7-69jgz" podUID="710b2ea2-bffd-45b2-b0fc-0e5a90defed1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused"
Oct 31 01:43:05.212343 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:43:05.211758    4964 patch_prober.go:29] interesting pod/e2e-ipp-b-b2d49b-openshift-default-d7d8667b7-69jgz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" start-of-body=
Oct 31 01:43:05.212343 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:43:05.211822    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-b2d49b-openshift-default-d7d8667b7-69jgz" podUID="710b2ea2-bffd-45b2-b0fc-0e5a90defed1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused"
Oct 31 01:43:05.829443 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:43:05.829405    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-b2d49b-openshift-default-d7d8667b7-69jgz"
Oct 31 01:43:06.211399 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:43:06.209650    4964 patch_prober.go:29] interesting pod/e2e-ipp-b-b2d49b-openshift-default-d7d8667b7-69jgz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" start-of-body=
Oct 31 01:43:06.211399 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:43:06.209712    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-b2d49b-openshift-default-d7d8667b7-69jgz" podUID="710b2ea2-bffd-45b2-b0fc-0e5a90defed1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused"
Oct 31 01:43:07.216346 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:43:07.212664    4964 patch_prober.go:29] interesting pod/e2e-ipp-b-b2d49b-openshift-default-d7d8667b7-69jgz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" start-of-body=
Oct 31 01:43:07.216346 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:43:07.212727    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-b2d49b-openshift-default-d7d8667b7-69jgz" podUID="710b2ea2-bffd-45b2-b0fc-0e5a90defed1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused"
Oct 31 01:43:07.899311 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:43:07.899277    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:43:08.210831 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:43:08.210722    4964 patch_prober.go:29] interesting pod/e2e-ipp-b-b2d49b-openshift-default-d7d8667b7-69jgz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" start-of-body=
Oct 31 01:43:08.210831 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:43:08.210808    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-b2d49b-openshift-default-d7d8667b7-69jgz" podUID="710b2ea2-bffd-45b2-b0fc-0e5a90defed1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused"
Oct 31 01:43:09.212944 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:43:09.212902    4964 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-b2d49b-openshift-default-d7d8667b7-69jgz"
Oct 31 01:43:09.214007 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:43:09.213981    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-b2d49b-openshift-default-d7d8667b7-69jgz"
Oct 31 01:43:11.241084 ip-10-0-133-197 kubenswrapper[4964]: E1031 01:43:11.241051    4964 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q56x5" containerName="extractor"
Oct 31 01:43:11.735642 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:43:11.735601    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:43:13.085706 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:43:13.085662    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:43:13.756055 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:43:13.756025    4964 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-3d3023-f97fd979b-b7trx"]
Oct 31 01:43:13.756265 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:43:13.756250    4964 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3d3023-f97fd979b-b7trx" podUID="26492a66-0b7e-4fe0-a2e1-a52b345c12e7" containerName="maas-api" containerID="cri-o://98b079983ed765fda422c2fde7b30d74757fa0d8d3cc47e82de9f616926706fb" gracePeriod=30
Oct 31 01:43:13.904282 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:43:13.904241    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:43:14.001674 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:43:14.001639    4964 generic.go:356] "Generic (PLEG): container finished" podID="26492a66-0b7e-4fe0-a2e1-a52b345c12e7" containerID="98b079983ed765fda422c2fde7b30d74757fa0d8d3cc47e82de9f616926706fb" exitCode=0
Oct 31 01:43:14.001836 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:43:14.001704    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3d3023-f97fd979b-b7trx" event={"ID":"26492a66-0b7e-4fe0-a2e1-a52b345c12e7","Type":"ContainerDied","Data":"98b079983ed765fda422c2fde7b30d74757fa0d8d3cc47e82de9f616926706fb"}
Oct 31 01:43:14.047887 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:43:14.047856    4964 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3d3023-f97fd979b-b7trx"
Oct 31 01:43:14.082080 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:43:14.082048    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:43:14.098894 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:43:14.098867    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/26492a66-0b7e-4fe0-a2e1-a52b345c12e7-metrics-certs\" (UniqueName: \"kubernetes.io/secret/26492a66-0b7e-4fe0-a2e1-a52b345c12e7-metrics-certs\") pod \"26492a66-0b7e-4fe0-a2e1-a52b345c12e7\" (UID: \"26492a66-0b7e-4fe0-a2e1-a52b345c12e7\") "
Oct 31 01:43:14.099293 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:43:14.098920    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/26492a66-0b7e-4fe0-a2e1-a52b345c12e7-kube-api-access-xqcfz\" (UniqueName: \"kubernetes.io/projected/26492a66-0b7e-4fe0-a2e1-a52b345c12e7-kube-api-access-xqcfz\") pod \"26492a66-0b7e-4fe0-a2e1-a52b345c12e7\" (UID: \"26492a66-0b7e-4fe0-a2e1-a52b345c12e7\") "
Oct 31 01:43:14.099293 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:43:14.099013    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/26492a66-0b7e-4fe0-a2e1-a52b345c12e7-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/26492a66-0b7e-4fe0-a2e1-a52b345c12e7-maas-api-tls\") pod \"26492a66-0b7e-4fe0-a2e1-a52b345c12e7\" (UID: \"26492a66-0b7e-4fe0-a2e1-a52b345c12e7\") "
Oct 31 01:43:14.099293 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:43:14.099171    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/26492a66-0b7e-4fe0-a2e1-a52b345c12e7-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/26492a66-0b7e-4fe0-a2e1-a52b345c12e7-openshift-service-ca\") pod \"26492a66-0b7e-4fe0-a2e1-a52b345c12e7\" (UID: \"26492a66-0b7e-4fe0-a2e1-a52b345c12e7\") "
Oct 31 01:43:14.099661 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:43:14.099632    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/26492a66-0b7e-4fe0-a2e1-a52b345c12e7-openshift-service-ca" pod "26492a66-0b7e-4fe0-a2e1-a52b345c12e7" (UID: "26492a66-0b7e-4fe0-a2e1-a52b345c12e7"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:43:14.101245 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:43:14.101221    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/26492a66-0b7e-4fe0-a2e1-a52b345c12e7-metrics-certs" pod "26492a66-0b7e-4fe0-a2e1-a52b345c12e7" (UID: "26492a66-0b7e-4fe0-a2e1-a52b345c12e7"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:43:14.101856 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:43:14.101834    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/26492a66-0b7e-4fe0-a2e1-a52b345c12e7-maas-api-tls" pod "26492a66-0b7e-4fe0-a2e1-a52b345c12e7" (UID: "26492a66-0b7e-4fe0-a2e1-a52b345c12e7"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:43:14.101943 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:43:14.101863    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/26492a66-0b7e-4fe0-a2e1-a52b345c12e7-kube-api-access-xqcfz" pod "26492a66-0b7e-4fe0-a2e1-a52b345c12e7" (UID: "26492a66-0b7e-4fe0-a2e1-a52b345c12e7"). InnerVolumeSpecName "kube-api-access-xqcfz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:43:14.200781 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:43:14.200741    4964 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/26492a66-0b7e-4fe0-a2e1-a52b345c12e7-metrics-certs\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:43:14.200781 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:43:14.200771    4964 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xqcfz\" (UniqueName: \"kubernetes.io/projected/26492a66-0b7e-4fe0-a2e1-a52b345c12e7-kube-api-access-xqcfz\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:43:14.200781 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:43:14.200782    4964 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/26492a66-0b7e-4fe0-a2e1-a52b345c12e7-maas-api-tls\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:43:14.200781 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:43:14.200790    4964 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/26492a66-0b7e-4fe0-a2e1-a52b345c12e7-openshift-service-ca\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:43:14.413731 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:43:14.413645    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:43:14.646179 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:43:14.646139    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:43:14.953038 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:43:14.953004    4964 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-9e5368-565c76c-jrl5z"]
Oct 31 01:43:14.953266 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:43:14.953184    4964 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-9e5368-565c76c-jrl5z" podUID="c2c5cd7c-7776-4e22-b91b-34fd4e71da5d" containerName="maas-api" containerID="cri-o://57ec4f181bb5826be30b5ec62c2ca406344c851af9678c8ef1c23990232c45c0" gracePeriod=30
Oct 31 01:43:15.016396 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:43:15.016363    4964 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3d3023-f97fd979b-b7trx"
Oct 31 01:43:15.016578 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:43:15.016359    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3d3023-f97fd979b-b7trx" event={"ID":"26492a66-0b7e-4fe0-a2e1-a52b345c12e7","Type":"ContainerDied","Data":"a67b731eaef5dde9e931aa0062ab3492c1b5e9941c345dcc35ed9876e8f9e74b"}
Oct 31 01:43:15.016578 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:43:15.016521    4964 scope.go:122] "RemoveContainer" containerID="98b079983ed765fda422c2fde7b30d74757fa0d8d3cc47e82de9f616926706fb"
Oct 31 01:43:15.108147 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:43:15.108111    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:43:15.192183 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:43:15.192114    4964 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-3d3023-f97fd979b-b7trx"]
Oct 31 01:43:15.195903 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:43:15.195866    4964 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-3d3023-f97fd979b-b7trx"]
Oct 31 01:43:15.239402 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:43:15.239378    4964 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-9e5368-565c76c-jrl5z"
Oct 31 01:43:15.317208 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:43:15.317179    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c2c5cd7c-7776-4e22-b91b-34fd4e71da5d-metrics-certs\" (UniqueName: \"kubernetes.io/secret/c2c5cd7c-7776-4e22-b91b-34fd4e71da5d-metrics-certs\") pod \"c2c5cd7c-7776-4e22-b91b-34fd4e71da5d\" (UID: \"c2c5cd7c-7776-4e22-b91b-34fd4e71da5d\") "
Oct 31 01:43:15.317464 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:43:15.317255    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c2c5cd7c-7776-4e22-b91b-34fd4e71da5d-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c2c5cd7c-7776-4e22-b91b-34fd4e71da5d-maas-api-tls\") pod \"c2c5cd7c-7776-4e22-b91b-34fd4e71da5d\" (UID: \"c2c5cd7c-7776-4e22-b91b-34fd4e71da5d\") "
Oct 31 01:43:15.317464 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:43:15.317302    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c2c5cd7c-7776-4e22-b91b-34fd4e71da5d-kube-api-access-c5mjk\" (UniqueName: \"kubernetes.io/projected/c2c5cd7c-7776-4e22-b91b-34fd4e71da5d-kube-api-access-c5mjk\") pod \"c2c5cd7c-7776-4e22-b91b-34fd4e71da5d\" (UID: \"c2c5cd7c-7776-4e22-b91b-34fd4e71da5d\") "
Oct 31 01:43:15.317464 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:43:15.317435    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c2c5cd7c-7776-4e22-b91b-34fd4e71da5d-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c2c5cd7c-7776-4e22-b91b-34fd4e71da5d-openshift-service-ca\") pod \"c2c5cd7c-7776-4e22-b91b-34fd4e71da5d\" (UID: \"c2c5cd7c-7776-4e22-b91b-34fd4e71da5d\") "
Oct 31 01:43:15.317858 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:43:15.317825    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c2c5cd7c-7776-4e22-b91b-34fd4e71da5d-openshift-service-ca" pod "c2c5cd7c-7776-4e22-b91b-34fd4e71da5d" (UID: "c2c5cd7c-7776-4e22-b91b-34fd4e71da5d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:43:15.317977 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:43:15.317923    4964 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c2c5cd7c-7776-4e22-b91b-34fd4e71da5d-openshift-service-ca\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:43:15.319694 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:43:15.319670    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c2c5cd7c-7776-4e22-b91b-34fd4e71da5d-kube-api-access-c5mjk" pod "c2c5cd7c-7776-4e22-b91b-34fd4e71da5d" (UID: "c2c5cd7c-7776-4e22-b91b-34fd4e71da5d"). InnerVolumeSpecName "kube-api-access-c5mjk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:43:15.320070 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:43:15.320053    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c2c5cd7c-7776-4e22-b91b-34fd4e71da5d-maas-api-tls" pod "c2c5cd7c-7776-4e22-b91b-34fd4e71da5d" (UID: "c2c5cd7c-7776-4e22-b91b-34fd4e71da5d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:43:15.320137 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:43:15.320077    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c2c5cd7c-7776-4e22-b91b-34fd4e71da5d-metrics-certs" pod "c2c5cd7c-7776-4e22-b91b-34fd4e71da5d" (UID: "c2c5cd7c-7776-4e22-b91b-34fd4e71da5d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:43:15.418941 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:43:15.418905    4964 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c2c5cd7c-7776-4e22-b91b-34fd4e71da5d-metrics-certs\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:43:15.418941 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:43:15.418936    4964 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c2c5cd7c-7776-4e22-b91b-34fd4e71da5d-maas-api-tls\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:43:15.418941 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:43:15.418947    4964 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c5mjk\" (UniqueName: \"kubernetes.io/projected/c2c5cd7c-7776-4e22-b91b-34fd4e71da5d-kube-api-access-c5mjk\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:43:15.464624 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:43:15.464544    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:43:15.659275 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:43:15.659244    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:43:15.778013 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:43:15.777970    4964 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="26492a66-0b7e-4fe0-a2e1-a52b345c12e7" path="/var/lib/kubelet/pods/26492a66-0b7e-4fe0-a2e1-a52b345c12e7/volumes"
Oct 31 01:43:15.837885 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:43:15.837847    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:43:16.035410 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:43:16.035300    4964 generic.go:356] "Generic (PLEG): container finished" podID="c2c5cd7c-7776-4e22-b91b-34fd4e71da5d" containerID="57ec4f181bb5826be30b5ec62c2ca406344c851af9678c8ef1c23990232c45c0" exitCode=0
Oct 31 01:43:16.035410 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:43:16.035347    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-9e5368-565c76c-jrl5z" event={"ID":"c2c5cd7c-7776-4e22-b91b-34fd4e71da5d","Type":"ContainerDied","Data":"57ec4f181bb5826be30b5ec62c2ca406344c851af9678c8ef1c23990232c45c0"}
Oct 31 01:43:16.035410 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:43:16.035378    4964 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-9e5368-565c76c-jrl5z"
Oct 31 01:43:16.035410 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:43:16.035400    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-9e5368-565c76c-jrl5z" event={"ID":"c2c5cd7c-7776-4e22-b91b-34fd4e71da5d","Type":"ContainerDied","Data":"aac449051b7d640f1d5870684b81b059d47e41153e95dfb0c25c84c250e65d79"}
Oct 31 01:43:16.035760 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:43:16.035427    4964 scope.go:122] "RemoveContainer" containerID="57ec4f181bb5826be30b5ec62c2ca406344c851af9678c8ef1c23990232c45c0"
Oct 31 01:43:16.056861 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:43:16.056668    4964 scope.go:122] "RemoveContainer" containerID="57ec4f181bb5826be30b5ec62c2ca406344c851af9678c8ef1c23990232c45c0"
Oct 31 01:43:16.057942 ip-10-0-133-197 kubenswrapper[4964]: E1031 01:43:16.057302    4964 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"57ec4f181bb5826be30b5ec62c2ca406344c851af9678c8ef1c23990232c45c0\": container with ID starting with 57ec4f181bb5826be30b5ec62c2ca406344c851af9678c8ef1c23990232c45c0 not found: ID does not exist" containerID="57ec4f181bb5826be30b5ec62c2ca406344c851af9678c8ef1c23990232c45c0"
Oct 31 01:43:16.057942 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:43:16.057373    4964 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"57ec4f181bb5826be30b5ec62c2ca406344c851af9678c8ef1c23990232c45c0"} err="failed to get container status \"57ec4f181bb5826be30b5ec62c2ca406344c851af9678c8ef1c23990232c45c0\": rpc error: code = NotFound desc = could not find container \"57ec4f181bb5826be30b5ec62c2ca406344c851af9678c8ef1c23990232c45c0\": container with ID starting with 57ec4f181bb5826be30b5ec62c2ca406344c851af9678c8ef1c23990232c45c0 not found: ID does not exist"
Oct 31 01:43:16.201835 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:43:16.201796    4964 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-9e5368-565c76c-jrl5z"]
Oct 31 01:43:16.250934 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:43:16.250900    4964 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-9e5368-565c76c-jrl5z"]
Oct 31 01:43:16.434833 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:43:16.434737    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:43:16.727555 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:43:16.727520    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:43:17.052483 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:43:17.052402    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:43:17.342995 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:43:17.342916    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:43:17.777536 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:43:17.777508    4964 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c2c5cd7c-7776-4e22-b91b-34fd4e71da5d" path="/var/lib/kubelet/pods/c2c5cd7c-7776-4e22-b91b-34fd4e71da5d/volumes"
Oct 31 01:43:22.467032 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:43:22.466996    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:43:22.694167 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:43:22.694129    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:43:22.911507 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:43:22.911472    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:43:23.141356 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:43:23.140691    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:43:27.814358 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:43:27.813368    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:43:28.027307 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:43:28.027270    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:43:33.053463 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:43:33.053378    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:43:34.713833 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:43:34.713802    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:43:35.994651 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:43:35.994610    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:43:36.238571 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:43:36.238535    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:43:38.041514 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:43:38.041369    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:43:41.244265 ip-10-0-133-197 kubenswrapper[4964]: E1031 01:43:41.244209    4964 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q56x5" containerName="extractor"
Oct 31 01:43:41.258361 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:43:41.258336    4964 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-71bc98-7fbd6bc889-kc2f4"]
Oct 31 01:43:41.258588 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:43:41.258564    4964 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-71bc98-7fbd6bc889-kc2f4" podUID="95f88410-b151-4896-af94-ab455dfc8ad2" containerName="maas-api" containerID="cri-o://c79b5eb636d9b814a65a6ae238c71056061895b813fb44067e89f857fd8ec3ef" gracePeriod=30
Oct 31 01:43:41.527145 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:43:41.527114    4964 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-71bc98-7fbd6bc889-kc2f4"
Oct 31 01:43:41.566341 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:43:41.566290    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/95f88410-b151-4896-af94-ab455dfc8ad2-kube-api-access-rg7j6\" (UniqueName: \"kubernetes.io/projected/95f88410-b151-4896-af94-ab455dfc8ad2-kube-api-access-rg7j6\") pod \"95f88410-b151-4896-af94-ab455dfc8ad2\" (UID: \"95f88410-b151-4896-af94-ab455dfc8ad2\") "
Oct 31 01:43:41.566472 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:43:41.566451    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/95f88410-b151-4896-af94-ab455dfc8ad2-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/95f88410-b151-4896-af94-ab455dfc8ad2-maas-api-tls\") pod \"95f88410-b151-4896-af94-ab455dfc8ad2\" (UID: \"95f88410-b151-4896-af94-ab455dfc8ad2\") "
Oct 31 01:43:41.566519 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:43:41.566505    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/95f88410-b151-4896-af94-ab455dfc8ad2-metrics-certs\" (UniqueName: \"kubernetes.io/secret/95f88410-b151-4896-af94-ab455dfc8ad2-metrics-certs\") pod \"95f88410-b151-4896-af94-ab455dfc8ad2\" (UID: \"95f88410-b151-4896-af94-ab455dfc8ad2\") "
Oct 31 01:43:41.566646 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:43:41.566621    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/95f88410-b151-4896-af94-ab455dfc8ad2-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/95f88410-b151-4896-af94-ab455dfc8ad2-openshift-service-ca\") pod \"95f88410-b151-4896-af94-ab455dfc8ad2\" (UID: \"95f88410-b151-4896-af94-ab455dfc8ad2\") "
Oct 31 01:43:41.566945 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:43:41.566919    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/95f88410-b151-4896-af94-ab455dfc8ad2-openshift-service-ca" pod "95f88410-b151-4896-af94-ab455dfc8ad2" (UID: "95f88410-b151-4896-af94-ab455dfc8ad2"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:43:41.567128 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:43:41.567075    4964 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/95f88410-b151-4896-af94-ab455dfc8ad2-openshift-service-ca\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:43:41.568540 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:43:41.568515    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/95f88410-b151-4896-af94-ab455dfc8ad2-kube-api-access-rg7j6" pod "95f88410-b151-4896-af94-ab455dfc8ad2" (UID: "95f88410-b151-4896-af94-ab455dfc8ad2"). InnerVolumeSpecName "kube-api-access-rg7j6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:43:41.568735 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:43:41.568714    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/95f88410-b151-4896-af94-ab455dfc8ad2-maas-api-tls" pod "95f88410-b151-4896-af94-ab455dfc8ad2" (UID: "95f88410-b151-4896-af94-ab455dfc8ad2"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:43:41.568802 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:43:41.568740    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/95f88410-b151-4896-af94-ab455dfc8ad2-metrics-certs" pod "95f88410-b151-4896-af94-ab455dfc8ad2" (UID: "95f88410-b151-4896-af94-ab455dfc8ad2"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:43:41.668522 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:43:41.668490    4964 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rg7j6\" (UniqueName: \"kubernetes.io/projected/95f88410-b151-4896-af94-ab455dfc8ad2-kube-api-access-rg7j6\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:43:41.668522 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:43:41.668523    4964 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/95f88410-b151-4896-af94-ab455dfc8ad2-maas-api-tls\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:43:41.668708 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:43:41.668536    4964 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/95f88410-b151-4896-af94-ab455dfc8ad2-metrics-certs\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:43:42.238808 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:43:42.238776    4964 generic.go:356] "Generic (PLEG): container finished" podID="95f88410-b151-4896-af94-ab455dfc8ad2" containerID="c79b5eb636d9b814a65a6ae238c71056061895b813fb44067e89f857fd8ec3ef" exitCode=0
Oct 31 01:43:42.238994 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:43:42.238830    4964 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-71bc98-7fbd6bc889-kc2f4"
Oct 31 01:43:42.238994 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:43:42.238865    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-71bc98-7fbd6bc889-kc2f4" event={"ID":"95f88410-b151-4896-af94-ab455dfc8ad2","Type":"ContainerDied","Data":"c79b5eb636d9b814a65a6ae238c71056061895b813fb44067e89f857fd8ec3ef"}
Oct 31 01:43:42.238994 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:43:42.238934    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-71bc98-7fbd6bc889-kc2f4" event={"ID":"95f88410-b151-4896-af94-ab455dfc8ad2","Type":"ContainerDied","Data":"25d5aba1ecfe6c2b01d99a835a72692320fc048d953d597bbc72c0a55e9a7bc7"}
Oct 31 01:43:42.238994 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:43:42.238963    4964 scope.go:122] "RemoveContainer" containerID="c79b5eb636d9b814a65a6ae238c71056061895b813fb44067e89f857fd8ec3ef"
Oct 31 01:43:42.249421 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:43:42.249389    4964 scope.go:122] "RemoveContainer" containerID="c79b5eb636d9b814a65a6ae238c71056061895b813fb44067e89f857fd8ec3ef"
Oct 31 01:43:42.249742 ip-10-0-133-197 kubenswrapper[4964]: E1031 01:43:42.249694    4964 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c79b5eb636d9b814a65a6ae238c71056061895b813fb44067e89f857fd8ec3ef\": container with ID starting with c79b5eb636d9b814a65a6ae238c71056061895b813fb44067e89f857fd8ec3ef not found: ID does not exist" containerID="c79b5eb636d9b814a65a6ae238c71056061895b813fb44067e89f857fd8ec3ef"
Oct 31 01:43:42.249742 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:43:42.249719    4964 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c79b5eb636d9b814a65a6ae238c71056061895b813fb44067e89f857fd8ec3ef"} err="failed to get container status \"c79b5eb636d9b814a65a6ae238c71056061895b813fb44067e89f857fd8ec3ef\": rpc error: code = NotFound desc = could not find container \"c79b5eb636d9b814a65a6ae238c71056061895b813fb44067e89f857fd8ec3ef\": container with ID starting with c79b5eb636d9b814a65a6ae238c71056061895b813fb44067e89f857fd8ec3ef not found: ID does not exist"
Oct 31 01:43:42.407777 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:43:42.407747    4964 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-71bc98-7fbd6bc889-kc2f4"]
Oct 31 01:43:42.456632 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:43:42.456605    4964 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-71bc98-7fbd6bc889-kc2f4"]
Oct 31 01:43:43.147376 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:43:43.147345    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:43:43.777340 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:43:43.777290    4964 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="95f88410-b151-4896-af94-ab455dfc8ad2" path="/var/lib/kubelet/pods/95f88410-b151-4896-af94-ab455dfc8ad2/volumes"
Oct 31 01:43:44.344450 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:43:44.344420    4964 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-9e5368-openshift-default-5ff7fc6fd7-fp6pw"]
Oct 31 01:43:44.344655 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:43:44.344635    4964 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-9e5368-openshift-default-5ff7fc6fd7-fp6pw" podUID="5ab8d4bd-eefa-4a6c-8b3a-34593e689827" containerName="istio-proxy" containerID="cri-o://38e6ea22ddbc2d642aac2580828fee2853259d7990f397bb2948d7de2c0e4c11" gracePeriod=30
Oct 31 01:43:44.441345 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:43:44.436935    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:43:49.625679 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:43:49.625581    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:43:49.629263 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:43:49.629239    4964 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-9e5368-openshift-default-5ff7fc6fd7-fp6pw"
Oct 31 01:43:49.772101 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:43:49.772066    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5ab8d4bd-eefa-4a6c-8b3a-34593e689827-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ab8d4bd-eefa-4a6c-8b3a-34593e689827-istio-envoy\") pod \"5ab8d4bd-eefa-4a6c-8b3a-34593e689827\" (UID: \"5ab8d4bd-eefa-4a6c-8b3a-34593e689827\") "
Oct 31 01:43:49.772299 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:43:49.772116    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5ab8d4bd-eefa-4a6c-8b3a-34593e689827-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ab8d4bd-eefa-4a6c-8b3a-34593e689827-workload-socket\") pod \"5ab8d4bd-eefa-4a6c-8b3a-34593e689827\" (UID: \"5ab8d4bd-eefa-4a6c-8b3a-34593e689827\") "
Oct 31 01:43:49.772299 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:43:49.772169    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5ab8d4bd-eefa-4a6c-8b3a-34593e689827-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ab8d4bd-eefa-4a6c-8b3a-34593e689827-istiod-ca-cert\") pod \"5ab8d4bd-eefa-4a6c-8b3a-34593e689827\" (UID: \"5ab8d4bd-eefa-4a6c-8b3a-34593e689827\") "
Oct 31 01:43:49.772299 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:43:49.772200    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5ab8d4bd-eefa-4a6c-8b3a-34593e689827-istio-token\" (UniqueName: \"kubernetes.io/projected/5ab8d4bd-eefa-4a6c-8b3a-34593e689827-istio-token\") pod \"5ab8d4bd-eefa-4a6c-8b3a-34593e689827\" (UID: \"5ab8d4bd-eefa-4a6c-8b3a-34593e689827\") "
Oct 31 01:43:49.772299 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:43:49.772224    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5ab8d4bd-eefa-4a6c-8b3a-34593e689827-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ab8d4bd-eefa-4a6c-8b3a-34593e689827-istio-podinfo\") pod \"5ab8d4bd-eefa-4a6c-8b3a-34593e689827\" (UID: \"5ab8d4bd-eefa-4a6c-8b3a-34593e689827\") "
Oct 31 01:43:49.772299 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:43:49.772252    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5ab8d4bd-eefa-4a6c-8b3a-34593e689827-kube-api-access-27vcq\" (UniqueName: \"kubernetes.io/projected/5ab8d4bd-eefa-4a6c-8b3a-34593e689827-kube-api-access-27vcq\") pod \"5ab8d4bd-eefa-4a6c-8b3a-34593e689827\" (UID: \"5ab8d4bd-eefa-4a6c-8b3a-34593e689827\") "
Oct 31 01:43:49.772606 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:43:49.772306    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5ab8d4bd-eefa-4a6c-8b3a-34593e689827-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ab8d4bd-eefa-4a6c-8b3a-34593e689827-istio-data\") pod \"5ab8d4bd-eefa-4a6c-8b3a-34593e689827\" (UID: \"5ab8d4bd-eefa-4a6c-8b3a-34593e689827\") "
Oct 31 01:43:49.772606 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:43:49.772451    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5ab8d4bd-eefa-4a6c-8b3a-34593e689827-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ab8d4bd-eefa-4a6c-8b3a-34593e689827-credential-socket\") pod \"5ab8d4bd-eefa-4a6c-8b3a-34593e689827\" (UID: \"5ab8d4bd-eefa-4a6c-8b3a-34593e689827\") "
Oct 31 01:43:49.772606 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:43:49.772505    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ab8d4bd-eefa-4a6c-8b3a-34593e689827-workload-socket" pod "5ab8d4bd-eefa-4a6c-8b3a-34593e689827" (UID: "5ab8d4bd-eefa-4a6c-8b3a-34593e689827"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:43:49.772977 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:43:49.772933    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5ab8d4bd-eefa-4a6c-8b3a-34593e689827-istiod-ca-cert" pod "5ab8d4bd-eefa-4a6c-8b3a-34593e689827" (UID: "5ab8d4bd-eefa-4a6c-8b3a-34593e689827"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:43:49.773196 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:43:49.773168    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ab8d4bd-eefa-4a6c-8b3a-34593e689827-istio-data" pod "5ab8d4bd-eefa-4a6c-8b3a-34593e689827" (UID: "5ab8d4bd-eefa-4a6c-8b3a-34593e689827"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:43:49.773301 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:43:49.773251    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5ab8d4bd-eefa-4a6c-8b3a-34593e689827-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ab8d4bd-eefa-4a6c-8b3a-34593e689827-workload-certs\") pod \"5ab8d4bd-eefa-4a6c-8b3a-34593e689827\" (UID: \"5ab8d4bd-eefa-4a6c-8b3a-34593e689827\") "
Oct 31 01:43:49.773413 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:43:49.773387    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ab8d4bd-eefa-4a6c-8b3a-34593e689827-credential-socket" pod "5ab8d4bd-eefa-4a6c-8b3a-34593e689827" (UID: "5ab8d4bd-eefa-4a6c-8b3a-34593e689827"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:43:49.775222 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:43:49.775191    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ab8d4bd-eefa-4a6c-8b3a-34593e689827-istio-envoy" pod "5ab8d4bd-eefa-4a6c-8b3a-34593e689827" (UID: "5ab8d4bd-eefa-4a6c-8b3a-34593e689827"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:43:49.775598 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:43:49.775572    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ab8d4bd-eefa-4a6c-8b3a-34593e689827-workload-certs" pod "5ab8d4bd-eefa-4a6c-8b3a-34593e689827" (UID: "5ab8d4bd-eefa-4a6c-8b3a-34593e689827"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:43:49.784675 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:43:49.777421    4964 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ab8d4bd-eefa-4a6c-8b3a-34593e689827-workload-certs\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:43:49.784675 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:43:49.777450    4964 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ab8d4bd-eefa-4a6c-8b3a-34593e689827-istio-envoy\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:43:49.784675 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:43:49.777475    4964 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ab8d4bd-eefa-4a6c-8b3a-34593e689827-workload-socket\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:43:49.784675 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:43:49.777491    4964 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ab8d4bd-eefa-4a6c-8b3a-34593e689827-istiod-ca-cert\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:43:49.784675 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:43:49.777505    4964 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ab8d4bd-eefa-4a6c-8b3a-34593e689827-istio-data\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:43:49.784675 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:43:49.777521    4964 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ab8d4bd-eefa-4a6c-8b3a-34593e689827-credential-socket\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:43:49.785680 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:43:49.785652    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ab8d4bd-eefa-4a6c-8b3a-34593e689827-kube-api-access-27vcq" pod "5ab8d4bd-eefa-4a6c-8b3a-34593e689827" (UID: "5ab8d4bd-eefa-4a6c-8b3a-34593e689827"). InnerVolumeSpecName "kube-api-access-27vcq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:43:49.785802 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:43:49.785688    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ab8d4bd-eefa-4a6c-8b3a-34593e689827-istio-token" pod "5ab8d4bd-eefa-4a6c-8b3a-34593e689827" (UID: "5ab8d4bd-eefa-4a6c-8b3a-34593e689827"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:43:49.785802 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:43:49.785689    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5ab8d4bd-eefa-4a6c-8b3a-34593e689827-istio-podinfo" pod "5ab8d4bd-eefa-4a6c-8b3a-34593e689827" (UID: "5ab8d4bd-eefa-4a6c-8b3a-34593e689827"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 01:43:49.879579 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:43:49.879489    4964 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5ab8d4bd-eefa-4a6c-8b3a-34593e689827-istio-token\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:43:49.879579 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:43:49.879526    4964 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ab8d4bd-eefa-4a6c-8b3a-34593e689827-istio-podinfo\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:43:49.879579 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:43:49.879537    4964 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-27vcq\" (UniqueName: \"kubernetes.io/projected/5ab8d4bd-eefa-4a6c-8b3a-34593e689827-kube-api-access-27vcq\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:43:50.307191 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:43:50.307154    4964 generic.go:356] "Generic (PLEG): container finished" podID="5ab8d4bd-eefa-4a6c-8b3a-34593e689827" containerID="38e6ea22ddbc2d642aac2580828fee2853259d7990f397bb2948d7de2c0e4c11" exitCode=0
Oct 31 01:43:50.307407 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:43:50.307222    4964 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-9e5368-openshift-default-5ff7fc6fd7-fp6pw"
Oct 31 01:43:50.307407 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:43:50.307225    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-9e5368-openshift-default-5ff7fc6fd7-fp6pw" event={"ID":"5ab8d4bd-eefa-4a6c-8b3a-34593e689827","Type":"ContainerDied","Data":"38e6ea22ddbc2d642aac2580828fee2853259d7990f397bb2948d7de2c0e4c11"}
Oct 31 01:43:50.307547 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:43:50.307409    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-9e5368-openshift-default-5ff7fc6fd7-fp6pw" event={"ID":"5ab8d4bd-eefa-4a6c-8b3a-34593e689827","Type":"ContainerDied","Data":"44e65863854dbee31a057d437c33450d426f61144222f7bc849fee49347591dc"}
Oct 31 01:43:50.307547 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:43:50.307446    4964 scope.go:122] "RemoveContainer" containerID="38e6ea22ddbc2d642aac2580828fee2853259d7990f397bb2948d7de2c0e4c11"
Oct 31 01:43:50.317844 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:43:50.317825    4964 scope.go:122] "RemoveContainer" containerID="84b91dfbaff657c035aaadf786b3fadae6fc10b902a4346df8a0db03a6b1abcd"
Oct 31 01:43:50.327903 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:43:50.327882    4964 scope.go:122] "RemoveContainer" containerID="38e6ea22ddbc2d642aac2580828fee2853259d7990f397bb2948d7de2c0e4c11"
Oct 31 01:43:50.328252 ip-10-0-133-197 kubenswrapper[4964]: E1031 01:43:50.328230    4964 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"38e6ea22ddbc2d642aac2580828fee2853259d7990f397bb2948d7de2c0e4c11\": container with ID starting with 38e6ea22ddbc2d642aac2580828fee2853259d7990f397bb2948d7de2c0e4c11 not found: ID does not exist" containerID="38e6ea22ddbc2d642aac2580828fee2853259d7990f397bb2948d7de2c0e4c11"
Oct 31 01:43:50.328361 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:43:50.328262    4964 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"38e6ea22ddbc2d642aac2580828fee2853259d7990f397bb2948d7de2c0e4c11"} err="failed to get container status \"38e6ea22ddbc2d642aac2580828fee2853259d7990f397bb2948d7de2c0e4c11\": rpc error: code = NotFound desc = could not find container \"38e6ea22ddbc2d642aac2580828fee2853259d7990f397bb2948d7de2c0e4c11\": container with ID starting with 38e6ea22ddbc2d642aac2580828fee2853259d7990f397bb2948d7de2c0e4c11 not found: ID does not exist"
Oct 31 01:43:50.328361 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:43:50.328281    4964 scope.go:122] "RemoveContainer" containerID="84b91dfbaff657c035aaadf786b3fadae6fc10b902a4346df8a0db03a6b1abcd"
Oct 31 01:43:50.328586 ip-10-0-133-197 kubenswrapper[4964]: E1031 01:43:50.328568    4964 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"84b91dfbaff657c035aaadf786b3fadae6fc10b902a4346df8a0db03a6b1abcd\": container with ID starting with 84b91dfbaff657c035aaadf786b3fadae6fc10b902a4346df8a0db03a6b1abcd not found: ID does not exist" containerID="84b91dfbaff657c035aaadf786b3fadae6fc10b902a4346df8a0db03a6b1abcd"
Oct 31 01:43:50.328646 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:43:50.328597    4964 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"84b91dfbaff657c035aaadf786b3fadae6fc10b902a4346df8a0db03a6b1abcd"} err="failed to get container status \"84b91dfbaff657c035aaadf786b3fadae6fc10b902a4346df8a0db03a6b1abcd\": rpc error: code = NotFound desc = could not find container \"84b91dfbaff657c035aaadf786b3fadae6fc10b902a4346df8a0db03a6b1abcd\": container with ID starting with 84b91dfbaff657c035aaadf786b3fadae6fc10b902a4346df8a0db03a6b1abcd not found: ID does not exist"
Oct 31 01:43:50.524091 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:43:50.524053    4964 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-9e5368-openshift-default-5ff7fc6fd7-fp6pw"]
Oct 31 01:43:50.570818 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:43:50.570744    4964 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-9e5368-openshift-default-5ff7fc6fd7-fp6pw"]
Oct 31 01:43:51.778867 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:43:51.778827    4964 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5ab8d4bd-eefa-4a6c-8b3a-34593e689827" path="/var/lib/kubelet/pods/5ab8d4bd-eefa-4a6c-8b3a-34593e689827/volumes"
Oct 31 01:43:52.133966 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:43:52.133879    4964 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-9257n_f0c0ada7-7688-4dc2-97fc-813e4d0c4ef3/console-operator/2.log"
Oct 31 01:43:52.141116 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:43:52.141093    4964 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-9257n_f0c0ada7-7688-4dc2-97fc-813e4d0c4ef3/console-operator/2.log"
Oct 31 01:43:54.023765 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:43:54.023726    4964 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-df5d0b-6f9767f49c-gxnsd"]
Oct 31 01:43:54.024288 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:43:54.023987    4964 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-df5d0b-6f9767f49c-gxnsd" podUID="70860802-a6eb-4d36-a11d-77f9c006a93a" containerName="maas-api" containerID="cri-o://a5871846f0d6622ac926109861ae68e725c79d0dc8292fa2e3b809d2be05b5b3" gracePeriod=30
Oct 31 01:43:54.208584 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:43:54.208551    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:43:54.336540 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:43:54.336508    4964 generic.go:356] "Generic (PLEG): container finished" podID="70860802-a6eb-4d36-a11d-77f9c006a93a" containerID="a5871846f0d6622ac926109861ae68e725c79d0dc8292fa2e3b809d2be05b5b3" exitCode=0
Oct 31 01:43:54.336691 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:43:54.336581    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-df5d0b-6f9767f49c-gxnsd" event={"ID":"70860802-a6eb-4d36-a11d-77f9c006a93a","Type":"ContainerDied","Data":"a5871846f0d6622ac926109861ae68e725c79d0dc8292fa2e3b809d2be05b5b3"}
Oct 31 01:43:54.336691 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:43:54.336634    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-df5d0b-6f9767f49c-gxnsd" event={"ID":"70860802-a6eb-4d36-a11d-77f9c006a93a","Type":"ContainerDied","Data":"b7cc86bffe62dbb4354f34dc345d91fe762b7e0f15768894cc6fa9f468831c50"}
Oct 31 01:43:54.336691 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:43:54.336652    4964 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b7cc86bffe62dbb4354f34dc345d91fe762b7e0f15768894cc6fa9f468831c50"
Oct 31 01:43:54.344602 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:43:54.344581    4964 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-df5d0b-6f9767f49c-gxnsd"
Oct 31 01:43:54.441392 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:43:54.436854    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/70860802-a6eb-4d36-a11d-77f9c006a93a-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/70860802-a6eb-4d36-a11d-77f9c006a93a-openshift-service-ca\") pod \"70860802-a6eb-4d36-a11d-77f9c006a93a\" (UID: \"70860802-a6eb-4d36-a11d-77f9c006a93a\") "
Oct 31 01:43:54.441392 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:43:54.436959    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/70860802-a6eb-4d36-a11d-77f9c006a93a-kube-api-access-hbjjs\" (UniqueName: \"kubernetes.io/projected/70860802-a6eb-4d36-a11d-77f9c006a93a-kube-api-access-hbjjs\") pod \"70860802-a6eb-4d36-a11d-77f9c006a93a\" (UID: \"70860802-a6eb-4d36-a11d-77f9c006a93a\") "
Oct 31 01:43:54.441392 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:43:54.437012    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/70860802-a6eb-4d36-a11d-77f9c006a93a-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/70860802-a6eb-4d36-a11d-77f9c006a93a-maas-api-tls\") pod \"70860802-a6eb-4d36-a11d-77f9c006a93a\" (UID: \"70860802-a6eb-4d36-a11d-77f9c006a93a\") "
Oct 31 01:43:54.441392 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:43:54.437046    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/70860802-a6eb-4d36-a11d-77f9c006a93a-metrics-certs\" (UniqueName: \"kubernetes.io/secret/70860802-a6eb-4d36-a11d-77f9c006a93a-metrics-certs\") pod \"70860802-a6eb-4d36-a11d-77f9c006a93a\" (UID: \"70860802-a6eb-4d36-a11d-77f9c006a93a\") "
Oct 31 01:43:54.441392 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:43:54.438447    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/70860802-a6eb-4d36-a11d-77f9c006a93a-openshift-service-ca" pod "70860802-a6eb-4d36-a11d-77f9c006a93a" (UID: "70860802-a6eb-4d36-a11d-77f9c006a93a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:43:54.441392 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:43:54.440830    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/70860802-a6eb-4d36-a11d-77f9c006a93a-kube-api-access-hbjjs" pod "70860802-a6eb-4d36-a11d-77f9c006a93a" (UID: "70860802-a6eb-4d36-a11d-77f9c006a93a"). InnerVolumeSpecName "kube-api-access-hbjjs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:43:54.441935 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:43:54.441913    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/70860802-a6eb-4d36-a11d-77f9c006a93a-maas-api-tls" pod "70860802-a6eb-4d36-a11d-77f9c006a93a" (UID: "70860802-a6eb-4d36-a11d-77f9c006a93a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:43:54.444831 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:43:54.444796    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/70860802-a6eb-4d36-a11d-77f9c006a93a-metrics-certs" pod "70860802-a6eb-4d36-a11d-77f9c006a93a" (UID: "70860802-a6eb-4d36-a11d-77f9c006a93a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:43:54.539284 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:43:54.539249    4964 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/70860802-a6eb-4d36-a11d-77f9c006a93a-openshift-service-ca\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:43:54.539284 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:43:54.539286    4964 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hbjjs\" (UniqueName: \"kubernetes.io/projected/70860802-a6eb-4d36-a11d-77f9c006a93a-kube-api-access-hbjjs\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:43:54.539508 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:43:54.539297    4964 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/70860802-a6eb-4d36-a11d-77f9c006a93a-maas-api-tls\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:43:54.539508 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:43:54.539306    4964 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/70860802-a6eb-4d36-a11d-77f9c006a93a-metrics-certs\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:43:54.713859 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:43:54.713724    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:43:54.935060 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:43:54.935021    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:43:55.343914 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:43:55.343883    4964 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-df5d0b-6f9767f49c-gxnsd"
Oct 31 01:43:55.506726 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:43:55.506693    4964 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-df5d0b-6f9767f49c-gxnsd"]
Oct 31 01:43:55.509592 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:43:55.509570    4964 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-df5d0b-6f9767f49c-gxnsd"]
Oct 31 01:43:55.778109 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:43:55.778066    4964 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="70860802-a6eb-4d36-a11d-77f9c006a93a" path="/var/lib/kubelet/pods/70860802-a6eb-4d36-a11d-77f9c006a93a/volumes"
Oct 31 01:43:55.918903 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:43:55.918872    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:43:56.188606 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:43:56.188524    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:43:56.368194 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:43:56.368161    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:43:56.877599 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:43:56.877564    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:43:57.187240 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:43:57.187148    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:43:58.638817 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:43:58.638780    4964 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-b2d49b-b7b76c997-5vm2r"]
Oct 31 01:43:59.043961 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:43:59.043923    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:43:59.227107 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:43:59.227071    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:44:00.000684 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:00.000646    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:44:00.935309 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:00.935282    4964 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-tlb5m_c8461fdb-12e5-4691-8b95-139d5a901f0d/payload-processing/0.log"
Oct 31 01:44:01.215491 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:01.215402    4964 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-tlb5m_c8461fdb-12e5-4691-8b95-139d5a901f0d/payload-processing/0.log"
Oct 31 01:44:01.407709 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:01.407667    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:44:02.560430 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:02.560388    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:44:02.854978 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:02.854886    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:44:03.053920 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:03.053871    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:44:03.640382 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:03.640292    4964 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-b2d49b-b7b76c997-5vm2r" podUID="0cc66d3e-3edd-4399-a970-5ab7d0f6f415" containerName="payload-processing" containerID="cri-o://175a113a54711c378c346221f4d7d2b2f0ce109832211f2a402da22a3ee2850a" gracePeriod=25
Oct 31 01:44:03.917804 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:03.917777    4964 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-b2d49b-b7b76c997-5vm2r"
Oct 31 01:44:03.967794 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:03.967768    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0cc66d3e-3edd-4399-a970-5ab7d0f6f415-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0cc66d3e-3edd-4399-a970-5ab7d0f6f415-plugins-config-volume\") pod \"0cc66d3e-3edd-4399-a970-5ab7d0f6f415\" (UID: \"0cc66d3e-3edd-4399-a970-5ab7d0f6f415\") "
Oct 31 01:44:03.967973 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:03.967906    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0cc66d3e-3edd-4399-a970-5ab7d0f6f415-kube-api-access-4gwdx\" (UniqueName: \"kubernetes.io/projected/0cc66d3e-3edd-4399-a970-5ab7d0f6f415-kube-api-access-4gwdx\") pod \"0cc66d3e-3edd-4399-a970-5ab7d0f6f415\" (UID: \"0cc66d3e-3edd-4399-a970-5ab7d0f6f415\") "
Oct 31 01:44:03.968190 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:03.968161    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0cc66d3e-3edd-4399-a970-5ab7d0f6f415-plugins-config-volume" pod "0cc66d3e-3edd-4399-a970-5ab7d0f6f415" (UID: "0cc66d3e-3edd-4399-a970-5ab7d0f6f415"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:44:03.968389 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:03.968374    4964 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0cc66d3e-3edd-4399-a970-5ab7d0f6f415-plugins-config-volume\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:44:03.970305 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:03.970281    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0cc66d3e-3edd-4399-a970-5ab7d0f6f415-kube-api-access-4gwdx" pod "0cc66d3e-3edd-4399-a970-5ab7d0f6f415" (UID: "0cc66d3e-3edd-4399-a970-5ab7d0f6f415"). InnerVolumeSpecName "kube-api-access-4gwdx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:44:04.070161 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:04.070123    4964 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4gwdx\" (UniqueName: \"kubernetes.io/projected/0cc66d3e-3edd-4399-a970-5ab7d0f6f415-kube-api-access-4gwdx\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:44:04.412869 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:04.412828    4964 generic.go:356] "Generic (PLEG): container finished" podID="0cc66d3e-3edd-4399-a970-5ab7d0f6f415" containerID="175a113a54711c378c346221f4d7d2b2f0ce109832211f2a402da22a3ee2850a" exitCode=0
Oct 31 01:44:04.413050 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:04.412888    4964 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-b2d49b-b7b76c997-5vm2r"
Oct 31 01:44:04.413050 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:04.412900    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-b2d49b-b7b76c997-5vm2r" event={"ID":"0cc66d3e-3edd-4399-a970-5ab7d0f6f415","Type":"ContainerDied","Data":"175a113a54711c378c346221f4d7d2b2f0ce109832211f2a402da22a3ee2850a"}
Oct 31 01:44:04.413050 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:04.412958    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-b2d49b-b7b76c997-5vm2r" event={"ID":"0cc66d3e-3edd-4399-a970-5ab7d0f6f415","Type":"ContainerDied","Data":"29a4442d76a7bedac51dc99b65d332f4179be0a417dc76d44c7416ddfd2a0ac3"}
Oct 31 01:44:04.413050 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:04.412986    4964 scope.go:122] "RemoveContainer" containerID="175a113a54711c378c346221f4d7d2b2f0ce109832211f2a402da22a3ee2850a"
Oct 31 01:44:04.424597 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:04.424575    4964 scope.go:122] "RemoveContainer" containerID="175a113a54711c378c346221f4d7d2b2f0ce109832211f2a402da22a3ee2850a"
Oct 31 01:44:04.424939 ip-10-0-133-197 kubenswrapper[4964]: E1031 01:44:04.424914    4964 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"175a113a54711c378c346221f4d7d2b2f0ce109832211f2a402da22a3ee2850a\": container with ID starting with 175a113a54711c378c346221f4d7d2b2f0ce109832211f2a402da22a3ee2850a not found: ID does not exist" containerID="175a113a54711c378c346221f4d7d2b2f0ce109832211f2a402da22a3ee2850a"
Oct 31 01:44:04.425008 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:04.424953    4964 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"175a113a54711c378c346221f4d7d2b2f0ce109832211f2a402da22a3ee2850a"} err="failed to get container status \"175a113a54711c378c346221f4d7d2b2f0ce109832211f2a402da22a3ee2850a\": rpc error: code = NotFound desc = could not find container \"175a113a54711c378c346221f4d7d2b2f0ce109832211f2a402da22a3ee2850a\": container with ID starting with 175a113a54711c378c346221f4d7d2b2f0ce109832211f2a402da22a3ee2850a not found: ID does not exist"
Oct 31 01:44:04.506572 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:04.506515    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:44:04.578311 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:04.578280    4964 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-b2d49b-b7b76c997-5vm2r"]
Oct 31 01:44:04.628033 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:04.627991    4964 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-b2d49b-b7b76c997-5vm2r"]
Oct 31 01:44:04.972188 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:04.972153    4964 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-a-8a3796-openshift-default-857c657d9-8vlxd"]
Oct 31 01:44:04.972724 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:04.972710    4964 cpu_manager.go:414] "containerMap: removing container" podUID="26492a66-0b7e-4fe0-a2e1-a52b345c12e7" containerName="maas-api"
Oct 31 01:44:04.972777 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:04.972726    4964 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="26492a66-0b7e-4fe0-a2e1-a52b345c12e7" containerName="maas-api"
Oct 31 01:44:04.972777 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:04.972741    4964 cpu_manager.go:414] "containerMap: removing container" podUID="0cc66d3e-3edd-4399-a970-5ab7d0f6f415" containerName="payload-processing"
Oct 31 01:44:04.972777 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:04.972746    4964 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0cc66d3e-3edd-4399-a970-5ab7d0f6f415" containerName="payload-processing"
Oct 31 01:44:04.972777 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:04.972756    4964 cpu_manager.go:414] "containerMap: removing container" podUID="2bf44719-ff6b-4ada-9fe7-a6f466992512" containerName="istio-proxy"
Oct 31 01:44:04.972777 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:04.972762    4964 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2bf44719-ff6b-4ada-9fe7-a6f466992512" containerName="istio-proxy"
Oct 31 01:44:04.972777 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:04.972769    4964 cpu_manager.go:414] "containerMap: removing container" podUID="70860802-a6eb-4d36-a11d-77f9c006a93a" containerName="maas-api"
Oct 31 01:44:04.972777 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:04.972773    4964 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="70860802-a6eb-4d36-a11d-77f9c006a93a" containerName="maas-api"
Oct 31 01:44:04.972777 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:04.972779    4964 cpu_manager.go:414] "containerMap: removing container" podUID="5ab8d4bd-eefa-4a6c-8b3a-34593e689827" containerName="istio-proxy"
Oct 31 01:44:04.973032 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:04.972784    4964 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5ab8d4bd-eefa-4a6c-8b3a-34593e689827" containerName="istio-proxy"
Oct 31 01:44:04.973032 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:04.972798    4964 cpu_manager.go:414] "containerMap: removing container" podUID="c1bf5a83-ee88-42cd-8d94-9502ee38bfab" containerName="maas-api"
Oct 31 01:44:04.973032 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:04.972802    4964 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c1bf5a83-ee88-42cd-8d94-9502ee38bfab" containerName="maas-api"
Oct 31 01:44:04.973032 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:04.972811    4964 cpu_manager.go:414] "containerMap: removing container" podUID="95f88410-b151-4896-af94-ab455dfc8ad2" containerName="maas-api"
Oct 31 01:44:04.973032 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:04.972815    4964 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="95f88410-b151-4896-af94-ab455dfc8ad2" containerName="maas-api"
Oct 31 01:44:04.973032 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:04.972821    4964 cpu_manager.go:414] "containerMap: removing container" podUID="8b94036e-3ae4-4026-9171-2ab95b1db68a" containerName="payload-pre-processing"
Oct 31 01:44:04.973032 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:04.972825    4964 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8b94036e-3ae4-4026-9171-2ab95b1db68a" containerName="payload-pre-processing"
Oct 31 01:44:04.973032 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:04.972838    4964 cpu_manager.go:414] "containerMap: removing container" podUID="197445f1-d262-48ee-87c5-7354132a69ba" containerName="istio-proxy"
Oct 31 01:44:04.973032 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:04.972842    4964 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="197445f1-d262-48ee-87c5-7354132a69ba" containerName="istio-proxy"
Oct 31 01:44:04.973032 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:04.972849    4964 cpu_manager.go:414] "containerMap: removing container" podUID="b8249620-2e20-4e80-a8b8-7d8e566f8973" containerName="istio-proxy"
Oct 31 01:44:04.973032 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:04.972854    4964 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b8249620-2e20-4e80-a8b8-7d8e566f8973" containerName="istio-proxy"
Oct 31 01:44:04.973032 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:04.972864    4964 cpu_manager.go:414] "containerMap: removing container" podUID="197445f1-d262-48ee-87c5-7354132a69ba" containerName="istio-proxy"
Oct 31 01:44:04.973032 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:04.972869    4964 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="197445f1-d262-48ee-87c5-7354132a69ba" containerName="istio-proxy"
Oct 31 01:44:04.973032 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:04.972874    4964 cpu_manager.go:414] "containerMap: removing container" podUID="c2c5cd7c-7776-4e22-b91b-34fd4e71da5d" containerName="maas-api"
Oct 31 01:44:04.973032 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:04.972879    4964 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c2c5cd7c-7776-4e22-b91b-34fd4e71da5d" containerName="maas-api"
Oct 31 01:44:04.973032 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:04.972897    4964 cpu_manager.go:414] "containerMap: removing container" podUID="b8249620-2e20-4e80-a8b8-7d8e566f8973" containerName="istio-proxy"
Oct 31 01:44:04.973032 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:04.972901    4964 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b8249620-2e20-4e80-a8b8-7d8e566f8973" containerName="istio-proxy"
Oct 31 01:44:04.973032 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:04.972977    4964 memory_manager.go:365] "RemoveStaleState removing state" podUID="5ab8d4bd-eefa-4a6c-8b3a-34593e689827" containerName="istio-proxy"
Oct 31 01:44:04.973032 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:04.972987    4964 memory_manager.go:365] "RemoveStaleState removing state" podUID="c1bf5a83-ee88-42cd-8d94-9502ee38bfab" containerName="maas-api"
Oct 31 01:44:04.973032 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:04.972993    4964 memory_manager.go:365] "RemoveStaleState removing state" podUID="5ab8d4bd-eefa-4a6c-8b3a-34593e689827" containerName="istio-proxy"
Oct 31 01:44:04.973032 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:04.972999    4964 memory_manager.go:365] "RemoveStaleState removing state" podUID="95f88410-b151-4896-af94-ab455dfc8ad2" containerName="maas-api"
Oct 31 01:44:04.973032 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:04.973006    4964 memory_manager.go:365] "RemoveStaleState removing state" podUID="8b94036e-3ae4-4026-9171-2ab95b1db68a" containerName="payload-pre-processing"
Oct 31 01:44:04.973032 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:04.973014    4964 memory_manager.go:365] "RemoveStaleState removing state" podUID="197445f1-d262-48ee-87c5-7354132a69ba" containerName="istio-proxy"
Oct 31 01:44:04.973032 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:04.973019    4964 memory_manager.go:365] "RemoveStaleState removing state" podUID="b8249620-2e20-4e80-a8b8-7d8e566f8973" containerName="istio-proxy"
Oct 31 01:44:04.973032 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:04.973024    4964 memory_manager.go:365] "RemoveStaleState removing state" podUID="197445f1-d262-48ee-87c5-7354132a69ba" containerName="istio-proxy"
Oct 31 01:44:04.973032 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:04.973028    4964 memory_manager.go:365] "RemoveStaleState removing state" podUID="c2c5cd7c-7776-4e22-b91b-34fd4e71da5d" containerName="maas-api"
Oct 31 01:44:04.973032 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:04.973037    4964 memory_manager.go:365] "RemoveStaleState removing state" podUID="26492a66-0b7e-4fe0-a2e1-a52b345c12e7" containerName="maas-api"
Oct 31 01:44:04.973032 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:04.973042    4964 memory_manager.go:365] "RemoveStaleState removing state" podUID="0cc66d3e-3edd-4399-a970-5ab7d0f6f415" containerName="payload-processing"
Oct 31 01:44:04.973913 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:04.973047    4964 memory_manager.go:365] "RemoveStaleState removing state" podUID="2bf44719-ff6b-4ada-9fe7-a6f466992512" containerName="istio-proxy"
Oct 31 01:44:04.973913 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:04.973053    4964 memory_manager.go:365] "RemoveStaleState removing state" podUID="70860802-a6eb-4d36-a11d-77f9c006a93a" containerName="maas-api"
Oct 31 01:44:04.977997 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:04.977977    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-8a3796-openshift-default-857c657d9-8vlxd"
Oct 31 01:44:05.065522 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:05.034400    4964 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-a-8a3796-openshift-default-dockercfg-nxsb9\""
Oct 31 01:44:05.081946 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:05.081912    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9td2s\" (UniqueName: \"kubernetes.io/projected/11d36add-45f3-4bf4-9372-7d7cec791eff-kube-api-access-9td2s\") pod \"e2e-praxis-a-8a3796-openshift-default-857c657d9-8vlxd\" (UID: \"11d36add-45f3-4bf4-9372-7d7cec791eff\") " pod="openshift-ingress/e2e-praxis-a-8a3796-openshift-default-857c657d9-8vlxd"
Oct 31 01:44:05.082144 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:05.081995    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/11d36add-45f3-4bf4-9372-7d7cec791eff-credential-socket\") pod \"e2e-praxis-a-8a3796-openshift-default-857c657d9-8vlxd\" (UID: \"11d36add-45f3-4bf4-9372-7d7cec791eff\") " pod="openshift-ingress/e2e-praxis-a-8a3796-openshift-default-857c657d9-8vlxd"
Oct 31 01:44:05.082144 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:05.082024    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/11d36add-45f3-4bf4-9372-7d7cec791eff-istio-podinfo\") pod \"e2e-praxis-a-8a3796-openshift-default-857c657d9-8vlxd\" (UID: \"11d36add-45f3-4bf4-9372-7d7cec791eff\") " pod="openshift-ingress/e2e-praxis-a-8a3796-openshift-default-857c657d9-8vlxd"
Oct 31 01:44:05.082144 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:05.082107    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/11d36add-45f3-4bf4-9372-7d7cec791eff-workload-certs\") pod \"e2e-praxis-a-8a3796-openshift-default-857c657d9-8vlxd\" (UID: \"11d36add-45f3-4bf4-9372-7d7cec791eff\") " pod="openshift-ingress/e2e-praxis-a-8a3796-openshift-default-857c657d9-8vlxd"
Oct 31 01:44:05.082360 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:05.082156    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/11d36add-45f3-4bf4-9372-7d7cec791eff-istio-data\") pod \"e2e-praxis-a-8a3796-openshift-default-857c657d9-8vlxd\" (UID: \"11d36add-45f3-4bf4-9372-7d7cec791eff\") " pod="openshift-ingress/e2e-praxis-a-8a3796-openshift-default-857c657d9-8vlxd"
Oct 31 01:44:05.082360 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:05.082195    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/11d36add-45f3-4bf4-9372-7d7cec791eff-istio-envoy\") pod \"e2e-praxis-a-8a3796-openshift-default-857c657d9-8vlxd\" (UID: \"11d36add-45f3-4bf4-9372-7d7cec791eff\") " pod="openshift-ingress/e2e-praxis-a-8a3796-openshift-default-857c657d9-8vlxd"
Oct 31 01:44:05.082360 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:05.082282    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/11d36add-45f3-4bf4-9372-7d7cec791eff-workload-socket\") pod \"e2e-praxis-a-8a3796-openshift-default-857c657d9-8vlxd\" (UID: \"11d36add-45f3-4bf4-9372-7d7cec791eff\") " pod="openshift-ingress/e2e-praxis-a-8a3796-openshift-default-857c657d9-8vlxd"
Oct 31 01:44:05.082537 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:05.082366    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/11d36add-45f3-4bf4-9372-7d7cec791eff-istiod-ca-cert\") pod \"e2e-praxis-a-8a3796-openshift-default-857c657d9-8vlxd\" (UID: \"11d36add-45f3-4bf4-9372-7d7cec791eff\") " pod="openshift-ingress/e2e-praxis-a-8a3796-openshift-default-857c657d9-8vlxd"
Oct 31 01:44:05.082537 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:05.082399    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/11d36add-45f3-4bf4-9372-7d7cec791eff-istio-token\") pod \"e2e-praxis-a-8a3796-openshift-default-857c657d9-8vlxd\" (UID: \"11d36add-45f3-4bf4-9372-7d7cec791eff\") " pod="openshift-ingress/e2e-praxis-a-8a3796-openshift-default-857c657d9-8vlxd"
Oct 31 01:44:05.102055 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:05.099843    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-8a3796-openshift-default-857c657d9-8vlxd"]
Oct 31 01:44:05.130065 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:05.130026    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:44:05.184214 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:05.184169    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/11d36add-45f3-4bf4-9372-7d7cec791eff-workload-certs\") pod \"e2e-praxis-a-8a3796-openshift-default-857c657d9-8vlxd\" (UID: \"11d36add-45f3-4bf4-9372-7d7cec791eff\") " pod="openshift-ingress/e2e-praxis-a-8a3796-openshift-default-857c657d9-8vlxd"
Oct 31 01:44:05.184471 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:05.184250    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/11d36add-45f3-4bf4-9372-7d7cec791eff-istio-data\") pod \"e2e-praxis-a-8a3796-openshift-default-857c657d9-8vlxd\" (UID: \"11d36add-45f3-4bf4-9372-7d7cec791eff\") " pod="openshift-ingress/e2e-praxis-a-8a3796-openshift-default-857c657d9-8vlxd"
Oct 31 01:44:05.184471 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:05.184292    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/11d36add-45f3-4bf4-9372-7d7cec791eff-istio-envoy\") pod \"e2e-praxis-a-8a3796-openshift-default-857c657d9-8vlxd\" (UID: \"11d36add-45f3-4bf4-9372-7d7cec791eff\") " pod="openshift-ingress/e2e-praxis-a-8a3796-openshift-default-857c657d9-8vlxd"
Oct 31 01:44:05.184724 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:05.184703    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/11d36add-45f3-4bf4-9372-7d7cec791eff-workload-socket\") pod \"e2e-praxis-a-8a3796-openshift-default-857c657d9-8vlxd\" (UID: \"11d36add-45f3-4bf4-9372-7d7cec791eff\") " pod="openshift-ingress/e2e-praxis-a-8a3796-openshift-default-857c657d9-8vlxd"
Oct 31 01:44:05.184798 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:05.184732    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/11d36add-45f3-4bf4-9372-7d7cec791eff-workload-certs\") pod \"e2e-praxis-a-8a3796-openshift-default-857c657d9-8vlxd\" (UID: \"11d36add-45f3-4bf4-9372-7d7cec791eff\") " pod="openshift-ingress/e2e-praxis-a-8a3796-openshift-default-857c657d9-8vlxd"
Oct 31 01:44:05.184866 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:05.184801    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/11d36add-45f3-4bf4-9372-7d7cec791eff-istiod-ca-cert\") pod \"e2e-praxis-a-8a3796-openshift-default-857c657d9-8vlxd\" (UID: \"11d36add-45f3-4bf4-9372-7d7cec791eff\") " pod="openshift-ingress/e2e-praxis-a-8a3796-openshift-default-857c657d9-8vlxd"
Oct 31 01:44:05.184866 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:05.184835    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/11d36add-45f3-4bf4-9372-7d7cec791eff-istio-token\") pod \"e2e-praxis-a-8a3796-openshift-default-857c657d9-8vlxd\" (UID: \"11d36add-45f3-4bf4-9372-7d7cec791eff\") " pod="openshift-ingress/e2e-praxis-a-8a3796-openshift-default-857c657d9-8vlxd"
Oct 31 01:44:05.185035 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:05.184873    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9td2s\" (UniqueName: \"kubernetes.io/projected/11d36add-45f3-4bf4-9372-7d7cec791eff-kube-api-access-9td2s\") pod \"e2e-praxis-a-8a3796-openshift-default-857c657d9-8vlxd\" (UID: \"11d36add-45f3-4bf4-9372-7d7cec791eff\") " pod="openshift-ingress/e2e-praxis-a-8a3796-openshift-default-857c657d9-8vlxd"
Oct 31 01:44:05.185035 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:05.184954    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/11d36add-45f3-4bf4-9372-7d7cec791eff-credential-socket\") pod \"e2e-praxis-a-8a3796-openshift-default-857c657d9-8vlxd\" (UID: \"11d36add-45f3-4bf4-9372-7d7cec791eff\") " pod="openshift-ingress/e2e-praxis-a-8a3796-openshift-default-857c657d9-8vlxd"
Oct 31 01:44:05.185035 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:05.184979    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/11d36add-45f3-4bf4-9372-7d7cec791eff-istio-podinfo\") pod \"e2e-praxis-a-8a3796-openshift-default-857c657d9-8vlxd\" (UID: \"11d36add-45f3-4bf4-9372-7d7cec791eff\") " pod="openshift-ingress/e2e-praxis-a-8a3796-openshift-default-857c657d9-8vlxd"
Oct 31 01:44:05.185224 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:05.185032    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/11d36add-45f3-4bf4-9372-7d7cec791eff-workload-socket\") pod \"e2e-praxis-a-8a3796-openshift-default-857c657d9-8vlxd\" (UID: \"11d36add-45f3-4bf4-9372-7d7cec791eff\") " pod="openshift-ingress/e2e-praxis-a-8a3796-openshift-default-857c657d9-8vlxd"
Oct 31 01:44:05.185387 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:05.185291    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/11d36add-45f3-4bf4-9372-7d7cec791eff-istio-data\") pod \"e2e-praxis-a-8a3796-openshift-default-857c657d9-8vlxd\" (UID: \"11d36add-45f3-4bf4-9372-7d7cec791eff\") " pod="openshift-ingress/e2e-praxis-a-8a3796-openshift-default-857c657d9-8vlxd"
Oct 31 01:44:05.186670 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:05.186190    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/11d36add-45f3-4bf4-9372-7d7cec791eff-istiod-ca-cert\") pod \"e2e-praxis-a-8a3796-openshift-default-857c657d9-8vlxd\" (UID: \"11d36add-45f3-4bf4-9372-7d7cec791eff\") " pod="openshift-ingress/e2e-praxis-a-8a3796-openshift-default-857c657d9-8vlxd"
Oct 31 01:44:05.186670 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:05.186301    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/11d36add-45f3-4bf4-9372-7d7cec791eff-credential-socket\") pod \"e2e-praxis-a-8a3796-openshift-default-857c657d9-8vlxd\" (UID: \"11d36add-45f3-4bf4-9372-7d7cec791eff\") " pod="openshift-ingress/e2e-praxis-a-8a3796-openshift-default-857c657d9-8vlxd"
Oct 31 01:44:05.188967 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:05.188940    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/11d36add-45f3-4bf4-9372-7d7cec791eff-istio-envoy\") pod \"e2e-praxis-a-8a3796-openshift-default-857c657d9-8vlxd\" (UID: \"11d36add-45f3-4bf4-9372-7d7cec791eff\") " pod="openshift-ingress/e2e-praxis-a-8a3796-openshift-default-857c657d9-8vlxd"
Oct 31 01:44:05.191587 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:05.191557    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/11d36add-45f3-4bf4-9372-7d7cec791eff-istio-podinfo\") pod \"e2e-praxis-a-8a3796-openshift-default-857c657d9-8vlxd\" (UID: \"11d36add-45f3-4bf4-9372-7d7cec791eff\") " pod="openshift-ingress/e2e-praxis-a-8a3796-openshift-default-857c657d9-8vlxd"
Oct 31 01:44:05.242157 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:05.242083    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/11d36add-45f3-4bf4-9372-7d7cec791eff-istio-token\") pod \"e2e-praxis-a-8a3796-openshift-default-857c657d9-8vlxd\" (UID: \"11d36add-45f3-4bf4-9372-7d7cec791eff\") " pod="openshift-ingress/e2e-praxis-a-8a3796-openshift-default-857c657d9-8vlxd"
Oct 31 01:44:05.242306 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:05.242275    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9td2s\" (UniqueName: \"kubernetes.io/projected/11d36add-45f3-4bf4-9372-7d7cec791eff-kube-api-access-9td2s\") pod \"e2e-praxis-a-8a3796-openshift-default-857c657d9-8vlxd\" (UID: \"11d36add-45f3-4bf4-9372-7d7cec791eff\") " pod="openshift-ingress/e2e-praxis-a-8a3796-openshift-default-857c657d9-8vlxd"
Oct 31 01:44:05.292867 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:05.292829    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-8a3796-openshift-default-857c657d9-8vlxd"
Oct 31 01:44:05.344925 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:05.344888    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:44:05.556472 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:05.556387    4964 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-8af859-6b6b774599-r4ql2"]
Oct 31 01:44:05.556654 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:05.556603    4964 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8af859-6b6b774599-r4ql2" podUID="cf5d4bc6-4111-4c96-8ab9-27a461e3e000" containerName="maas-api" containerID="cri-o://3a9b3304ac4a85060fbfae35964d3977f0e18b23291e66dfc5117dce2979cd41" gracePeriod=30
Oct 31 01:44:05.695080 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:05.695032    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:44:05.746235 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:05.745755    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-8a3796-openshift-default-857c657d9-8vlxd"]
Oct 31 01:44:05.766565 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:05.766523    4964 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 01:44:05.766678 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:05.766599    4964 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 01:44:05.766678 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:05.766628    4964 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 01:44:05.801889 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:05.801857    4964 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0cc66d3e-3edd-4399-a970-5ab7d0f6f415" path="/var/lib/kubelet/pods/0cc66d3e-3edd-4399-a970-5ab7d0f6f415/volumes"
Oct 31 01:44:05.843381 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:05.843357    4964 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8af859-6b6b774599-r4ql2"
Oct 31 01:44:05.894365 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:05.894295    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cf5d4bc6-4111-4c96-8ab9-27a461e3e000-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cf5d4bc6-4111-4c96-8ab9-27a461e3e000-maas-api-tls\") pod \"cf5d4bc6-4111-4c96-8ab9-27a461e3e000\" (UID: \"cf5d4bc6-4111-4c96-8ab9-27a461e3e000\") "
Oct 31 01:44:05.894365 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:05.894360    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cf5d4bc6-4111-4c96-8ab9-27a461e3e000-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cf5d4bc6-4111-4c96-8ab9-27a461e3e000-openshift-service-ca\") pod \"cf5d4bc6-4111-4c96-8ab9-27a461e3e000\" (UID: \"cf5d4bc6-4111-4c96-8ab9-27a461e3e000\") "
Oct 31 01:44:05.894626 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:05.894432    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cf5d4bc6-4111-4c96-8ab9-27a461e3e000-kube-api-access-7wlrn\" (UniqueName: \"kubernetes.io/projected/cf5d4bc6-4111-4c96-8ab9-27a461e3e000-kube-api-access-7wlrn\") pod \"cf5d4bc6-4111-4c96-8ab9-27a461e3e000\" (UID: \"cf5d4bc6-4111-4c96-8ab9-27a461e3e000\") "
Oct 31 01:44:05.894626 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:05.894501    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cf5d4bc6-4111-4c96-8ab9-27a461e3e000-metrics-certs\" (UniqueName: \"kubernetes.io/secret/cf5d4bc6-4111-4c96-8ab9-27a461e3e000-metrics-certs\") pod \"cf5d4bc6-4111-4c96-8ab9-27a461e3e000\" (UID: \"cf5d4bc6-4111-4c96-8ab9-27a461e3e000\") "
Oct 31 01:44:05.894894 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:05.894787    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cf5d4bc6-4111-4c96-8ab9-27a461e3e000-openshift-service-ca" pod "cf5d4bc6-4111-4c96-8ab9-27a461e3e000" (UID: "cf5d4bc6-4111-4c96-8ab9-27a461e3e000"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:44:05.896489 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:05.896461    4964 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cf5d4bc6-4111-4c96-8ab9-27a461e3e000-openshift-service-ca\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:44:05.897264 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:05.897236    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cf5d4bc6-4111-4c96-8ab9-27a461e3e000-metrics-certs" pod "cf5d4bc6-4111-4c96-8ab9-27a461e3e000" (UID: "cf5d4bc6-4111-4c96-8ab9-27a461e3e000"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:44:05.898909 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:05.898877    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf5d4bc6-4111-4c96-8ab9-27a461e3e000-kube-api-access-7wlrn" pod "cf5d4bc6-4111-4c96-8ab9-27a461e3e000" (UID: "cf5d4bc6-4111-4c96-8ab9-27a461e3e000"). InnerVolumeSpecName "kube-api-access-7wlrn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:44:05.899361 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:05.899304    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cf5d4bc6-4111-4c96-8ab9-27a461e3e000-maas-api-tls" pod "cf5d4bc6-4111-4c96-8ab9-27a461e3e000" (UID: "cf5d4bc6-4111-4c96-8ab9-27a461e3e000"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:44:05.997817 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:05.997742    4964 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7wlrn\" (UniqueName: \"kubernetes.io/projected/cf5d4bc6-4111-4c96-8ab9-27a461e3e000-kube-api-access-7wlrn\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:44:05.997817 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:05.997773    4964 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cf5d4bc6-4111-4c96-8ab9-27a461e3e000-metrics-certs\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:44:05.997817 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:05.997783    4964 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cf5d4bc6-4111-4c96-8ab9-27a461e3e000-maas-api-tls\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:44:06.264707 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:06.264672    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:44:06.439624 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:06.439577    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:44:06.439624 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:06.439608    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8af859-6b6b774599-r4ql2" event={"ID":"cf5d4bc6-4111-4c96-8ab9-27a461e3e000","Type":"ContainerDied","Data":"3a9b3304ac4a85060fbfae35964d3977f0e18b23291e66dfc5117dce2979cd41"}
Oct 31 01:44:06.439898 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:06.439634    4964 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8af859-6b6b774599-r4ql2"
Oct 31 01:44:06.439898 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:06.439580    4964 generic.go:356] "Generic (PLEG): container finished" podID="cf5d4bc6-4111-4c96-8ab9-27a461e3e000" containerID="3a9b3304ac4a85060fbfae35964d3977f0e18b23291e66dfc5117dce2979cd41" exitCode=0
Oct 31 01:44:06.439898 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:06.439665    4964 scope.go:122] "RemoveContainer" containerID="3a9b3304ac4a85060fbfae35964d3977f0e18b23291e66dfc5117dce2979cd41"
Oct 31 01:44:06.439898 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:06.439770    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8af859-6b6b774599-r4ql2" event={"ID":"cf5d4bc6-4111-4c96-8ab9-27a461e3e000","Type":"ContainerDied","Data":"3d87ef629a92740b10008ddd5cb305b5ab841ad13c863a8bfddf3ca53fe6d011"}
Oct 31 01:44:06.441749 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:06.441713    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-8a3796-openshift-default-857c657d9-8vlxd" event={"ID":"11d36add-45f3-4bf4-9372-7d7cec791eff","Type":"ContainerStarted","Data":"c91c1fe8e115676b727f5825d66641c22ba6c0d04d6c035d7d8bcbafb620504e"}
Oct 31 01:44:06.441871 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:06.441780    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-8a3796-openshift-default-857c657d9-8vlxd" event={"ID":"11d36add-45f3-4bf4-9372-7d7cec791eff","Type":"ContainerStarted","Data":"1a1945691ffbeea5e4716dbba07dd72da41b79ee8b88832f8ddc352764de53df"}
Oct 31 01:44:06.450881 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:06.450860    4964 scope.go:122] "RemoveContainer" containerID="3a9b3304ac4a85060fbfae35964d3977f0e18b23291e66dfc5117dce2979cd41"
Oct 31 01:44:06.451201 ip-10-0-133-197 kubenswrapper[4964]: E1031 01:44:06.451180    4964 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3a9b3304ac4a85060fbfae35964d3977f0e18b23291e66dfc5117dce2979cd41\": container with ID starting with 3a9b3304ac4a85060fbfae35964d3977f0e18b23291e66dfc5117dce2979cd41 not found: ID does not exist" containerID="3a9b3304ac4a85060fbfae35964d3977f0e18b23291e66dfc5117dce2979cd41"
Oct 31 01:44:06.451254 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:06.451210    4964 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3a9b3304ac4a85060fbfae35964d3977f0e18b23291e66dfc5117dce2979cd41"} err="failed to get container status \"3a9b3304ac4a85060fbfae35964d3977f0e18b23291e66dfc5117dce2979cd41\": rpc error: code = NotFound desc = could not find container \"3a9b3304ac4a85060fbfae35964d3977f0e18b23291e66dfc5117dce2979cd41\": container with ID starting with 3a9b3304ac4a85060fbfae35964d3977f0e18b23291e66dfc5117dce2979cd41 not found: ID does not exist"
Oct 31 01:44:06.615720 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:06.615666    4964 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-8af859-6b6b774599-r4ql2"]
Oct 31 01:44:06.617821 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:06.617781    4964 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-8af859-6b6b774599-r4ql2"]
Oct 31 01:44:06.723603 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:06.723544    4964 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-a-8a3796-openshift-default-857c657d9-8vlxd" podStartSLOduration=2.723531043 podStartE2EDuration="2.723531043s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:44:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:44:06.720251085 +0000 UTC m=+2415.196817465" watchObservedRunningTime="2026-10-31 01:44:06.723531043 +0000 UTC m=+2415.200097422"
Oct 31 01:44:07.264261 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:07.264225    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:44:07.293737 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:07.293708    4964 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-8a3796-openshift-default-857c657d9-8vlxd"
Oct 31 01:44:07.300818 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:07.300790    4964 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-8a3796-openshift-default-857c657d9-8vlxd"
Oct 31 01:44:07.451026 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:07.450997    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-8a3796-openshift-default-857c657d9-8vlxd"
Oct 31 01:44:07.452070 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:07.452047    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-8a3796-openshift-default-857c657d9-8vlxd"
Oct 31 01:44:07.491509 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:07.491475    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:44:07.690418 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:07.690285    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:44:07.777916 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:07.777885    4964 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cf5d4bc6-4111-4c96-8ab9-27a461e3e000" path="/var/lib/kubelet/pods/cf5d4bc6-4111-4c96-8ab9-27a461e3e000/volumes"
Oct 31 01:44:08.177608 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:08.177571    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:44:09.581507 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:09.581473    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:44:09.771710 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:09.771671    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:44:09.955811 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:09.955723    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:44:11.237455 ip-10-0-133-197 kubenswrapper[4964]: E1031 01:44:11.237418    4964 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q56x5" containerName="extractor"
Oct 31 01:44:12.070701 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:12.070671    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:44:12.859916 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:12.859882    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:44:13.033266 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:13.033232    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:44:13.440541 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:13.440509    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:44:14.358406 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:14.358330    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:44:14.568589 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:14.568507    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:44:14.591237 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:14.591206    4964 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-8af859-openshift-default-59576db9cf-6mc8p"]
Oct 31 01:44:14.591542 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:14.591517    4964 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-8af859-openshift-default-59576db9cf-6mc8p" podUID="940b399a-de68-4440-a629-df5ec46dc796" containerName="istio-proxy" containerID="cri-o://f0641e882dcbb25e307f2b537f6e081248b46cedf5b4439852ec3519bddd3642" gracePeriod=30
Oct 31 01:44:14.603239 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:14.603210    4964 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-8af859-openshift-default-59576db9cf-cnbq6"]
Oct 31 01:44:14.603430 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:14.603409    4964 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-8af859-openshift-default-59576db9cf-cnbq6" podUID="503330d0-a5f9-470b-8fd6-3ad2e4cf4698" containerName="istio-proxy" containerID="cri-o://c05faf4dfa15809705f5312ca188b291bfb39dbe3e0d6722c252ad6fd4bef3b6" gracePeriod=30
Oct 31 01:44:14.746842 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:14.746782    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:44:16.133121 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:16.133087    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:44:16.739118 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:16.739082    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:44:17.015802 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:17.015715    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:44:19.491085 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:19.491032    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:44:19.888950 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:19.888914    4964 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-8af859-openshift-default-59576db9cf-6mc8p"
Oct 31 01:44:19.947756 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:19.947729    4964 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-8af859-openshift-default-59576db9cf-cnbq6"
Oct 31 01:44:20.008095 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:20.007923    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/940b399a-de68-4440-a629-df5ec46dc796-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/940b399a-de68-4440-a629-df5ec46dc796-istio-podinfo\") pod \"940b399a-de68-4440-a629-df5ec46dc796\" (UID: \"940b399a-de68-4440-a629-df5ec46dc796\") "
Oct 31 01:44:20.008638 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:20.008263    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/940b399a-de68-4440-a629-df5ec46dc796-kube-api-access-gk2bl\" (UniqueName: \"kubernetes.io/projected/940b399a-de68-4440-a629-df5ec46dc796-kube-api-access-gk2bl\") pod \"940b399a-de68-4440-a629-df5ec46dc796\" (UID: \"940b399a-de68-4440-a629-df5ec46dc796\") "
Oct 31 01:44:20.008791 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:20.008728    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/940b399a-de68-4440-a629-df5ec46dc796-istio-token\" (UniqueName: \"kubernetes.io/projected/940b399a-de68-4440-a629-df5ec46dc796-istio-token\") pod \"940b399a-de68-4440-a629-df5ec46dc796\" (UID: \"940b399a-de68-4440-a629-df5ec46dc796\") "
Oct 31 01:44:20.008865 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:20.008792    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/940b399a-de68-4440-a629-df5ec46dc796-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/940b399a-de68-4440-a629-df5ec46dc796-istiod-ca-cert\") pod \"940b399a-de68-4440-a629-df5ec46dc796\" (UID: \"940b399a-de68-4440-a629-df5ec46dc796\") "
Oct 31 01:44:20.008865 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:20.008832    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/940b399a-de68-4440-a629-df5ec46dc796-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/940b399a-de68-4440-a629-df5ec46dc796-workload-socket\") pod \"940b399a-de68-4440-a629-df5ec46dc796\" (UID: \"940b399a-de68-4440-a629-df5ec46dc796\") "
Oct 31 01:44:20.008971 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:20.008860    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/940b399a-de68-4440-a629-df5ec46dc796-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/940b399a-de68-4440-a629-df5ec46dc796-workload-certs\") pod \"940b399a-de68-4440-a629-df5ec46dc796\" (UID: \"940b399a-de68-4440-a629-df5ec46dc796\") "
Oct 31 01:44:20.009609 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:20.009264    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/940b399a-de68-4440-a629-df5ec46dc796-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/940b399a-de68-4440-a629-df5ec46dc796-istio-data\") pod \"940b399a-de68-4440-a629-df5ec46dc796\" (UID: \"940b399a-de68-4440-a629-df5ec46dc796\") "
Oct 31 01:44:20.009609 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:20.009116    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/940b399a-de68-4440-a629-df5ec46dc796-workload-socket" pod "940b399a-de68-4440-a629-df5ec46dc796" (UID: "940b399a-de68-4440-a629-df5ec46dc796"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:44:20.009609 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:20.009363    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/940b399a-de68-4440-a629-df5ec46dc796-istiod-ca-cert" pod "940b399a-de68-4440-a629-df5ec46dc796" (UID: "940b399a-de68-4440-a629-df5ec46dc796"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:44:20.009609 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:20.009476    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/940b399a-de68-4440-a629-df5ec46dc796-istio-data" pod "940b399a-de68-4440-a629-df5ec46dc796" (UID: "940b399a-de68-4440-a629-df5ec46dc796"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:44:20.009609 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:20.009560    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/940b399a-de68-4440-a629-df5ec46dc796-workload-certs" pod "940b399a-de68-4440-a629-df5ec46dc796" (UID: "940b399a-de68-4440-a629-df5ec46dc796"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:44:20.011795 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:20.010757    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/940b399a-de68-4440-a629-df5ec46dc796-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/940b399a-de68-4440-a629-df5ec46dc796-istio-envoy\") pod \"940b399a-de68-4440-a629-df5ec46dc796\" (UID: \"940b399a-de68-4440-a629-df5ec46dc796\") "
Oct 31 01:44:20.011795 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:20.011055    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/940b399a-de68-4440-a629-df5ec46dc796-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/940b399a-de68-4440-a629-df5ec46dc796-credential-socket\") pod \"940b399a-de68-4440-a629-df5ec46dc796\" (UID: \"940b399a-de68-4440-a629-df5ec46dc796\") "
Oct 31 01:44:20.011795 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:20.011393    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/940b399a-de68-4440-a629-df5ec46dc796-credential-socket" pod "940b399a-de68-4440-a629-df5ec46dc796" (UID: "940b399a-de68-4440-a629-df5ec46dc796"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:44:20.011795 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:20.011712    4964 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/940b399a-de68-4440-a629-df5ec46dc796-credential-socket\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:44:20.011795 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:20.011734    4964 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/940b399a-de68-4440-a629-df5ec46dc796-istiod-ca-cert\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:44:20.011795 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:20.011739    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/940b399a-de68-4440-a629-df5ec46dc796-istio-podinfo" pod "940b399a-de68-4440-a629-df5ec46dc796" (UID: "940b399a-de68-4440-a629-df5ec46dc796"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 01:44:20.011795 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:20.011750    4964 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/940b399a-de68-4440-a629-df5ec46dc796-workload-socket\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:44:20.011795 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:20.011767    4964 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/940b399a-de68-4440-a629-df5ec46dc796-workload-certs\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:44:20.011795 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:20.011782    4964 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/940b399a-de68-4440-a629-df5ec46dc796-istio-data\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:44:20.011795 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:20.011762    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/940b399a-de68-4440-a629-df5ec46dc796-kube-api-access-gk2bl" pod "940b399a-de68-4440-a629-df5ec46dc796" (UID: "940b399a-de68-4440-a629-df5ec46dc796"). InnerVolumeSpecName "kube-api-access-gk2bl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:44:20.011795 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:20.011781    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/940b399a-de68-4440-a629-df5ec46dc796-istio-token" pod "940b399a-de68-4440-a629-df5ec46dc796" (UID: "940b399a-de68-4440-a629-df5ec46dc796"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:44:20.014432 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:20.014408    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/940b399a-de68-4440-a629-df5ec46dc796-istio-envoy" pod "940b399a-de68-4440-a629-df5ec46dc796" (UID: "940b399a-de68-4440-a629-df5ec46dc796"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:44:20.113143 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:20.113059    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/503330d0-a5f9-470b-8fd6-3ad2e4cf4698-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/503330d0-a5f9-470b-8fd6-3ad2e4cf4698-workload-socket\") pod \"503330d0-a5f9-470b-8fd6-3ad2e4cf4698\" (UID: \"503330d0-a5f9-470b-8fd6-3ad2e4cf4698\") "
Oct 31 01:44:20.113143 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:20.113119    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/503330d0-a5f9-470b-8fd6-3ad2e4cf4698-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/503330d0-a5f9-470b-8fd6-3ad2e4cf4698-istio-envoy\") pod \"503330d0-a5f9-470b-8fd6-3ad2e4cf4698\" (UID: \"503330d0-a5f9-470b-8fd6-3ad2e4cf4698\") "
Oct 31 01:44:20.113422 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:20.113158    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/503330d0-a5f9-470b-8fd6-3ad2e4cf4698-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/503330d0-a5f9-470b-8fd6-3ad2e4cf4698-istiod-ca-cert\") pod \"503330d0-a5f9-470b-8fd6-3ad2e4cf4698\" (UID: \"503330d0-a5f9-470b-8fd6-3ad2e4cf4698\") "
Oct 31 01:44:20.113422 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:20.113404    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/503330d0-a5f9-470b-8fd6-3ad2e4cf4698-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/503330d0-a5f9-470b-8fd6-3ad2e4cf4698-istio-podinfo\") pod \"503330d0-a5f9-470b-8fd6-3ad2e4cf4698\" (UID: \"503330d0-a5f9-470b-8fd6-3ad2e4cf4698\") "
Oct 31 01:44:20.113547 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:20.113437    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/503330d0-a5f9-470b-8fd6-3ad2e4cf4698-workload-socket" pod "503330d0-a5f9-470b-8fd6-3ad2e4cf4698" (UID: "503330d0-a5f9-470b-8fd6-3ad2e4cf4698"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:44:20.113547 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:20.113450    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/503330d0-a5f9-470b-8fd6-3ad2e4cf4698-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/503330d0-a5f9-470b-8fd6-3ad2e4cf4698-istio-data\") pod \"503330d0-a5f9-470b-8fd6-3ad2e4cf4698\" (UID: \"503330d0-a5f9-470b-8fd6-3ad2e4cf4698\") "
Oct 31 01:44:20.113656 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:20.113606    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/503330d0-a5f9-470b-8fd6-3ad2e4cf4698-kube-api-access-jwl4b\" (UniqueName: \"kubernetes.io/projected/503330d0-a5f9-470b-8fd6-3ad2e4cf4698-kube-api-access-jwl4b\") pod \"503330d0-a5f9-470b-8fd6-3ad2e4cf4698\" (UID: \"503330d0-a5f9-470b-8fd6-3ad2e4cf4698\") "
Oct 31 01:44:20.113713 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:20.113653    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/503330d0-a5f9-470b-8fd6-3ad2e4cf4698-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/503330d0-a5f9-470b-8fd6-3ad2e4cf4698-workload-certs\") pod \"503330d0-a5f9-470b-8fd6-3ad2e4cf4698\" (UID: \"503330d0-a5f9-470b-8fd6-3ad2e4cf4698\") "
Oct 31 01:44:20.113713 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:20.113664    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/503330d0-a5f9-470b-8fd6-3ad2e4cf4698-istio-data" pod "503330d0-a5f9-470b-8fd6-3ad2e4cf4698" (UID: "503330d0-a5f9-470b-8fd6-3ad2e4cf4698"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:44:20.113713 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:20.113686    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/503330d0-a5f9-470b-8fd6-3ad2e4cf4698-istiod-ca-cert" pod "503330d0-a5f9-470b-8fd6-3ad2e4cf4698" (UID: "503330d0-a5f9-470b-8fd6-3ad2e4cf4698"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:44:20.113713 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:20.113694    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/503330d0-a5f9-470b-8fd6-3ad2e4cf4698-istio-token\" (UniqueName: \"kubernetes.io/projected/503330d0-a5f9-470b-8fd6-3ad2e4cf4698-istio-token\") pod \"503330d0-a5f9-470b-8fd6-3ad2e4cf4698\" (UID: \"503330d0-a5f9-470b-8fd6-3ad2e4cf4698\") "
Oct 31 01:44:20.113933 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:20.113768    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/503330d0-a5f9-470b-8fd6-3ad2e4cf4698-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/503330d0-a5f9-470b-8fd6-3ad2e4cf4698-credential-socket\") pod \"503330d0-a5f9-470b-8fd6-3ad2e4cf4698\" (UID: \"503330d0-a5f9-470b-8fd6-3ad2e4cf4698\") "
Oct 31 01:44:20.114391 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:20.114362    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/503330d0-a5f9-470b-8fd6-3ad2e4cf4698-credential-socket" pod "503330d0-a5f9-470b-8fd6-3ad2e4cf4698" (UID: "503330d0-a5f9-470b-8fd6-3ad2e4cf4698"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:44:20.114655 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:20.114427    4964 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/940b399a-de68-4440-a629-df5ec46dc796-istio-envoy\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:44:20.114655 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:20.114447    4964 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/503330d0-a5f9-470b-8fd6-3ad2e4cf4698-workload-socket\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:44:20.114655 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:20.114463    4964 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/503330d0-a5f9-470b-8fd6-3ad2e4cf4698-istiod-ca-cert\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:44:20.114655 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:20.114478    4964 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/940b399a-de68-4440-a629-df5ec46dc796-istio-podinfo\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:44:20.114655 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:20.114494    4964 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gk2bl\" (UniqueName: \"kubernetes.io/projected/940b399a-de68-4440-a629-df5ec46dc796-kube-api-access-gk2bl\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:44:20.114655 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:20.114509    4964 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/503330d0-a5f9-470b-8fd6-3ad2e4cf4698-istio-data\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:44:20.114655 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:20.114523    4964 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/940b399a-de68-4440-a629-df5ec46dc796-istio-token\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:44:20.114655 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:20.114569    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/503330d0-a5f9-470b-8fd6-3ad2e4cf4698-workload-certs" pod "503330d0-a5f9-470b-8fd6-3ad2e4cf4698" (UID: "503330d0-a5f9-470b-8fd6-3ad2e4cf4698"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:44:20.116488 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:20.116458    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/503330d0-a5f9-470b-8fd6-3ad2e4cf4698-istio-podinfo" pod "503330d0-a5f9-470b-8fd6-3ad2e4cf4698" (UID: "503330d0-a5f9-470b-8fd6-3ad2e4cf4698"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 01:44:20.116598 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:20.116489    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/503330d0-a5f9-470b-8fd6-3ad2e4cf4698-kube-api-access-jwl4b" pod "503330d0-a5f9-470b-8fd6-3ad2e4cf4698" (UID: "503330d0-a5f9-470b-8fd6-3ad2e4cf4698"). InnerVolumeSpecName "kube-api-access-jwl4b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:44:20.116598 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:20.116565    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/503330d0-a5f9-470b-8fd6-3ad2e4cf4698-istio-token" pod "503330d0-a5f9-470b-8fd6-3ad2e4cf4698" (UID: "503330d0-a5f9-470b-8fd6-3ad2e4cf4698"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:44:20.116790 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:20.116777    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/503330d0-a5f9-470b-8fd6-3ad2e4cf4698-istio-envoy" pod "503330d0-a5f9-470b-8fd6-3ad2e4cf4698" (UID: "503330d0-a5f9-470b-8fd6-3ad2e4cf4698"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:44:20.215636 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:20.215600    4964 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/503330d0-a5f9-470b-8fd6-3ad2e4cf4698-istio-envoy\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:44:20.215636 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:20.215631    4964 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/503330d0-a5f9-470b-8fd6-3ad2e4cf4698-istio-podinfo\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:44:20.215844 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:20.215644    4964 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jwl4b\" (UniqueName: \"kubernetes.io/projected/503330d0-a5f9-470b-8fd6-3ad2e4cf4698-kube-api-access-jwl4b\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:44:20.215844 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:20.215660    4964 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/503330d0-a5f9-470b-8fd6-3ad2e4cf4698-workload-certs\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:44:20.215844 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:20.215672    4964 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/503330d0-a5f9-470b-8fd6-3ad2e4cf4698-istio-token\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:44:20.215844 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:20.215684    4964 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/503330d0-a5f9-470b-8fd6-3ad2e4cf4698-credential-socket\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:44:20.529409 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:20.529360    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:44:20.542175 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:20.542141    4964 generic.go:356] "Generic (PLEG): container finished" podID="503330d0-a5f9-470b-8fd6-3ad2e4cf4698" containerID="c05faf4dfa15809705f5312ca188b291bfb39dbe3e0d6722c252ad6fd4bef3b6" exitCode=0
Oct 31 01:44:20.542371 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:20.542194    4964 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-8af859-openshift-default-59576db9cf-cnbq6"
Oct 31 01:44:20.542371 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:20.542230    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-8af859-openshift-default-59576db9cf-cnbq6" event={"ID":"503330d0-a5f9-470b-8fd6-3ad2e4cf4698","Type":"ContainerDied","Data":"c05faf4dfa15809705f5312ca188b291bfb39dbe3e0d6722c252ad6fd4bef3b6"}
Oct 31 01:44:20.542371 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:20.542282    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-8af859-openshift-default-59576db9cf-cnbq6" event={"ID":"503330d0-a5f9-470b-8fd6-3ad2e4cf4698","Type":"ContainerDied","Data":"dce1a7a834697b4311c1447c80e88dd00fe5131ae86329c3ae85e6df1db415e4"}
Oct 31 01:44:20.542371 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:20.542309    4964 scope.go:122] "RemoveContainer" containerID="c05faf4dfa15809705f5312ca188b291bfb39dbe3e0d6722c252ad6fd4bef3b6"
Oct 31 01:44:20.543778 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:20.543758    4964 generic.go:356] "Generic (PLEG): container finished" podID="940b399a-de68-4440-a629-df5ec46dc796" containerID="f0641e882dcbb25e307f2b537f6e081248b46cedf5b4439852ec3519bddd3642" exitCode=0
Oct 31 01:44:20.544048 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:20.543811    4964 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-8af859-openshift-default-59576db9cf-6mc8p"
Oct 31 01:44:20.544048 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:20.543843    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-8af859-openshift-default-59576db9cf-6mc8p" event={"ID":"940b399a-de68-4440-a629-df5ec46dc796","Type":"ContainerDied","Data":"f0641e882dcbb25e307f2b537f6e081248b46cedf5b4439852ec3519bddd3642"}
Oct 31 01:44:20.544048 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:20.543879    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-8af859-openshift-default-59576db9cf-6mc8p" event={"ID":"940b399a-de68-4440-a629-df5ec46dc796","Type":"ContainerDied","Data":"a967229cff894f28998520d8979bc953d3ad15fc4ef579bbb0fa77e6c205b545"}
Oct 31 01:44:20.553266 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:20.553245    4964 scope.go:122] "RemoveContainer" containerID="c05faf4dfa15809705f5312ca188b291bfb39dbe3e0d6722c252ad6fd4bef3b6"
Oct 31 01:44:20.553587 ip-10-0-133-197 kubenswrapper[4964]: E1031 01:44:20.553564    4964 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c05faf4dfa15809705f5312ca188b291bfb39dbe3e0d6722c252ad6fd4bef3b6\": container with ID starting with c05faf4dfa15809705f5312ca188b291bfb39dbe3e0d6722c252ad6fd4bef3b6 not found: ID does not exist" containerID="c05faf4dfa15809705f5312ca188b291bfb39dbe3e0d6722c252ad6fd4bef3b6"
Oct 31 01:44:20.553648 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:20.553593    4964 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c05faf4dfa15809705f5312ca188b291bfb39dbe3e0d6722c252ad6fd4bef3b6"} err="failed to get container status \"c05faf4dfa15809705f5312ca188b291bfb39dbe3e0d6722c252ad6fd4bef3b6\": rpc error: code = NotFound desc = could not find container \"c05faf4dfa15809705f5312ca188b291bfb39dbe3e0d6722c252ad6fd4bef3b6\": container with ID starting with c05faf4dfa15809705f5312ca188b291bfb39dbe3e0d6722c252ad6fd4bef3b6 not found: ID does not exist"
Oct 31 01:44:20.553648 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:20.553609    4964 scope.go:122] "RemoveContainer" containerID="f0641e882dcbb25e307f2b537f6e081248b46cedf5b4439852ec3519bddd3642"
Oct 31 01:44:20.563294 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:20.563279    4964 scope.go:122] "RemoveContainer" containerID="f0641e882dcbb25e307f2b537f6e081248b46cedf5b4439852ec3519bddd3642"
Oct 31 01:44:20.563614 ip-10-0-133-197 kubenswrapper[4964]: E1031 01:44:20.563595    4964 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f0641e882dcbb25e307f2b537f6e081248b46cedf5b4439852ec3519bddd3642\": container with ID starting with f0641e882dcbb25e307f2b537f6e081248b46cedf5b4439852ec3519bddd3642 not found: ID does not exist" containerID="f0641e882dcbb25e307f2b537f6e081248b46cedf5b4439852ec3519bddd3642"
Oct 31 01:44:20.563664 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:20.563621    4964 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f0641e882dcbb25e307f2b537f6e081248b46cedf5b4439852ec3519bddd3642"} err="failed to get container status \"f0641e882dcbb25e307f2b537f6e081248b46cedf5b4439852ec3519bddd3642\": rpc error: code = NotFound desc = could not find container \"f0641e882dcbb25e307f2b537f6e081248b46cedf5b4439852ec3519bddd3642\": container with ID starting with f0641e882dcbb25e307f2b537f6e081248b46cedf5b4439852ec3519bddd3642 not found: ID does not exist"
Oct 31 01:44:20.710400 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:20.710367    4964 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-8af859-openshift-default-59576db9cf-6mc8p"]
Oct 31 01:44:20.710815 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:20.710788    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:44:20.757043 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:20.756990    4964 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-8af859-openshift-default-59576db9cf-6mc8p"]
Oct 31 01:44:20.820245 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:20.820154    4964 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-8af859-openshift-default-59576db9cf-cnbq6"]
Oct 31 01:44:20.823348 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:20.823310    4964 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-8af859-openshift-default-59576db9cf-cnbq6"]
Oct 31 01:44:20.892250 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:20.892216    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:44:21.777081 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:21.777052    4964 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="503330d0-a5f9-470b-8fd6-3ad2e4cf4698" path="/var/lib/kubelet/pods/503330d0-a5f9-470b-8fd6-3ad2e4cf4698/volumes"
Oct 31 01:44:21.777532 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:21.777516    4964 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="940b399a-de68-4440-a629-df5ec46dc796" path="/var/lib/kubelet/pods/940b399a-de68-4440-a629-df5ec46dc796/volumes"
Oct 31 01:44:23.763105 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:23.763066    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:44:23.941390 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:23.941347    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:44:24.947772 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:24.947739    4964 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-b4hfr"]
Oct 31 01:44:24.948156 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:24.947956    4964 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-b4hfr" podUID="a8754d1f-9656-460d-879e-c8d3f917492e" containerName="istio-proxy" containerID="cri-o://f487c9e5f1066adefa450e1d49631a6c1cd66173a7c3e2ab10c9f9dbd71debf2" gracePeriod=30
Oct 31 01:44:24.984598 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:24.984559    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:44:29.717600 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:29.714777    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:44:30.211632 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:30.211603    4964 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-b4hfr"
Oct 31 01:44:30.236849 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:30.236788    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a8754d1f-9656-460d-879e-c8d3f917492e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8754d1f-9656-460d-879e-c8d3f917492e-workload-socket\") pod \"a8754d1f-9656-460d-879e-c8d3f917492e\" (UID: \"a8754d1f-9656-460d-879e-c8d3f917492e\") "
Oct 31 01:44:30.236849 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:30.236826    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a8754d1f-9656-460d-879e-c8d3f917492e-kube-api-access-6zgqj\" (UniqueName: \"kubernetes.io/projected/a8754d1f-9656-460d-879e-c8d3f917492e-kube-api-access-6zgqj\") pod \"a8754d1f-9656-460d-879e-c8d3f917492e\" (UID: \"a8754d1f-9656-460d-879e-c8d3f917492e\") "
Oct 31 01:44:30.237069 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:30.236898    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a8754d1f-9656-460d-879e-c8d3f917492e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a8754d1f-9656-460d-879e-c8d3f917492e-workload-certs\") pod \"a8754d1f-9656-460d-879e-c8d3f917492e\" (UID: \"a8754d1f-9656-460d-879e-c8d3f917492e\") "
Oct 31 01:44:30.237069 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:30.236932    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a8754d1f-9656-460d-879e-c8d3f917492e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8754d1f-9656-460d-879e-c8d3f917492e-credential-socket\") pod \"a8754d1f-9656-460d-879e-c8d3f917492e\" (UID: \"a8754d1f-9656-460d-879e-c8d3f917492e\") "
Oct 31 01:44:30.237069 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:30.236952    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a8754d1f-9656-460d-879e-c8d3f917492e-istio-token\" (UniqueName: \"kubernetes.io/projected/a8754d1f-9656-460d-879e-c8d3f917492e-istio-token\") pod \"a8754d1f-9656-460d-879e-c8d3f917492e\" (UID: \"a8754d1f-9656-460d-879e-c8d3f917492e\") "
Oct 31 01:44:30.237069 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:30.236997    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a8754d1f-9656-460d-879e-c8d3f917492e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a8754d1f-9656-460d-879e-c8d3f917492e-istio-data\") pod \"a8754d1f-9656-460d-879e-c8d3f917492e\" (UID: \"a8754d1f-9656-460d-879e-c8d3f917492e\") "
Oct 31 01:44:30.237282 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:30.237073    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a8754d1f-9656-460d-879e-c8d3f917492e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a8754d1f-9656-460d-879e-c8d3f917492e-istio-envoy\") pod \"a8754d1f-9656-460d-879e-c8d3f917492e\" (UID: \"a8754d1f-9656-460d-879e-c8d3f917492e\") "
Oct 31 01:44:30.237282 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:30.237105    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a8754d1f-9656-460d-879e-c8d3f917492e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a8754d1f-9656-460d-879e-c8d3f917492e-istio-podinfo\") pod \"a8754d1f-9656-460d-879e-c8d3f917492e\" (UID: \"a8754d1f-9656-460d-879e-c8d3f917492e\") "
Oct 31 01:44:30.237282 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:30.237104    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8754d1f-9656-460d-879e-c8d3f917492e-workload-socket" pod "a8754d1f-9656-460d-879e-c8d3f917492e" (UID: "a8754d1f-9656-460d-879e-c8d3f917492e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:44:30.237282 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:30.237127    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a8754d1f-9656-460d-879e-c8d3f917492e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a8754d1f-9656-460d-879e-c8d3f917492e-istiod-ca-cert\") pod \"a8754d1f-9656-460d-879e-c8d3f917492e\" (UID: \"a8754d1f-9656-460d-879e-c8d3f917492e\") "
Oct 31 01:44:30.237282 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:30.237245    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8754d1f-9656-460d-879e-c8d3f917492e-workload-certs" pod "a8754d1f-9656-460d-879e-c8d3f917492e" (UID: "a8754d1f-9656-460d-879e-c8d3f917492e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:44:30.237282 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:30.237256    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8754d1f-9656-460d-879e-c8d3f917492e-credential-socket" pod "a8754d1f-9656-460d-879e-c8d3f917492e" (UID: "a8754d1f-9656-460d-879e-c8d3f917492e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:44:30.237677 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:30.237584    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8754d1f-9656-460d-879e-c8d3f917492e-istio-data" pod "a8754d1f-9656-460d-879e-c8d3f917492e" (UID: "a8754d1f-9656-460d-879e-c8d3f917492e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:44:30.237677 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:30.237600    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a8754d1f-9656-460d-879e-c8d3f917492e-istiod-ca-cert" pod "a8754d1f-9656-460d-879e-c8d3f917492e" (UID: "a8754d1f-9656-460d-879e-c8d3f917492e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:44:30.237833 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:30.237703    4964 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a8754d1f-9656-460d-879e-c8d3f917492e-workload-certs\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:44:30.237833 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:30.237724    4964 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8754d1f-9656-460d-879e-c8d3f917492e-credential-socket\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:44:30.237833 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:30.237739    4964 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a8754d1f-9656-460d-879e-c8d3f917492e-istio-data\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:44:30.238125 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:30.237939    4964 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8754d1f-9656-460d-879e-c8d3f917492e-workload-socket\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:44:30.239450 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:30.239425    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8754d1f-9656-460d-879e-c8d3f917492e-istio-envoy" pod "a8754d1f-9656-460d-879e-c8d3f917492e" (UID: "a8754d1f-9656-460d-879e-c8d3f917492e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:44:30.239565 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:30.239498    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a8754d1f-9656-460d-879e-c8d3f917492e-istio-token" pod "a8754d1f-9656-460d-879e-c8d3f917492e" (UID: "a8754d1f-9656-460d-879e-c8d3f917492e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:44:30.239945 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:30.239924    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a8754d1f-9656-460d-879e-c8d3f917492e-istio-podinfo" pod "a8754d1f-9656-460d-879e-c8d3f917492e" (UID: "a8754d1f-9656-460d-879e-c8d3f917492e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 01:44:30.240007 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:30.239929    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a8754d1f-9656-460d-879e-c8d3f917492e-kube-api-access-6zgqj" pod "a8754d1f-9656-460d-879e-c8d3f917492e" (UID: "a8754d1f-9656-460d-879e-c8d3f917492e"). InnerVolumeSpecName "kube-api-access-6zgqj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:44:30.339840 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:30.339797    4964 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a8754d1f-9656-460d-879e-c8d3f917492e-istio-token\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:44:30.339840 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:30.339834    4964 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a8754d1f-9656-460d-879e-c8d3f917492e-istio-envoy\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:44:30.340040 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:30.339849    4964 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a8754d1f-9656-460d-879e-c8d3f917492e-istio-podinfo\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:44:30.340040 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:30.339863    4964 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a8754d1f-9656-460d-879e-c8d3f917492e-istiod-ca-cert\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:44:30.340040 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:30.339880    4964 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6zgqj\" (UniqueName: \"kubernetes.io/projected/a8754d1f-9656-460d-879e-c8d3f917492e-kube-api-access-6zgqj\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:44:30.618545 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:30.618455    4964 generic.go:356] "Generic (PLEG): container finished" podID="a8754d1f-9656-460d-879e-c8d3f917492e" containerID="f487c9e5f1066adefa450e1d49631a6c1cd66173a7c3e2ab10c9f9dbd71debf2" exitCode=0
Oct 31 01:44:30.618545 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:30.618508    4964 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-b4hfr"
Oct 31 01:44:30.618755 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:30.618537    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-b4hfr" event={"ID":"a8754d1f-9656-460d-879e-c8d3f917492e","Type":"ContainerDied","Data":"f487c9e5f1066adefa450e1d49631a6c1cd66173a7c3e2ab10c9f9dbd71debf2"}
Oct 31 01:44:30.618755 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:30.618592    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-b4hfr" event={"ID":"a8754d1f-9656-460d-879e-c8d3f917492e","Type":"ContainerDied","Data":"bb319f0db707e4ba7769ed4b949f6b30fcb4dbc9626e3f7732a8bb6406599362"}
Oct 31 01:44:30.618755 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:30.618613    4964 scope.go:122] "RemoveContainer" containerID="f487c9e5f1066adefa450e1d49631a6c1cd66173a7c3e2ab10c9f9dbd71debf2"
Oct 31 01:44:30.629939 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:30.629913    4964 scope.go:122] "RemoveContainer" containerID="f487c9e5f1066adefa450e1d49631a6c1cd66173a7c3e2ab10c9f9dbd71debf2"
Oct 31 01:44:30.630242 ip-10-0-133-197 kubenswrapper[4964]: E1031 01:44:30.630214    4964 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f487c9e5f1066adefa450e1d49631a6c1cd66173a7c3e2ab10c9f9dbd71debf2\": container with ID starting with f487c9e5f1066adefa450e1d49631a6c1cd66173a7c3e2ab10c9f9dbd71debf2 not found: ID does not exist" containerID="f487c9e5f1066adefa450e1d49631a6c1cd66173a7c3e2ab10c9f9dbd71debf2"
Oct 31 01:44:30.630304 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:30.630242    4964 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f487c9e5f1066adefa450e1d49631a6c1cd66173a7c3e2ab10c9f9dbd71debf2"} err="failed to get container status \"f487c9e5f1066adefa450e1d49631a6c1cd66173a7c3e2ab10c9f9dbd71debf2\": rpc error: code = NotFound desc = could not find container \"f487c9e5f1066adefa450e1d49631a6c1cd66173a7c3e2ab10c9f9dbd71debf2\": container with ID starting with f487c9e5f1066adefa450e1d49631a6c1cd66173a7c3e2ab10c9f9dbd71debf2 not found: ID does not exist"
Oct 31 01:44:30.784019 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:30.783984    4964 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-b4hfr"]
Oct 31 01:44:30.787089 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:30.787064    4964 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-b4hfr"]
Oct 31 01:44:31.777512 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:31.777483    4964 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a8754d1f-9656-460d-879e-c8d3f917492e" path="/var/lib/kubelet/pods/a8754d1f-9656-460d-879e-c8d3f917492e/volumes"
Oct 31 01:44:34.106273 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:34.106236    4964 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-71bc98-7fbd6bc889-zp8fz"]
Oct 31 01:44:34.107024 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:34.107009    4964 cpu_manager.go:414] "containerMap: removing container" podUID="940b399a-de68-4440-a629-df5ec46dc796" containerName="istio-proxy"
Oct 31 01:44:34.107081 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:34.107027    4964 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="940b399a-de68-4440-a629-df5ec46dc796" containerName="istio-proxy"
Oct 31 01:44:34.107081 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:34.107040    4964 cpu_manager.go:414] "containerMap: removing container" podUID="503330d0-a5f9-470b-8fd6-3ad2e4cf4698" containerName="istio-proxy"
Oct 31 01:44:34.107081 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:34.107047    4964 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="503330d0-a5f9-470b-8fd6-3ad2e4cf4698" containerName="istio-proxy"
Oct 31 01:44:34.107081 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:34.107072    4964 cpu_manager.go:414] "containerMap: removing container" podUID="5ab8d4bd-eefa-4a6c-8b3a-34593e689827" containerName="istio-proxy"
Oct 31 01:44:34.107248 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:34.107082    4964 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5ab8d4bd-eefa-4a6c-8b3a-34593e689827" containerName="istio-proxy"
Oct 31 01:44:34.107248 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:34.107115    4964 cpu_manager.go:414] "containerMap: removing container" podUID="cf5d4bc6-4111-4c96-8ab9-27a461e3e000" containerName="maas-api"
Oct 31 01:44:34.107248 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:34.107122    4964 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cf5d4bc6-4111-4c96-8ab9-27a461e3e000" containerName="maas-api"
Oct 31 01:44:34.107248 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:34.107145    4964 cpu_manager.go:414] "containerMap: removing container" podUID="a8754d1f-9656-460d-879e-c8d3f917492e" containerName="istio-proxy"
Oct 31 01:44:34.107248 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:34.107152    4964 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a8754d1f-9656-460d-879e-c8d3f917492e" containerName="istio-proxy"
Oct 31 01:44:34.107432 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:34.107259    4964 memory_manager.go:365] "RemoveStaleState removing state" podUID="cf5d4bc6-4111-4c96-8ab9-27a461e3e000" containerName="maas-api"
Oct 31 01:44:34.107432 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:34.107271    4964 memory_manager.go:365] "RemoveStaleState removing state" podUID="b8249620-2e20-4e80-a8b8-7d8e566f8973" containerName="istio-proxy"
Oct 31 01:44:34.107432 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:34.107281    4964 memory_manager.go:365] "RemoveStaleState removing state" podUID="a8754d1f-9656-460d-879e-c8d3f917492e" containerName="istio-proxy"
Oct 31 01:44:34.107432 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:34.107295    4964 memory_manager.go:365] "RemoveStaleState removing state" podUID="940b399a-de68-4440-a629-df5ec46dc796" containerName="istio-proxy"
Oct 31 01:44:34.107432 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:34.107304    4964 memory_manager.go:365] "RemoveStaleState removing state" podUID="503330d0-a5f9-470b-8fd6-3ad2e4cf4698" containerName="istio-proxy"
Oct 31 01:44:34.112593 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:34.112568    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-71bc98-7fbd6bc889-zp8fz"
Oct 31 01:44:34.165294 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:34.165266    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-71bc98-7fbd6bc889-zp8fz"]
Oct 31 01:44:34.182264 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:34.182238    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/26611751-3a0b-488f-a67e-5a79bb448cbb-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w1-71bc98-7fbd6bc889-zp8fz\" (UID: \"26611751-3a0b-488f-a67e-5a79bb448cbb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-71bc98-7fbd6bc889-zp8fz"
Oct 31 01:44:34.182411 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:34.182297    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z8d6q\" (UniqueName: \"kubernetes.io/projected/26611751-3a0b-488f-a67e-5a79bb448cbb-kube-api-access-z8d6q\") pod \"maas-api-e2e-shared-b-w1-71bc98-7fbd6bc889-zp8fz\" (UID: \"26611751-3a0b-488f-a67e-5a79bb448cbb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-71bc98-7fbd6bc889-zp8fz"
Oct 31 01:44:34.182478 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:34.182405    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/26611751-3a0b-488f-a67e-5a79bb448cbb-metrics-certs\") pod \"maas-api-e2e-shared-b-w1-71bc98-7fbd6bc889-zp8fz\" (UID: \"26611751-3a0b-488f-a67e-5a79bb448cbb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-71bc98-7fbd6bc889-zp8fz"
Oct 31 01:44:34.182478 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:34.182441    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/26611751-3a0b-488f-a67e-5a79bb448cbb-maas-api-tls\") pod \"maas-api-e2e-shared-b-w1-71bc98-7fbd6bc889-zp8fz\" (UID: \"26611751-3a0b-488f-a67e-5a79bb448cbb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-71bc98-7fbd6bc889-zp8fz"
Oct 31 01:44:34.211639 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:34.211604    4964 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w1-71bc98-serving-cert\""
Oct 31 01:44:34.283905 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:34.283875    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/26611751-3a0b-488f-a67e-5a79bb448cbb-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w1-71bc98-7fbd6bc889-zp8fz\" (UID: \"26611751-3a0b-488f-a67e-5a79bb448cbb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-71bc98-7fbd6bc889-zp8fz"
Oct 31 01:44:34.284040 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:34.283935    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z8d6q\" (UniqueName: \"kubernetes.io/projected/26611751-3a0b-488f-a67e-5a79bb448cbb-kube-api-access-z8d6q\") pod \"maas-api-e2e-shared-b-w1-71bc98-7fbd6bc889-zp8fz\" (UID: \"26611751-3a0b-488f-a67e-5a79bb448cbb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-71bc98-7fbd6bc889-zp8fz"
Oct 31 01:44:34.284040 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:34.283978    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/26611751-3a0b-488f-a67e-5a79bb448cbb-metrics-certs\") pod \"maas-api-e2e-shared-b-w1-71bc98-7fbd6bc889-zp8fz\" (UID: \"26611751-3a0b-488f-a67e-5a79bb448cbb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-71bc98-7fbd6bc889-zp8fz"
Oct 31 01:44:34.284040 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:34.284012    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/26611751-3a0b-488f-a67e-5a79bb448cbb-maas-api-tls\") pod \"maas-api-e2e-shared-b-w1-71bc98-7fbd6bc889-zp8fz\" (UID: \"26611751-3a0b-488f-a67e-5a79bb448cbb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-71bc98-7fbd6bc889-zp8fz"
Oct 31 01:44:34.284585 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:34.284566    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/26611751-3a0b-488f-a67e-5a79bb448cbb-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w1-71bc98-7fbd6bc889-zp8fz\" (UID: \"26611751-3a0b-488f-a67e-5a79bb448cbb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-71bc98-7fbd6bc889-zp8fz"
Oct 31 01:44:34.286586 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:34.286564    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/26611751-3a0b-488f-a67e-5a79bb448cbb-metrics-certs\") pod \"maas-api-e2e-shared-b-w1-71bc98-7fbd6bc889-zp8fz\" (UID: \"26611751-3a0b-488f-a67e-5a79bb448cbb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-71bc98-7fbd6bc889-zp8fz"
Oct 31 01:44:34.286689 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:34.286593    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/26611751-3a0b-488f-a67e-5a79bb448cbb-maas-api-tls\") pod \"maas-api-e2e-shared-b-w1-71bc98-7fbd6bc889-zp8fz\" (UID: \"26611751-3a0b-488f-a67e-5a79bb448cbb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-71bc98-7fbd6bc889-zp8fz"
Oct 31 01:44:34.340064 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:34.340035    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z8d6q\" (UniqueName: \"kubernetes.io/projected/26611751-3a0b-488f-a67e-5a79bb448cbb-kube-api-access-z8d6q\") pod \"maas-api-e2e-shared-b-w1-71bc98-7fbd6bc889-zp8fz\" (UID: \"26611751-3a0b-488f-a67e-5a79bb448cbb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-71bc98-7fbd6bc889-zp8fz"
Oct 31 01:44:34.427587 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:34.427510    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-71bc98-7fbd6bc889-zp8fz"
Oct 31 01:44:34.491099 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:34.490597    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:44:34.807103 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:34.807080    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-71bc98-7fbd6bc889-zp8fz"]
Oct 31 01:44:34.809026 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:44:34.808999    4964 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26611751_3a0b_488f_a67e_5a79bb448cbb.slice/crio-558e38fd130b73b9dcf3cb057b5dc9fc0f6151d5e9860ad7033fd563500d06dc WatchSource:0}: Error finding container 558e38fd130b73b9dcf3cb057b5dc9fc0f6151d5e9860ad7033fd563500d06dc: Status 404 returned error can't find the container with id 558e38fd130b73b9dcf3cb057b5dc9fc0f6151d5e9860ad7033fd563500d06dc
Oct 31 01:44:35.653915 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:35.653875    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-71bc98-7fbd6bc889-zp8fz" event={"ID":"26611751-3a0b-488f-a67e-5a79bb448cbb","Type":"ContainerStarted","Data":"558e38fd130b73b9dcf3cb057b5dc9fc0f6151d5e9860ad7033fd563500d06dc"}
Oct 31 01:44:36.661294 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:36.661262    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-71bc98-7fbd6bc889-zp8fz" event={"ID":"26611751-3a0b-488f-a67e-5a79bb448cbb","Type":"ContainerStarted","Data":"691a01e8e6ae5458f4153dc526e7deca689e0f73535fd4faa732ec66feaddc41"}
Oct 31 01:44:36.661720 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:36.661338    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-71bc98-7fbd6bc889-zp8fz"
Oct 31 01:44:36.776089 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:36.776039    4964 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-71bc98-7fbd6bc889-zp8fz" podStartSLOduration=1.874768057 podStartE2EDuration="2.776025843s" totalImagesPullingTime="901.257786ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:44:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:44:36.775087426 +0000 UTC m=+2445.251653805" watchObservedRunningTime="2026-10-31 01:44:36.776025843 +0000 UTC m=+2445.252592222"
Oct 31 01:44:39.317744 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:39.317702    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:44:41.237691 ip-10-0-133-197 kubenswrapper[4964]: E1031 01:44:41.237654    4964 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q56x5" containerName="extractor"
Oct 31 01:44:41.449028 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:41.448994    4964 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w1-71bc98-openshift-default-84786b6cdc-lstqn"]
Oct 31 01:44:41.453288 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:41.453265    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-71bc98-openshift-default-84786b6cdc-lstqn"
Oct 31 01:44:41.505303 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:41.505253    4964 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w1-71bc98-openshift-default-dockercfg-dbm7d\""
Oct 31 01:44:41.558060 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:41.558035    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-71bc98-openshift-default-84786b6cdc-lstqn"]
Oct 31 01:44:41.565408 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:41.565386    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4a07c54-1e5b-4aef-b912-dcbc3930ae43-credential-socket\") pod \"e2e-shared-b-w1-71bc98-openshift-default-84786b6cdc-lstqn\" (UID: \"c4a07c54-1e5b-4aef-b912-dcbc3930ae43\") " pod="openshift-ingress/e2e-shared-b-w1-71bc98-openshift-default-84786b6cdc-lstqn"
Oct 31 01:44:41.565490 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:41.565427    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c4a07c54-1e5b-4aef-b912-dcbc3930ae43-istio-data\") pod \"e2e-shared-b-w1-71bc98-openshift-default-84786b6cdc-lstqn\" (UID: \"c4a07c54-1e5b-4aef-b912-dcbc3930ae43\") " pod="openshift-ingress/e2e-shared-b-w1-71bc98-openshift-default-84786b6cdc-lstqn"
Oct 31 01:44:41.565490 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:41.565455    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d5ftv\" (UniqueName: \"kubernetes.io/projected/c4a07c54-1e5b-4aef-b912-dcbc3930ae43-kube-api-access-d5ftv\") pod \"e2e-shared-b-w1-71bc98-openshift-default-84786b6cdc-lstqn\" (UID: \"c4a07c54-1e5b-4aef-b912-dcbc3930ae43\") " pod="openshift-ingress/e2e-shared-b-w1-71bc98-openshift-default-84786b6cdc-lstqn"
Oct 31 01:44:41.565590 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:41.565556    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4a07c54-1e5b-4aef-b912-dcbc3930ae43-workload-socket\") pod \"e2e-shared-b-w1-71bc98-openshift-default-84786b6cdc-lstqn\" (UID: \"c4a07c54-1e5b-4aef-b912-dcbc3930ae43\") " pod="openshift-ingress/e2e-shared-b-w1-71bc98-openshift-default-84786b6cdc-lstqn"
Oct 31 01:44:41.565690 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:41.565677    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c4a07c54-1e5b-4aef-b912-dcbc3930ae43-istio-envoy\") pod \"e2e-shared-b-w1-71bc98-openshift-default-84786b6cdc-lstqn\" (UID: \"c4a07c54-1e5b-4aef-b912-dcbc3930ae43\") " pod="openshift-ingress/e2e-shared-b-w1-71bc98-openshift-default-84786b6cdc-lstqn"
Oct 31 01:44:41.565726 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:41.565705    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c4a07c54-1e5b-4aef-b912-dcbc3930ae43-istio-token\") pod \"e2e-shared-b-w1-71bc98-openshift-default-84786b6cdc-lstqn\" (UID: \"c4a07c54-1e5b-4aef-b912-dcbc3930ae43\") " pod="openshift-ingress/e2e-shared-b-w1-71bc98-openshift-default-84786b6cdc-lstqn"
Oct 31 01:44:41.565762 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:41.565738    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c4a07c54-1e5b-4aef-b912-dcbc3930ae43-istio-podinfo\") pod \"e2e-shared-b-w1-71bc98-openshift-default-84786b6cdc-lstqn\" (UID: \"c4a07c54-1e5b-4aef-b912-dcbc3930ae43\") " pod="openshift-ingress/e2e-shared-b-w1-71bc98-openshift-default-84786b6cdc-lstqn"
Oct 31 01:44:41.565886 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:41.565862    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c4a07c54-1e5b-4aef-b912-dcbc3930ae43-workload-certs\") pod \"e2e-shared-b-w1-71bc98-openshift-default-84786b6cdc-lstqn\" (UID: \"c4a07c54-1e5b-4aef-b912-dcbc3930ae43\") " pod="openshift-ingress/e2e-shared-b-w1-71bc98-openshift-default-84786b6cdc-lstqn"
Oct 31 01:44:41.565926 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:41.565902    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c4a07c54-1e5b-4aef-b912-dcbc3930ae43-istiod-ca-cert\") pod \"e2e-shared-b-w1-71bc98-openshift-default-84786b6cdc-lstqn\" (UID: \"c4a07c54-1e5b-4aef-b912-dcbc3930ae43\") " pod="openshift-ingress/e2e-shared-b-w1-71bc98-openshift-default-84786b6cdc-lstqn"
Oct 31 01:44:41.667019 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:41.666983    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c4a07c54-1e5b-4aef-b912-dcbc3930ae43-istio-envoy\") pod \"e2e-shared-b-w1-71bc98-openshift-default-84786b6cdc-lstqn\" (UID: \"c4a07c54-1e5b-4aef-b912-dcbc3930ae43\") " pod="openshift-ingress/e2e-shared-b-w1-71bc98-openshift-default-84786b6cdc-lstqn"
Oct 31 01:44:41.667207 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:41.667034    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c4a07c54-1e5b-4aef-b912-dcbc3930ae43-istio-token\") pod \"e2e-shared-b-w1-71bc98-openshift-default-84786b6cdc-lstqn\" (UID: \"c4a07c54-1e5b-4aef-b912-dcbc3930ae43\") " pod="openshift-ingress/e2e-shared-b-w1-71bc98-openshift-default-84786b6cdc-lstqn"
Oct 31 01:44:41.667207 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:41.667069    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c4a07c54-1e5b-4aef-b912-dcbc3930ae43-istio-podinfo\") pod \"e2e-shared-b-w1-71bc98-openshift-default-84786b6cdc-lstqn\" (UID: \"c4a07c54-1e5b-4aef-b912-dcbc3930ae43\") " pod="openshift-ingress/e2e-shared-b-w1-71bc98-openshift-default-84786b6cdc-lstqn"
Oct 31 01:44:41.667207 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:41.667161    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c4a07c54-1e5b-4aef-b912-dcbc3930ae43-workload-certs\") pod \"e2e-shared-b-w1-71bc98-openshift-default-84786b6cdc-lstqn\" (UID: \"c4a07c54-1e5b-4aef-b912-dcbc3930ae43\") " pod="openshift-ingress/e2e-shared-b-w1-71bc98-openshift-default-84786b6cdc-lstqn"
Oct 31 01:44:41.667207 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:41.667193    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c4a07c54-1e5b-4aef-b912-dcbc3930ae43-istiod-ca-cert\") pod \"e2e-shared-b-w1-71bc98-openshift-default-84786b6cdc-lstqn\" (UID: \"c4a07c54-1e5b-4aef-b912-dcbc3930ae43\") " pod="openshift-ingress/e2e-shared-b-w1-71bc98-openshift-default-84786b6cdc-lstqn"
Oct 31 01:44:41.667471 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:41.667242    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4a07c54-1e5b-4aef-b912-dcbc3930ae43-credential-socket\") pod \"e2e-shared-b-w1-71bc98-openshift-default-84786b6cdc-lstqn\" (UID: \"c4a07c54-1e5b-4aef-b912-dcbc3930ae43\") " pod="openshift-ingress/e2e-shared-b-w1-71bc98-openshift-default-84786b6cdc-lstqn"
Oct 31 01:44:41.667471 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:41.667278    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c4a07c54-1e5b-4aef-b912-dcbc3930ae43-istio-data\") pod \"e2e-shared-b-w1-71bc98-openshift-default-84786b6cdc-lstqn\" (UID: \"c4a07c54-1e5b-4aef-b912-dcbc3930ae43\") " pod="openshift-ingress/e2e-shared-b-w1-71bc98-openshift-default-84786b6cdc-lstqn"
Oct 31 01:44:41.667471 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:41.667303    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d5ftv\" (UniqueName: \"kubernetes.io/projected/c4a07c54-1e5b-4aef-b912-dcbc3930ae43-kube-api-access-d5ftv\") pod \"e2e-shared-b-w1-71bc98-openshift-default-84786b6cdc-lstqn\" (UID: \"c4a07c54-1e5b-4aef-b912-dcbc3930ae43\") " pod="openshift-ingress/e2e-shared-b-w1-71bc98-openshift-default-84786b6cdc-lstqn"
Oct 31 01:44:41.667471 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:41.667400    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4a07c54-1e5b-4aef-b912-dcbc3930ae43-workload-socket\") pod \"e2e-shared-b-w1-71bc98-openshift-default-84786b6cdc-lstqn\" (UID: \"c4a07c54-1e5b-4aef-b912-dcbc3930ae43\") " pod="openshift-ingress/e2e-shared-b-w1-71bc98-openshift-default-84786b6cdc-lstqn"
Oct 31 01:44:41.667676 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:41.667580    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c4a07c54-1e5b-4aef-b912-dcbc3930ae43-workload-certs\") pod \"e2e-shared-b-w1-71bc98-openshift-default-84786b6cdc-lstqn\" (UID: \"c4a07c54-1e5b-4aef-b912-dcbc3930ae43\") " pod="openshift-ingress/e2e-shared-b-w1-71bc98-openshift-default-84786b6cdc-lstqn"
Oct 31 01:44:41.667762 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:41.667739    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4a07c54-1e5b-4aef-b912-dcbc3930ae43-credential-socket\") pod \"e2e-shared-b-w1-71bc98-openshift-default-84786b6cdc-lstqn\" (UID: \"c4a07c54-1e5b-4aef-b912-dcbc3930ae43\") " pod="openshift-ingress/e2e-shared-b-w1-71bc98-openshift-default-84786b6cdc-lstqn"
Oct 31 01:44:41.667857 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:41.667841    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c4a07c54-1e5b-4aef-b912-dcbc3930ae43-istio-data\") pod \"e2e-shared-b-w1-71bc98-openshift-default-84786b6cdc-lstqn\" (UID: \"c4a07c54-1e5b-4aef-b912-dcbc3930ae43\") " pod="openshift-ingress/e2e-shared-b-w1-71bc98-openshift-default-84786b6cdc-lstqn"
Oct 31 01:44:41.668032 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:41.668010    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4a07c54-1e5b-4aef-b912-dcbc3930ae43-workload-socket\") pod \"e2e-shared-b-w1-71bc98-openshift-default-84786b6cdc-lstqn\" (UID: \"c4a07c54-1e5b-4aef-b912-dcbc3930ae43\") " pod="openshift-ingress/e2e-shared-b-w1-71bc98-openshift-default-84786b6cdc-lstqn"
Oct 31 01:44:41.668137 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:41.668119    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c4a07c54-1e5b-4aef-b912-dcbc3930ae43-istiod-ca-cert\") pod \"e2e-shared-b-w1-71bc98-openshift-default-84786b6cdc-lstqn\" (UID: \"c4a07c54-1e5b-4aef-b912-dcbc3930ae43\") " pod="openshift-ingress/e2e-shared-b-w1-71bc98-openshift-default-84786b6cdc-lstqn"
Oct 31 01:44:41.669431 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:41.669406    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c4a07c54-1e5b-4aef-b912-dcbc3930ae43-istio-envoy\") pod \"e2e-shared-b-w1-71bc98-openshift-default-84786b6cdc-lstqn\" (UID: \"c4a07c54-1e5b-4aef-b912-dcbc3930ae43\") " pod="openshift-ingress/e2e-shared-b-w1-71bc98-openshift-default-84786b6cdc-lstqn"
Oct 31 01:44:41.670124 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:41.670102    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c4a07c54-1e5b-4aef-b912-dcbc3930ae43-istio-podinfo\") pod \"e2e-shared-b-w1-71bc98-openshift-default-84786b6cdc-lstqn\" (UID: \"c4a07c54-1e5b-4aef-b912-dcbc3930ae43\") " pod="openshift-ingress/e2e-shared-b-w1-71bc98-openshift-default-84786b6cdc-lstqn"
Oct 31 01:44:41.723921 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:41.723887    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c4a07c54-1e5b-4aef-b912-dcbc3930ae43-istio-token\") pod \"e2e-shared-b-w1-71bc98-openshift-default-84786b6cdc-lstqn\" (UID: \"c4a07c54-1e5b-4aef-b912-dcbc3930ae43\") " pod="openshift-ingress/e2e-shared-b-w1-71bc98-openshift-default-84786b6cdc-lstqn"
Oct 31 01:44:41.724036 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:41.723954    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d5ftv\" (UniqueName: \"kubernetes.io/projected/c4a07c54-1e5b-4aef-b912-dcbc3930ae43-kube-api-access-d5ftv\") pod \"e2e-shared-b-w1-71bc98-openshift-default-84786b6cdc-lstqn\" (UID: \"c4a07c54-1e5b-4aef-b912-dcbc3930ae43\") " pod="openshift-ingress/e2e-shared-b-w1-71bc98-openshift-default-84786b6cdc-lstqn"
Oct 31 01:44:41.766929 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:41.766864    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-71bc98-openshift-default-84786b6cdc-lstqn"
Oct 31 01:44:42.201660 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:42.201633    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-71bc98-openshift-default-84786b6cdc-lstqn"]
Oct 31 01:44:42.202748 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:44:42.202721    4964 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc4a07c54_1e5b_4aef_b912_dcbc3930ae43.slice/crio-f7907db3eb97c7dc8c99beb190b38c3e13c8408b9eb0440eaf8cec22b732928d WatchSource:0}: Error finding container f7907db3eb97c7dc8c99beb190b38c3e13c8408b9eb0440eaf8cec22b732928d: Status 404 returned error can't find the container with id f7907db3eb97c7dc8c99beb190b38c3e13c8408b9eb0440eaf8cec22b732928d
Oct 31 01:44:42.205001 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:42.204964    4964 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 01:44:42.205109 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:42.205035    4964 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 01:44:42.205109 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:42.205079    4964 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 01:44:42.486491 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:42.486451    4964 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w1-71bc98/tenantcfg-model-71bc98-kserve-7b7d7bc7bf-kqn2l"]
Oct 31 01:44:42.572381 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:42.572333    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:44:42.675588 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:42.675560    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-71bc98-7fbd6bc889-zp8fz"
Oct 31 01:44:42.704414 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:42.704382    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-71bc98-openshift-default-84786b6cdc-lstqn" event={"ID":"c4a07c54-1e5b-4aef-b912-dcbc3930ae43","Type":"ContainerStarted","Data":"07df99b8ce7681f561f854d5f7b7b5276418b4d71813ef86a1675e153712b383"}
Oct 31 01:44:42.704574 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:42.704447    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-71bc98-openshift-default-84786b6cdc-lstqn" event={"ID":"c4a07c54-1e5b-4aef-b912-dcbc3930ae43","Type":"ContainerStarted","Data":"f7907db3eb97c7dc8c99beb190b38c3e13c8408b9eb0440eaf8cec22b732928d"}
Oct 31 01:44:42.767522 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:42.767425    4964 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w1-71bc98-openshift-default-84786b6cdc-lstqn"
Oct 31 01:44:42.926093 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:42.926046    4964 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w1-71bc98-openshift-default-84786b6cdc-lstqn" podStartSLOduration=1.926028726 podStartE2EDuration="1.926028726s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:44:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:44:42.922096734 +0000 UTC m=+2451.398663125" watchObservedRunningTime="2026-10-31 01:44:42.926028726 +0000 UTC m=+2451.402595107"
Oct 31 01:44:43.017340 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:43.017191    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:44:43.652377 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:43.648969    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:44:43.770457 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:43.770411    4964 patch_prober.go:29] interesting pod/e2e-shared-b-w1-71bc98-openshift-default-84786b6cdc-lstqn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused" start-of-body=
Oct 31 01:44:43.770691 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:43.770467    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-71bc98-openshift-default-84786b6cdc-lstqn" podUID="c4a07c54-1e5b-4aef-b912-dcbc3930ae43" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused"
Oct 31 01:44:43.849574 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:43.849502    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:44:44.029773 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:44.029736    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:44:44.417310 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:44.416906    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:44:44.642449 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:44.642418    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:44:44.767887 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:44.767858    4964 patch_prober.go:29] interesting pod/e2e-shared-b-w1-71bc98-openshift-default-84786b6cdc-lstqn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused" start-of-body=
Oct 31 01:44:44.768250 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:44.767905    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-71bc98-openshift-default-84786b6cdc-lstqn" podUID="c4a07c54-1e5b-4aef-b912-dcbc3930ae43" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused"
Oct 31 01:44:45.629728 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:45.629685    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:44:45.767895 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:45.767856    4964 patch_prober.go:29] interesting pod/e2e-shared-b-w1-71bc98-openshift-default-84786b6cdc-lstqn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused" start-of-body=
Oct 31 01:44:45.768292 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:45.767928    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-71bc98-openshift-default-84786b6cdc-lstqn" podUID="c4a07c54-1e5b-4aef-b912-dcbc3930ae43" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused"
Oct 31 01:44:46.069727 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:46.069694    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:44:46.248577 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:46.248539    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:44:46.664569 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:46.664534    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:44:46.767451 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:46.767415    4964 patch_prober.go:29] interesting pod/e2e-shared-b-w1-71bc98-openshift-default-84786b6cdc-lstqn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused" start-of-body=
Oct 31 01:44:46.767633 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:46.767477    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-71bc98-openshift-default-84786b6cdc-lstqn" podUID="c4a07c54-1e5b-4aef-b912-dcbc3930ae43" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused"
Oct 31 01:44:47.767575 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:47.767541    4964 patch_prober.go:29] interesting pod/e2e-shared-b-w1-71bc98-openshift-default-84786b6cdc-lstqn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused" start-of-body=
Oct 31 01:44:47.768053 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:47.767589    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-71bc98-openshift-default-84786b6cdc-lstqn" podUID="c4a07c54-1e5b-4aef-b912-dcbc3930ae43" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused"
Oct 31 01:44:48.522093 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:48.522055    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:44:48.708124 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:48.708086    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:44:48.773074 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:48.772985    4964 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w1-71bc98-openshift-default-84786b6cdc-lstqn"
Oct 31 01:44:48.773529 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:48.773443    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w1-71bc98-openshift-default-84786b6cdc-lstqn"
Oct 31 01:44:48.774145 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:48.774122    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w1-71bc98-openshift-default-84786b6cdc-lstqn"
Oct 31 01:44:48.884989 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:48.884958    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:44:50.015728 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:50.015692    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:44:50.569920 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:50.569886    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:44:51.779243 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:51.779201    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:44:52.061660 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:52.061549    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:44:52.168710 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:52.168672    4964 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-c0daf4-fb8556c8c-k4ggk"]
Oct 31 01:44:52.172944 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:52.172923    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-c0daf4-fb8556c8c-k4ggk"
Oct 31 01:44:52.226348 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:52.226285    4964 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-a-c0daf4-dockercfg-z7xkz\""
Oct 31 01:44:52.230747 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:52.230725    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-c0daf4-fb8556c8c-k4ggk"]
Oct 31 01:44:52.268576 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:52.268550    4964 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-a-c0daf4\""
Oct 31 01:44:52.281195 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:52.281172    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q9lxw\" (UniqueName: \"kubernetes.io/projected/2e38f4c2-b630-4a05-abb0-49fce8594ec3-kube-api-access-q9lxw\") pod \"payload-pre-processing-e2e-praxis-a-c0daf4-fb8556c8c-k4ggk\" (UID: \"2e38f4c2-b630-4a05-abb0-49fce8594ec3\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-c0daf4-fb8556c8c-k4ggk"
Oct 31 01:44:52.281359 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:52.281305    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2e38f4c2-b630-4a05-abb0-49fce8594ec3-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-c0daf4-fb8556c8c-k4ggk\" (UID: \"2e38f4c2-b630-4a05-abb0-49fce8594ec3\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-c0daf4-fb8556c8c-k4ggk"
Oct 31 01:44:52.383221 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:52.383142    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2e38f4c2-b630-4a05-abb0-49fce8594ec3-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-c0daf4-fb8556c8c-k4ggk\" (UID: \"2e38f4c2-b630-4a05-abb0-49fce8594ec3\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-c0daf4-fb8556c8c-k4ggk"
Oct 31 01:44:52.383451 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:52.383431    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q9lxw\" (UniqueName: \"kubernetes.io/projected/2e38f4c2-b630-4a05-abb0-49fce8594ec3-kube-api-access-q9lxw\") pod \"payload-pre-processing-e2e-praxis-a-c0daf4-fb8556c8c-k4ggk\" (UID: \"2e38f4c2-b630-4a05-abb0-49fce8594ec3\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-c0daf4-fb8556c8c-k4ggk"
Oct 31 01:44:52.383953 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:52.383935    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2e38f4c2-b630-4a05-abb0-49fce8594ec3-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-c0daf4-fb8556c8c-k4ggk\" (UID: \"2e38f4c2-b630-4a05-abb0-49fce8594ec3\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-c0daf4-fb8556c8c-k4ggk"
Oct 31 01:44:52.440526 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:52.440495    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q9lxw\" (UniqueName: \"kubernetes.io/projected/2e38f4c2-b630-4a05-abb0-49fce8594ec3-kube-api-access-q9lxw\") pod \"payload-pre-processing-e2e-praxis-a-c0daf4-fb8556c8c-k4ggk\" (UID: \"2e38f4c2-b630-4a05-abb0-49fce8594ec3\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-c0daf4-fb8556c8c-k4ggk"
Oct 31 01:44:52.487148 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:52.487118    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-c0daf4-fb8556c8c-k4ggk"
Oct 31 01:44:52.872451 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:52.872420    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-c0daf4-fb8556c8c-k4ggk"]
Oct 31 01:44:52.873931 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:44:52.873907    4964 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e38f4c2_b630_4a05_abb0_49fce8594ec3.slice/crio-dfa503c9274da2fee10e3d820d10522b6b69f5e3f8e97ca1c621aae42a8830bf WatchSource:0}: Error finding container dfa503c9274da2fee10e3d820d10522b6b69f5e3f8e97ca1c621aae42a8830bf: Status 404 returned error can't find the container with id dfa503c9274da2fee10e3d820d10522b6b69f5e3f8e97ca1c621aae42a8830bf
Oct 31 01:44:53.059872 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:53.059823    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:44:53.265033 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:53.264970    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:44:53.456389 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:53.453727    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:44:53.821268 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:53.821236    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-c0daf4-fb8556c8c-k4ggk" event={"ID":"2e38f4c2-b630-4a05-abb0-49fce8594ec3","Type":"ContainerStarted","Data":"6fed9f9efa0d238b8bfe654095d28e168804e352e9bb7846329132442452e608"}
Oct 31 01:44:53.821459 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:53.821294    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-c0daf4-fb8556c8c-k4ggk" event={"ID":"2e38f4c2-b630-4a05-abb0-49fce8594ec3","Type":"ContainerStarted","Data":"dfa503c9274da2fee10e3d820d10522b6b69f5e3f8e97ca1c621aae42a8830bf"}
Oct 31 01:44:53.821459 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:53.821342    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-c0daf4-fb8556c8c-k4ggk"
Oct 31 01:44:53.937070 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:53.937014    4964 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-c0daf4-fb8556c8c-k4ggk" podStartSLOduration=1.936999997 podStartE2EDuration="1.936999997s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:44:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:44:53.93377387 +0000 UTC m=+2462.410340289" watchObservedRunningTime="2026-10-31 01:44:53.936999997 +0000 UTC m=+2462.413566405"
Oct 31 01:44:54.994725 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:54.994684    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:44:55.664676 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:55.664622    4964 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-a-9d7b2c-openshift-default-76c779b789-tvtzf"]
Oct 31 01:44:55.668790 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:55.668768    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-9d7b2c-openshift-default-76c779b789-tvtzf"
Oct 31 01:44:55.740573 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:55.740545    4964 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-a-9d7b2c-openshift-default-dockercfg-4gdhm\""
Oct 31 01:44:55.781099 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:55.781069    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-9d7b2c-openshift-default-76c779b789-tvtzf"]
Oct 31 01:44:55.806457 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:55.806395    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:44:55.826857 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:55.826819    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/79d0fdfe-95ba-4800-9636-68f7afc6829e-istio-data\") pod \"e2e-praxis-a-9d7b2c-openshift-default-76c779b789-tvtzf\" (UID: \"79d0fdfe-95ba-4800-9636-68f7afc6829e\") " pod="openshift-ingress/e2e-praxis-a-9d7b2c-openshift-default-76c779b789-tvtzf"
Oct 31 01:44:55.827287 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:55.827263    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/79d0fdfe-95ba-4800-9636-68f7afc6829e-istio-podinfo\") pod \"e2e-praxis-a-9d7b2c-openshift-default-76c779b789-tvtzf\" (UID: \"79d0fdfe-95ba-4800-9636-68f7afc6829e\") " pod="openshift-ingress/e2e-praxis-a-9d7b2c-openshift-default-76c779b789-tvtzf"
Oct 31 01:44:55.827486 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:55.827468    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/79d0fdfe-95ba-4800-9636-68f7afc6829e-workload-socket\") pod \"e2e-praxis-a-9d7b2c-openshift-default-76c779b789-tvtzf\" (UID: \"79d0fdfe-95ba-4800-9636-68f7afc6829e\") " pod="openshift-ingress/e2e-praxis-a-9d7b2c-openshift-default-76c779b789-tvtzf"
Oct 31 01:44:55.827598 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:55.827583    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jdmlx\" (UniqueName: \"kubernetes.io/projected/79d0fdfe-95ba-4800-9636-68f7afc6829e-kube-api-access-jdmlx\") pod \"e2e-praxis-a-9d7b2c-openshift-default-76c779b789-tvtzf\" (UID: \"79d0fdfe-95ba-4800-9636-68f7afc6829e\") " pod="openshift-ingress/e2e-praxis-a-9d7b2c-openshift-default-76c779b789-tvtzf"
Oct 31 01:44:55.827759 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:55.827745    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/79d0fdfe-95ba-4800-9636-68f7afc6829e-istio-envoy\") pod \"e2e-praxis-a-9d7b2c-openshift-default-76c779b789-tvtzf\" (UID: \"79d0fdfe-95ba-4800-9636-68f7afc6829e\") " pod="openshift-ingress/e2e-praxis-a-9d7b2c-openshift-default-76c779b789-tvtzf"
Oct 31 01:44:55.827867 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:55.827851    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/79d0fdfe-95ba-4800-9636-68f7afc6829e-istiod-ca-cert\") pod \"e2e-praxis-a-9d7b2c-openshift-default-76c779b789-tvtzf\" (UID: \"79d0fdfe-95ba-4800-9636-68f7afc6829e\") " pod="openshift-ingress/e2e-praxis-a-9d7b2c-openshift-default-76c779b789-tvtzf"
Oct 31 01:44:55.828003 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:55.827987    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/79d0fdfe-95ba-4800-9636-68f7afc6829e-workload-certs\") pod \"e2e-praxis-a-9d7b2c-openshift-default-76c779b789-tvtzf\" (UID: \"79d0fdfe-95ba-4800-9636-68f7afc6829e\") " pod="openshift-ingress/e2e-praxis-a-9d7b2c-openshift-default-76c779b789-tvtzf"
Oct 31 01:44:55.828124 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:55.828110    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/79d0fdfe-95ba-4800-9636-68f7afc6829e-credential-socket\") pod \"e2e-praxis-a-9d7b2c-openshift-default-76c779b789-tvtzf\" (UID: \"79d0fdfe-95ba-4800-9636-68f7afc6829e\") " pod="openshift-ingress/e2e-praxis-a-9d7b2c-openshift-default-76c779b789-tvtzf"
Oct 31 01:44:55.828236 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:55.828224    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/79d0fdfe-95ba-4800-9636-68f7afc6829e-istio-token\") pod \"e2e-praxis-a-9d7b2c-openshift-default-76c779b789-tvtzf\" (UID: \"79d0fdfe-95ba-4800-9636-68f7afc6829e\") " pod="openshift-ingress/e2e-praxis-a-9d7b2c-openshift-default-76c779b789-tvtzf"
Oct 31 01:44:55.929941 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:55.929857    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/79d0fdfe-95ba-4800-9636-68f7afc6829e-credential-socket\") pod \"e2e-praxis-a-9d7b2c-openshift-default-76c779b789-tvtzf\" (UID: \"79d0fdfe-95ba-4800-9636-68f7afc6829e\") " pod="openshift-ingress/e2e-praxis-a-9d7b2c-openshift-default-76c779b789-tvtzf"
Oct 31 01:44:55.929941 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:55.929912    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/79d0fdfe-95ba-4800-9636-68f7afc6829e-istio-token\") pod \"e2e-praxis-a-9d7b2c-openshift-default-76c779b789-tvtzf\" (UID: \"79d0fdfe-95ba-4800-9636-68f7afc6829e\") " pod="openshift-ingress/e2e-praxis-a-9d7b2c-openshift-default-76c779b789-tvtzf"
Oct 31 01:44:55.930168 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:55.930016    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/79d0fdfe-95ba-4800-9636-68f7afc6829e-istio-data\") pod \"e2e-praxis-a-9d7b2c-openshift-default-76c779b789-tvtzf\" (UID: \"79d0fdfe-95ba-4800-9636-68f7afc6829e\") " pod="openshift-ingress/e2e-praxis-a-9d7b2c-openshift-default-76c779b789-tvtzf"
Oct 31 01:44:55.930168 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:55.930129    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/79d0fdfe-95ba-4800-9636-68f7afc6829e-istio-podinfo\") pod \"e2e-praxis-a-9d7b2c-openshift-default-76c779b789-tvtzf\" (UID: \"79d0fdfe-95ba-4800-9636-68f7afc6829e\") " pod="openshift-ingress/e2e-praxis-a-9d7b2c-openshift-default-76c779b789-tvtzf"
Oct 31 01:44:55.930289 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:55.930187    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/79d0fdfe-95ba-4800-9636-68f7afc6829e-workload-socket\") pod \"e2e-praxis-a-9d7b2c-openshift-default-76c779b789-tvtzf\" (UID: \"79d0fdfe-95ba-4800-9636-68f7afc6829e\") " pod="openshift-ingress/e2e-praxis-a-9d7b2c-openshift-default-76c779b789-tvtzf"
Oct 31 01:44:55.930397 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:55.930302    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/79d0fdfe-95ba-4800-9636-68f7afc6829e-credential-socket\") pod \"e2e-praxis-a-9d7b2c-openshift-default-76c779b789-tvtzf\" (UID: \"79d0fdfe-95ba-4800-9636-68f7afc6829e\") " pod="openshift-ingress/e2e-praxis-a-9d7b2c-openshift-default-76c779b789-tvtzf"
Oct 31 01:44:55.930397 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:55.930345    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jdmlx\" (UniqueName: \"kubernetes.io/projected/79d0fdfe-95ba-4800-9636-68f7afc6829e-kube-api-access-jdmlx\") pod \"e2e-praxis-a-9d7b2c-openshift-default-76c779b789-tvtzf\" (UID: \"79d0fdfe-95ba-4800-9636-68f7afc6829e\") " pod="openshift-ingress/e2e-praxis-a-9d7b2c-openshift-default-76c779b789-tvtzf"
Oct 31 01:44:55.930493 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:55.930432    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/79d0fdfe-95ba-4800-9636-68f7afc6829e-istio-envoy\") pod \"e2e-praxis-a-9d7b2c-openshift-default-76c779b789-tvtzf\" (UID: \"79d0fdfe-95ba-4800-9636-68f7afc6829e\") " pod="openshift-ingress/e2e-praxis-a-9d7b2c-openshift-default-76c779b789-tvtzf"
Oct 31 01:44:55.930493 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:55.930465    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/79d0fdfe-95ba-4800-9636-68f7afc6829e-istiod-ca-cert\") pod \"e2e-praxis-a-9d7b2c-openshift-default-76c779b789-tvtzf\" (UID: \"79d0fdfe-95ba-4800-9636-68f7afc6829e\") " pod="openshift-ingress/e2e-praxis-a-9d7b2c-openshift-default-76c779b789-tvtzf"
Oct 31 01:44:55.930600 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:55.930497    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/79d0fdfe-95ba-4800-9636-68f7afc6829e-workload-socket\") pod \"e2e-praxis-a-9d7b2c-openshift-default-76c779b789-tvtzf\" (UID: \"79d0fdfe-95ba-4800-9636-68f7afc6829e\") " pod="openshift-ingress/e2e-praxis-a-9d7b2c-openshift-default-76c779b789-tvtzf"
Oct 31 01:44:55.930600 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:55.930508    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/79d0fdfe-95ba-4800-9636-68f7afc6829e-workload-certs\") pod \"e2e-praxis-a-9d7b2c-openshift-default-76c779b789-tvtzf\" (UID: \"79d0fdfe-95ba-4800-9636-68f7afc6829e\") " pod="openshift-ingress/e2e-praxis-a-9d7b2c-openshift-default-76c779b789-tvtzf"
Oct 31 01:44:55.930745 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:55.930722    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/79d0fdfe-95ba-4800-9636-68f7afc6829e-istio-data\") pod \"e2e-praxis-a-9d7b2c-openshift-default-76c779b789-tvtzf\" (UID: \"79d0fdfe-95ba-4800-9636-68f7afc6829e\") " pod="openshift-ingress/e2e-praxis-a-9d7b2c-openshift-default-76c779b789-tvtzf"
Oct 31 01:44:55.930805 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:55.930786    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/79d0fdfe-95ba-4800-9636-68f7afc6829e-workload-certs\") pod \"e2e-praxis-a-9d7b2c-openshift-default-76c779b789-tvtzf\" (UID: \"79d0fdfe-95ba-4800-9636-68f7afc6829e\") " pod="openshift-ingress/e2e-praxis-a-9d7b2c-openshift-default-76c779b789-tvtzf"
Oct 31 01:44:55.931449 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:55.931420    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/79d0fdfe-95ba-4800-9636-68f7afc6829e-istiod-ca-cert\") pod \"e2e-praxis-a-9d7b2c-openshift-default-76c779b789-tvtzf\" (UID: \"79d0fdfe-95ba-4800-9636-68f7afc6829e\") " pod="openshift-ingress/e2e-praxis-a-9d7b2c-openshift-default-76c779b789-tvtzf"
Oct 31 01:44:55.932827 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:55.932798    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/79d0fdfe-95ba-4800-9636-68f7afc6829e-istio-envoy\") pod \"e2e-praxis-a-9d7b2c-openshift-default-76c779b789-tvtzf\" (UID: \"79d0fdfe-95ba-4800-9636-68f7afc6829e\") " pod="openshift-ingress/e2e-praxis-a-9d7b2c-openshift-default-76c779b789-tvtzf"
Oct 31 01:44:55.932971 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:55.932954    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/79d0fdfe-95ba-4800-9636-68f7afc6829e-istio-podinfo\") pod \"e2e-praxis-a-9d7b2c-openshift-default-76c779b789-tvtzf\" (UID: \"79d0fdfe-95ba-4800-9636-68f7afc6829e\") " pod="openshift-ingress/e2e-praxis-a-9d7b2c-openshift-default-76c779b789-tvtzf"
Oct 31 01:44:55.989701 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:55.989673    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/79d0fdfe-95ba-4800-9636-68f7afc6829e-istio-token\") pod \"e2e-praxis-a-9d7b2c-openshift-default-76c779b789-tvtzf\" (UID: \"79d0fdfe-95ba-4800-9636-68f7afc6829e\") " pod="openshift-ingress/e2e-praxis-a-9d7b2c-openshift-default-76c779b789-tvtzf"
Oct 31 01:44:55.989897 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:55.989877    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jdmlx\" (UniqueName: \"kubernetes.io/projected/79d0fdfe-95ba-4800-9636-68f7afc6829e-kube-api-access-jdmlx\") pod \"e2e-praxis-a-9d7b2c-openshift-default-76c779b789-tvtzf\" (UID: \"79d0fdfe-95ba-4800-9636-68f7afc6829e\") " pod="openshift-ingress/e2e-praxis-a-9d7b2c-openshift-default-76c779b789-tvtzf"
Oct 31 01:44:56.081720 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:56.081685    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:44:56.282515 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:56.282483    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-9d7b2c-openshift-default-76c779b789-tvtzf"
Oct 31 01:44:56.724153 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:56.724118    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-9d7b2c-openshift-default-76c779b789-tvtzf"]
Oct 31 01:44:56.728416 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:44:56.728373    4964 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79d0fdfe_95ba_4800_9636_68f7afc6829e.slice/crio-6f38be7ae52e46f5e53ecdd3b58779e78a622a0dc70e9d82bf87d3c102f7057f WatchSource:0}: Error finding container 6f38be7ae52e46f5e53ecdd3b58779e78a622a0dc70e9d82bf87d3c102f7057f: Status 404 returned error can't find the container with id 6f38be7ae52e46f5e53ecdd3b58779e78a622a0dc70e9d82bf87d3c102f7057f
Oct 31 01:44:56.731532 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:56.731492    4964 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 01:44:56.731627 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:56.731578    4964 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 01:44:56.731627 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:56.731617    4964 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 01:44:56.850708 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:56.850667    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-9d7b2c-openshift-default-76c779b789-tvtzf" event={"ID":"79d0fdfe-95ba-4800-9636-68f7afc6829e","Type":"ContainerStarted","Data":"15e2887cfd42d5b3335dcbf8df443b50617bce918ae92dc6c7c8490d2b2440f6"}
Oct 31 01:44:56.850898 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:56.850733    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-9d7b2c-openshift-default-76c779b789-tvtzf" event={"ID":"79d0fdfe-95ba-4800-9636-68f7afc6829e","Type":"ContainerStarted","Data":"6f38be7ae52e46f5e53ecdd3b58779e78a622a0dc70e9d82bf87d3c102f7057f"}
Oct 31 01:44:56.967607 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:56.967553    4964 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-a-9d7b2c-openshift-default-76c779b789-tvtzf" podStartSLOduration=1.967536416 podStartE2EDuration="1.967536416s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:44:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:44:56.96560843 +0000 UTC m=+2465.442174811" watchObservedRunningTime="2026-10-31 01:44:56.967536416 +0000 UTC m=+2465.444102798"
Oct 31 01:44:57.283555 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:57.283526    4964 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-9d7b2c-openshift-default-76c779b789-tvtzf"
Oct 31 01:44:57.289021 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:57.288996    4964 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-9d7b2c-openshift-default-76c779b789-tvtzf"
Oct 31 01:44:57.509848 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:57.509807    4964 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w1-71bc98/tenantcfg-model-71bc98-kserve-7b7d7bc7bf-kqn2l" podUID="847d367e-516e-4240-924c-3e62c532a8d6" containerName="main" containerID="cri-o://18b53db5af341eb75b877294864d5a13cbd961e82a25657b7540bc930d628eb8" gracePeriod=45
Oct 31 01:44:57.794425 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:57.794396    4964 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-71bc98/tenantcfg-model-71bc98-kserve-7b7d7bc7bf-kqn2l"
Oct 31 01:44:57.857895 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:57.857865    4964 generic.go:356] "Generic (PLEG): container finished" podID="847d367e-516e-4240-924c-3e62c532a8d6" containerID="18b53db5af341eb75b877294864d5a13cbd961e82a25657b7540bc930d628eb8" exitCode=0
Oct 31 01:44:57.858037 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:57.857905    4964 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-71bc98/tenantcfg-model-71bc98-kserve-7b7d7bc7bf-kqn2l"
Oct 31 01:44:57.858037 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:57.857951    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-71bc98/tenantcfg-model-71bc98-kserve-7b7d7bc7bf-kqn2l" event={"ID":"847d367e-516e-4240-924c-3e62c532a8d6","Type":"ContainerDied","Data":"18b53db5af341eb75b877294864d5a13cbd961e82a25657b7540bc930d628eb8"}
Oct 31 01:44:57.858037 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:57.857991    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-71bc98/tenantcfg-model-71bc98-kserve-7b7d7bc7bf-kqn2l" event={"ID":"847d367e-516e-4240-924c-3e62c532a8d6","Type":"ContainerDied","Data":"5fc07c8937d63dd6e2b4cad8301e6fb430fad650a4b58233e16722c9688b8e2f"}
Oct 31 01:44:57.858037 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:57.858018    4964 scope.go:122] "RemoveContainer" containerID="18b53db5af341eb75b877294864d5a13cbd961e82a25657b7540bc930d628eb8"
Oct 31 01:44:57.858272 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:57.858151    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-9d7b2c-openshift-default-76c779b789-tvtzf"
Oct 31 01:44:57.859131 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:57.859087    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-9d7b2c-openshift-default-76c779b789-tvtzf"
Oct 31 01:44:57.868127 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:57.868111    4964 scope.go:122] "RemoveContainer" containerID="18b53db5af341eb75b877294864d5a13cbd961e82a25657b7540bc930d628eb8"
Oct 31 01:44:57.868431 ip-10-0-133-197 kubenswrapper[4964]: E1031 01:44:57.868410    4964 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"18b53db5af341eb75b877294864d5a13cbd961e82a25657b7540bc930d628eb8\": container with ID starting with 18b53db5af341eb75b877294864d5a13cbd961e82a25657b7540bc930d628eb8 not found: ID does not exist" containerID="18b53db5af341eb75b877294864d5a13cbd961e82a25657b7540bc930d628eb8"
Oct 31 01:44:57.868511 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:57.868442    4964 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"18b53db5af341eb75b877294864d5a13cbd961e82a25657b7540bc930d628eb8"} err="failed to get container status \"18b53db5af341eb75b877294864d5a13cbd961e82a25657b7540bc930d628eb8\": rpc error: code = NotFound desc = could not find container \"18b53db5af341eb75b877294864d5a13cbd961e82a25657b7540bc930d628eb8\": container with ID starting with 18b53db5af341eb75b877294864d5a13cbd961e82a25657b7540bc930d628eb8 not found: ID does not exist"
Oct 31 01:44:57.957684 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:57.957619    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/847d367e-516e-4240-924c-3e62c532a8d6-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/847d367e-516e-4240-924c-3e62c532a8d6-tmp-dir\") pod \"847d367e-516e-4240-924c-3e62c532a8d6\" (UID: \"847d367e-516e-4240-924c-3e62c532a8d6\") "
Oct 31 01:44:57.957684 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:57.957664    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/847d367e-516e-4240-924c-3e62c532a8d6-tls-certs\" (UniqueName: \"kubernetes.io/secret/847d367e-516e-4240-924c-3e62c532a8d6-tls-certs\") pod \"847d367e-516e-4240-924c-3e62c532a8d6\" (UID: \"847d367e-516e-4240-924c-3e62c532a8d6\") "
Oct 31 01:44:57.957915 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:57.957702    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/847d367e-516e-4240-924c-3e62c532a8d6-home\" (UniqueName: \"kubernetes.io/empty-dir/847d367e-516e-4240-924c-3e62c532a8d6-home\") pod \"847d367e-516e-4240-924c-3e62c532a8d6\" (UID: \"847d367e-516e-4240-924c-3e62c532a8d6\") "
Oct 31 01:44:57.957915 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:57.957731    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/847d367e-516e-4240-924c-3e62c532a8d6-kube-api-access-lzdht\" (UniqueName: \"kubernetes.io/projected/847d367e-516e-4240-924c-3e62c532a8d6-kube-api-access-lzdht\") pod \"847d367e-516e-4240-924c-3e62c532a8d6\" (UID: \"847d367e-516e-4240-924c-3e62c532a8d6\") "
Oct 31 01:44:57.957915 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:57.957905    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/847d367e-516e-4240-924c-3e62c532a8d6-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/847d367e-516e-4240-924c-3e62c532a8d6-model-cache\") pod \"847d367e-516e-4240-924c-3e62c532a8d6\" (UID: \"847d367e-516e-4240-924c-3e62c532a8d6\") "
Oct 31 01:44:57.958084 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:57.957910    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/847d367e-516e-4240-924c-3e62c532a8d6-tmp-dir" pod "847d367e-516e-4240-924c-3e62c532a8d6" (UID: "847d367e-516e-4240-924c-3e62c532a8d6"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:44:57.958084 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:57.958026    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/847d367e-516e-4240-924c-3e62c532a8d6-dshm\" (UniqueName: \"kubernetes.io/empty-dir/847d367e-516e-4240-924c-3e62c532a8d6-dshm\") pod \"847d367e-516e-4240-924c-3e62c532a8d6\" (UID: \"847d367e-516e-4240-924c-3e62c532a8d6\") "
Oct 31 01:44:57.958084 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:57.958030    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/847d367e-516e-4240-924c-3e62c532a8d6-home" pod "847d367e-516e-4240-924c-3e62c532a8d6" (UID: "847d367e-516e-4240-924c-3e62c532a8d6"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:44:57.958250 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:57.958108    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/847d367e-516e-4240-924c-3e62c532a8d6-model-cache" pod "847d367e-516e-4240-924c-3e62c532a8d6" (UID: "847d367e-516e-4240-924c-3e62c532a8d6"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:44:57.959120 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:57.959091    4964 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/847d367e-516e-4240-924c-3e62c532a8d6-tmp-dir\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:44:57.959120 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:57.959120    4964 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/847d367e-516e-4240-924c-3e62c532a8d6-home\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:44:57.959403 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:57.959135    4964 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/847d367e-516e-4240-924c-3e62c532a8d6-model-cache\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:44:57.960435 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:57.960415    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/847d367e-516e-4240-924c-3e62c532a8d6-kube-api-access-lzdht" pod "847d367e-516e-4240-924c-3e62c532a8d6" (UID: "847d367e-516e-4240-924c-3e62c532a8d6"). InnerVolumeSpecName "kube-api-access-lzdht". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:44:57.960533 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:57.960430    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/847d367e-516e-4240-924c-3e62c532a8d6-dshm" pod "847d367e-516e-4240-924c-3e62c532a8d6" (UID: "847d367e-516e-4240-924c-3e62c532a8d6"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:44:57.960591 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:57.960554    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/847d367e-516e-4240-924c-3e62c532a8d6-tls-certs" pod "847d367e-516e-4240-924c-3e62c532a8d6" (UID: "847d367e-516e-4240-924c-3e62c532a8d6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:44:58.060162 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:58.060133    4964 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/847d367e-516e-4240-924c-3e62c532a8d6-tls-certs\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:44:58.060162 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:58.060162    4964 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lzdht\" (UniqueName: \"kubernetes.io/projected/847d367e-516e-4240-924c-3e62c532a8d6-kube-api-access-lzdht\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:44:58.060371 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:58.060181    4964 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/847d367e-516e-4240-924c-3e62c532a8d6-dshm\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:44:58.279121 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:58.279090    4964 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w1-71bc98/tenantcfg-model-71bc98-kserve-7b7d7bc7bf-kqn2l"]
Oct 31 01:44:58.287382 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:58.287352    4964 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w1-71bc98/tenantcfg-model-71bc98-kserve-7b7d7bc7bf-kqn2l"]
Oct 31 01:44:58.311290 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:58.311259    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:44:59.777826 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:59.777790    4964 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="847d367e-516e-4240-924c-3e62c532a8d6" path="/var/lib/kubelet/pods/847d367e-516e-4240-924c-3e62c532a8d6/volumes"
Oct 31 01:44:59.888918 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:44:59.888885    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:45:00.185171 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:45:00.184962    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:45:00.444423 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:45:00.444297    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:45:03.158360 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:45:03.158326    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:45:03.339794 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:45:03.339755    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:45:03.513615 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:45:03.513580    4964 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-9d7b2c-b59bbf96c-7m86r"]
Oct 31 01:45:03.514574 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:45:03.514553    4964 cpu_manager.go:414] "containerMap: removing container" podUID="847d367e-516e-4240-924c-3e62c532a8d6" containerName="main"
Oct 31 01:45:03.514629 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:45:03.514577    4964 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="847d367e-516e-4240-924c-3e62c532a8d6" containerName="main"
Oct 31 01:45:03.514725 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:45:03.514714    4964 memory_manager.go:365] "RemoveStaleState removing state" podUID="847d367e-516e-4240-924c-3e62c532a8d6" containerName="main"
Oct 31 01:45:03.520409 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:45:03.520387    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-9d7b2c-b59bbf96c-7m86r"
Oct 31 01:45:03.520787 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:45:03.520749    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:45:03.572078 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:45:03.572046    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-9d7b2c-b59bbf96c-7m86r"]
Oct 31 01:45:03.637114 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:45:03.622641    4964 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-a-9d7b2c\""
Oct 31 01:45:03.656435 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:45:03.643105    4964 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-a-9d7b2c-dockercfg-fp6lp\""
Oct 31 01:45:03.678205 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:45:03.678173    4964 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-9d7b2c-7548d86fb9-djhv5"]
Oct 31 01:45:03.682609 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:45:03.682593    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-9d7b2c-7548d86fb9-djhv5"
Oct 31 01:45:03.725091 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:45:03.724510    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-9d7b2c-7548d86fb9-djhv5"]
Oct 31 01:45:03.739327 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:45:03.739269    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zpw4x\" (UniqueName: \"kubernetes.io/projected/655b500a-1f9c-424f-9bd2-376464e7151f-kube-api-access-zpw4x\") pod \"payload-pre-processing-e2e-praxis-a-9d7b2c-b59bbf96c-7m86r\" (UID: \"655b500a-1f9c-424f-9bd2-376464e7151f\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-9d7b2c-b59bbf96c-7m86r"
Oct 31 01:45:03.739481 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:45:03.739406    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/655b500a-1f9c-424f-9bd2-376464e7151f-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-9d7b2c-b59bbf96c-7m86r\" (UID: \"655b500a-1f9c-424f-9bd2-376464e7151f\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-9d7b2c-b59bbf96c-7m86r"
Oct 31 01:45:03.840838 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:45:03.840760    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/850c4489-7821-4e37-95f0-258336f5e75e-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-9d7b2c-7548d86fb9-djhv5\" (UID: \"850c4489-7821-4e37-95f0-258336f5e75e\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-9d7b2c-7548d86fb9-djhv5"
Oct 31 01:45:03.840838 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:45:03.840802    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zpw4x\" (UniqueName: \"kubernetes.io/projected/655b500a-1f9c-424f-9bd2-376464e7151f-kube-api-access-zpw4x\") pod \"payload-pre-processing-e2e-praxis-a-9d7b2c-b59bbf96c-7m86r\" (UID: \"655b500a-1f9c-424f-9bd2-376464e7151f\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-9d7b2c-b59bbf96c-7m86r"
Oct 31 01:45:03.841064 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:45:03.840903    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4bkfk\" (UniqueName: \"kubernetes.io/projected/850c4489-7821-4e37-95f0-258336f5e75e-kube-api-access-4bkfk\") pod \"payload-processing-e2e-praxis-a-9d7b2c-7548d86fb9-djhv5\" (UID: \"850c4489-7821-4e37-95f0-258336f5e75e\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-9d7b2c-7548d86fb9-djhv5"
Oct 31 01:45:03.841064 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:45:03.840958    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/655b500a-1f9c-424f-9bd2-376464e7151f-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-9d7b2c-b59bbf96c-7m86r\" (UID: \"655b500a-1f9c-424f-9bd2-376464e7151f\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-9d7b2c-b59bbf96c-7m86r"
Oct 31 01:45:03.841853 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:45:03.841829    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/655b500a-1f9c-424f-9bd2-376464e7151f-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-9d7b2c-b59bbf96c-7m86r\" (UID: \"655b500a-1f9c-424f-9bd2-376464e7151f\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-9d7b2c-b59bbf96c-7m86r"
Oct 31 01:45:03.898286 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:45:03.898253    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zpw4x\" (UniqueName: \"kubernetes.io/projected/655b500a-1f9c-424f-9bd2-376464e7151f-kube-api-access-zpw4x\") pod \"payload-pre-processing-e2e-praxis-a-9d7b2c-b59bbf96c-7m86r\" (UID: \"655b500a-1f9c-424f-9bd2-376464e7151f\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-9d7b2c-b59bbf96c-7m86r"
Oct 31 01:45:03.942691 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:45:03.942660    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/850c4489-7821-4e37-95f0-258336f5e75e-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-9d7b2c-7548d86fb9-djhv5\" (UID: \"850c4489-7821-4e37-95f0-258336f5e75e\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-9d7b2c-7548d86fb9-djhv5"
Oct 31 01:45:03.942869 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:45:03.942840    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4bkfk\" (UniqueName: \"kubernetes.io/projected/850c4489-7821-4e37-95f0-258336f5e75e-kube-api-access-4bkfk\") pod \"payload-processing-e2e-praxis-a-9d7b2c-7548d86fb9-djhv5\" (UID: \"850c4489-7821-4e37-95f0-258336f5e75e\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-9d7b2c-7548d86fb9-djhv5"
Oct 31 01:45:03.943332 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:45:03.943298    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/850c4489-7821-4e37-95f0-258336f5e75e-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-9d7b2c-7548d86fb9-djhv5\" (UID: \"850c4489-7821-4e37-95f0-258336f5e75e\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-9d7b2c-7548d86fb9-djhv5"
Oct 31 01:45:03.999287 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:45:03.999255    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4bkfk\" (UniqueName: \"kubernetes.io/projected/850c4489-7821-4e37-95f0-258336f5e75e-kube-api-access-4bkfk\") pod \"payload-processing-e2e-praxis-a-9d7b2c-7548d86fb9-djhv5\" (UID: \"850c4489-7821-4e37-95f0-258336f5e75e\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-9d7b2c-7548d86fb9-djhv5"
Oct 31 01:45:04.135101 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:45:04.135019    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-9d7b2c-b59bbf96c-7m86r"
Oct 31 01:45:04.297161 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:45:04.297129    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-9d7b2c-7548d86fb9-djhv5"
Oct 31 01:45:04.518723 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:45:04.518698    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-9d7b2c-b59bbf96c-7m86r"]
Oct 31 01:45:04.519125 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:45:04.519102    4964 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod655b500a_1f9c_424f_9bd2_376464e7151f.slice/crio-a55ecab697687bbad1edea50bea78680f8c89512b42cf37b2f087b83cb4ea0a2 WatchSource:0}: Error finding container a55ecab697687bbad1edea50bea78680f8c89512b42cf37b2f087b83cb4ea0a2: Status 404 returned error can't find the container with id a55ecab697687bbad1edea50bea78680f8c89512b42cf37b2f087b83cb4ea0a2
Oct 31 01:45:04.681172 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:45:04.681148    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-9d7b2c-7548d86fb9-djhv5"]
Oct 31 01:45:04.682470 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:45:04.682442    4964 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod850c4489_7821_4e37_95f0_258336f5e75e.slice/crio-7550cc0111e5c2324137a9ffb827c5ce4ef5238bc48a6a183418506a6df2897e WatchSource:0}: Error finding container 7550cc0111e5c2324137a9ffb827c5ce4ef5238bc48a6a183418506a6df2897e: Status 404 returned error can't find the container with id 7550cc0111e5c2324137a9ffb827c5ce4ef5238bc48a6a183418506a6df2897e
Oct 31 01:45:04.833223 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:45:04.833139    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-c0daf4-fb8556c8c-k4ggk"
Oct 31 01:45:04.917310 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:45:04.917249    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-9d7b2c-7548d86fb9-djhv5" event={"ID":"850c4489-7821-4e37-95f0-258336f5e75e","Type":"ContainerStarted","Data":"87c35bdf46a217bbc1ffef1dea2e93e80da9e1f03f337948024539f3a9871c9f"}
Oct 31 01:45:04.917493 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:45:04.917374    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-9d7b2c-7548d86fb9-djhv5" event={"ID":"850c4489-7821-4e37-95f0-258336f5e75e","Type":"ContainerStarted","Data":"7550cc0111e5c2324137a9ffb827c5ce4ef5238bc48a6a183418506a6df2897e"}
Oct 31 01:45:04.917493 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:45:04.917409    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-9d7b2c-7548d86fb9-djhv5"
Oct 31 01:45:04.919234 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:45:04.919206    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-9d7b2c-b59bbf96c-7m86r" event={"ID":"655b500a-1f9c-424f-9bd2-376464e7151f","Type":"ContainerStarted","Data":"8937701057d857935308ef0e65ce7d3c66023a942e4535c3ae7a8805f71d5ce9"}
Oct 31 01:45:04.919426 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:45:04.919259    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-9d7b2c-b59bbf96c-7m86r"
Oct 31 01:45:04.919426 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:45:04.919285    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-9d7b2c-b59bbf96c-7m86r" event={"ID":"655b500a-1f9c-424f-9bd2-376464e7151f","Type":"ContainerStarted","Data":"a55ecab697687bbad1edea50bea78680f8c89512b42cf37b2f087b83cb4ea0a2"}
Oct 31 01:45:05.061902 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:45:05.061849    4964 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-9d7b2c-b59bbf96c-7m86r" podStartSLOduration=2.061835991 podStartE2EDuration="2.061835991s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:45:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:45:05.0588325 +0000 UTC m=+2473.535398881" watchObservedRunningTime="2026-10-31 01:45:05.061835991 +0000 UTC m=+2473.538402371"
Oct 31 01:45:05.176618 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:45:05.173349    4964 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-a-9d7b2c-7548d86fb9-djhv5" podStartSLOduration=2.1733096339999998 podStartE2EDuration="2.173309634s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:45:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:45:05.171921807 +0000 UTC m=+2473.648488190" watchObservedRunningTime="2026-10-31 01:45:05.173309634 +0000 UTC m=+2473.649876016"
Oct 31 01:45:08.338621 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:45:08.338583    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:45:08.521808 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:45:08.521773    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:45:08.702500 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:45:08.702355    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:45:11.241098 ip-10-0-133-197 kubenswrapper[4964]: E1031 01:45:11.241061    4964 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q56x5" containerName="extractor"
Oct 31 01:45:12.406805 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:45:12.406771    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:45:13.446577 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:45:13.446544    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:45:14.725735 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:45:14.725696    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:45:15.933444 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:45:15.933419    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-9d7b2c-7548d86fb9-djhv5"
Oct 31 01:45:15.933864 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:45:15.933481    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-9d7b2c-b59bbf96c-7m86r"
Oct 31 01:45:19.835899 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:45:19.835862    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:45:20.015806 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:45:20.015754    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:45:20.193471 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:45:20.193382    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:45:24.833013 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:45:24.832963    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:45:27.428468 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:45:27.428430    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:45:27.682696 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:45:27.682611    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:45:29.609337 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:45:29.608959    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:45:34.363266 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:45:34.363211    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:45:37.186311 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:45:37.186277    4964 generic.go:356] "Generic (PLEG): container finished" podID="11d36add-45f3-4bf4-9372-7d7cec791eff" containerID="c91c1fe8e115676b727f5825d66641c22ba6c0d04d6c035d7d8bcbafb620504e" exitCode=0
Oct 31 01:45:37.186845 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:45:37.186379    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-8a3796-openshift-default-857c657d9-8vlxd" event={"ID":"11d36add-45f3-4bf4-9372-7d7cec791eff","Type":"ContainerDied","Data":"c91c1fe8e115676b727f5825d66641c22ba6c0d04d6c035d7d8bcbafb620504e"}
Oct 31 01:45:37.186914 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:45:37.186850    4964 scope.go:122] "RemoveContainer" containerID="c91c1fe8e115676b727f5825d66641c22ba6c0d04d6c035d7d8bcbafb620504e"
Oct 31 01:45:37.188498 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:45:37.188469    4964 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 01:45:37.188596 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:45:37.188536    4964 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 01:45:37.188596 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:45:37.188591    4964 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 01:45:37.451256 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:45:37.451178    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-8a3796-openshift-default-857c657d9-8vlxd"
Oct 31 01:45:38.220225 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:45:38.220183    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-8a3796-openshift-default-857c657d9-8vlxd" event={"ID":"11d36add-45f3-4bf4-9372-7d7cec791eff","Type":"ContainerStarted","Data":"7a4a5bfb6732af6a7fb4dd00f1f8e2a9b9b59611cfebc5aabfe6feb8442880bd"}
Oct 31 01:45:38.315398 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:45:38.304032    4964 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-8a3796-openshift-default-857c657d9-8vlxd"
Oct 31 01:45:38.315620 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:45:38.315491    4964 patch_prober.go:29] interesting pod/e2e-praxis-a-8a3796-openshift-default-857c657d9-8vlxd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused" start-of-body=
Oct 31 01:45:38.315620 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:45:38.315540    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-8a3796-openshift-default-857c657d9-8vlxd" podUID="11d36add-45f3-4bf4-9372-7d7cec791eff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused"
Oct 31 01:45:39.118700 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:45:39.118647    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:45:39.306343 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:45:39.294632    4964 patch_prober.go:29] interesting pod/e2e-praxis-a-8a3796-openshift-default-857c657d9-8vlxd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused" start-of-body=
Oct 31 01:45:39.306343 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:45:39.294685    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-8a3796-openshift-default-857c657d9-8vlxd" podUID="11d36add-45f3-4bf4-9372-7d7cec791eff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused"
Oct 31 01:45:40.301866 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:45:40.301835    4964 patch_prober.go:29] interesting pod/e2e-praxis-a-8a3796-openshift-default-857c657d9-8vlxd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused" start-of-body=
Oct 31 01:45:40.302150 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:45:40.302121    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-8a3796-openshift-default-857c657d9-8vlxd" podUID="11d36add-45f3-4bf4-9372-7d7cec791eff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused"
Oct 31 01:45:41.245859 ip-10-0-133-197 kubenswrapper[4964]: E1031 01:45:41.245822    4964 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q56x5" containerName="extractor"
Oct 31 01:45:41.307340 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:45:41.306499    4964 patch_prober.go:29] interesting pod/e2e-praxis-a-8a3796-openshift-default-857c657d9-8vlxd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused" start-of-body=
Oct 31 01:45:41.307340 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:45:41.306557    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-8a3796-openshift-default-857c657d9-8vlxd" podUID="11d36add-45f3-4bf4-9372-7d7cec791eff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused"
Oct 31 01:45:42.309348 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:45:42.299670    4964 patch_prober.go:29] interesting pod/e2e-praxis-a-8a3796-openshift-default-857c657d9-8vlxd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused" start-of-body=
Oct 31 01:45:42.309348 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:45:42.299729    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-8a3796-openshift-default-857c657d9-8vlxd" podUID="11d36add-45f3-4bf4-9372-7d7cec791eff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused"
Oct 31 01:45:43.297690 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:45:43.297661    4964 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-8a3796-openshift-default-857c657d9-8vlxd"
Oct 31 01:45:43.298041 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:45:43.298022    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-8a3796-openshift-default-857c657d9-8vlxd"
Oct 31 01:45:43.298742 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:45:43.298721    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-8a3796-openshift-default-857c657d9-8vlxd"
Oct 31 01:45:43.916415 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:45:43.916209    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:45:48.891398 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:45:48.891359    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:45:54.998888 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:45:54.998856    4964 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-c0daf4-7f578c867c-84zhj"]
Oct 31 01:45:55.004467 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:45:55.004438    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-c0daf4-7f578c867c-84zhj"
Oct 31 01:45:55.060284 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:45:55.060257    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-c0daf4-7f578c867c-84zhj"]
Oct 31 01:45:55.069657 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:45:55.069625    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/15340841-26cc-4a0e-adbe-3bbba9e6bb4f-metrics-certs\") pod \"maas-api-e2e-praxis-a-c0daf4-7f578c867c-84zhj\" (UID: \"15340841-26cc-4a0e-adbe-3bbba9e6bb4f\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-c0daf4-7f578c867c-84zhj"
Oct 31 01:45:55.069792 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:45:55.069672    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qdlrk\" (UniqueName: \"kubernetes.io/projected/15340841-26cc-4a0e-adbe-3bbba9e6bb4f-kube-api-access-qdlrk\") pod \"maas-api-e2e-praxis-a-c0daf4-7f578c867c-84zhj\" (UID: \"15340841-26cc-4a0e-adbe-3bbba9e6bb4f\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-c0daf4-7f578c867c-84zhj"
Oct 31 01:45:55.069842 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:45:55.069809    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/15340841-26cc-4a0e-adbe-3bbba9e6bb4f-maas-api-tls\") pod \"maas-api-e2e-praxis-a-c0daf4-7f578c867c-84zhj\" (UID: \"15340841-26cc-4a0e-adbe-3bbba9e6bb4f\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-c0daf4-7f578c867c-84zhj"
Oct 31 01:45:55.069901 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:45:55.069853    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/15340841-26cc-4a0e-adbe-3bbba9e6bb4f-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-c0daf4-7f578c867c-84zhj\" (UID: \"15340841-26cc-4a0e-adbe-3bbba9e6bb4f\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-c0daf4-7f578c867c-84zhj"
Oct 31 01:45:55.105195 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:45:55.105164    4964 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-a-c0daf4-serving-cert\""
Oct 31 01:45:55.171592 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:45:55.171559    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/15340841-26cc-4a0e-adbe-3bbba9e6bb4f-metrics-certs\") pod \"maas-api-e2e-praxis-a-c0daf4-7f578c867c-84zhj\" (UID: \"15340841-26cc-4a0e-adbe-3bbba9e6bb4f\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-c0daf4-7f578c867c-84zhj"
Oct 31 01:45:55.171592 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:45:55.171590    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qdlrk\" (UniqueName: \"kubernetes.io/projected/15340841-26cc-4a0e-adbe-3bbba9e6bb4f-kube-api-access-qdlrk\") pod \"maas-api-e2e-praxis-a-c0daf4-7f578c867c-84zhj\" (UID: \"15340841-26cc-4a0e-adbe-3bbba9e6bb4f\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-c0daf4-7f578c867c-84zhj"
Oct 31 01:45:55.171834 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:45:55.171656    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/15340841-26cc-4a0e-adbe-3bbba9e6bb4f-maas-api-tls\") pod \"maas-api-e2e-praxis-a-c0daf4-7f578c867c-84zhj\" (UID: \"15340841-26cc-4a0e-adbe-3bbba9e6bb4f\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-c0daf4-7f578c867c-84zhj"
Oct 31 01:45:55.171834 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:45:55.171694    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/15340841-26cc-4a0e-adbe-3bbba9e6bb4f-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-c0daf4-7f578c867c-84zhj\" (UID: \"15340841-26cc-4a0e-adbe-3bbba9e6bb4f\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-c0daf4-7f578c867c-84zhj"
Oct 31 01:45:55.171834 ip-10-0-133-197 kubenswrapper[4964]: E1031 01:45:55.171763    4964 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-praxis-a-c0daf4-serving-cert: secret "maas-api-e2e-praxis-a-c0daf4-serving-cert" not found
Oct 31 01:45:55.171992 ip-10-0-133-197 kubenswrapper[4964]: E1031 01:45:55.171837    4964 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/15340841-26cc-4a0e-adbe-3bbba9e6bb4f-maas-api-tls podName:15340841-26cc-4a0e-adbe-3bbba9e6bb4f nodeName:}" failed. No retries permitted until 2026-10-31 01:45:55.67182045 +0000 UTC m=+2524.148386813 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/15340841-26cc-4a0e-adbe-3bbba9e6bb4f-maas-api-tls") pod "maas-api-e2e-praxis-a-c0daf4-7f578c867c-84zhj" (UID: "15340841-26cc-4a0e-adbe-3bbba9e6bb4f") : secret "maas-api-e2e-praxis-a-c0daf4-serving-cert" not found
Oct 31 01:45:55.172425 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:45:55.172405    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/15340841-26cc-4a0e-adbe-3bbba9e6bb4f-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-c0daf4-7f578c867c-84zhj\" (UID: \"15340841-26cc-4a0e-adbe-3bbba9e6bb4f\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-c0daf4-7f578c867c-84zhj"
Oct 31 01:45:55.174646 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:45:55.174626    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/15340841-26cc-4a0e-adbe-3bbba9e6bb4f-metrics-certs\") pod \"maas-api-e2e-praxis-a-c0daf4-7f578c867c-84zhj\" (UID: \"15340841-26cc-4a0e-adbe-3bbba9e6bb4f\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-c0daf4-7f578c867c-84zhj"
Oct 31 01:45:55.230302 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:45:55.230279    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qdlrk\" (UniqueName: \"kubernetes.io/projected/15340841-26cc-4a0e-adbe-3bbba9e6bb4f-kube-api-access-qdlrk\") pod \"maas-api-e2e-praxis-a-c0daf4-7f578c867c-84zhj\" (UID: \"15340841-26cc-4a0e-adbe-3bbba9e6bb4f\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-c0daf4-7f578c867c-84zhj"
Oct 31 01:45:55.259613 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:45:55.259524    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:45:55.484540 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:45:55.484493    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:45:55.658666 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:45:55.658581    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:45:55.679071 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:45:55.679044    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/15340841-26cc-4a0e-adbe-3bbba9e6bb4f-maas-api-tls\") pod \"maas-api-e2e-praxis-a-c0daf4-7f578c867c-84zhj\" (UID: \"15340841-26cc-4a0e-adbe-3bbba9e6bb4f\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-c0daf4-7f578c867c-84zhj"
Oct 31 01:45:55.681574 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:45:55.681549    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/15340841-26cc-4a0e-adbe-3bbba9e6bb4f-maas-api-tls\") pod \"maas-api-e2e-praxis-a-c0daf4-7f578c867c-84zhj\" (UID: \"15340841-26cc-4a0e-adbe-3bbba9e6bb4f\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-c0daf4-7f578c867c-84zhj"
Oct 31 01:45:55.918804 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:45:55.918722    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-c0daf4-7f578c867c-84zhj"
Oct 31 01:45:56.302332 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:45:56.302291    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-c0daf4-7f578c867c-84zhj"]
Oct 31 01:45:56.303866 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:45:56.303827    4964 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod15340841_26cc_4a0e_adbe_3bbba9e6bb4f.slice/crio-c6c39d711b723bcd62d2f8180a38b3ebc2537a19f372b87881791b9d58bec451 WatchSource:0}: Error finding container c6c39d711b723bcd62d2f8180a38b3ebc2537a19f372b87881791b9d58bec451: Status 404 returned error can't find the container with id c6c39d711b723bcd62d2f8180a38b3ebc2537a19f372b87881791b9d58bec451
Oct 31 01:45:56.305303 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:45:56.305284    4964 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 31 01:45:56.348632 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:45:56.348603    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-c0daf4-7f578c867c-84zhj" event={"ID":"15340841-26cc-4a0e-adbe-3bbba9e6bb4f","Type":"ContainerStarted","Data":"c6c39d711b723bcd62d2f8180a38b3ebc2537a19f372b87881791b9d58bec451"}
Oct 31 01:45:57.356657 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:45:57.356627    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-c0daf4-7f578c867c-84zhj" event={"ID":"15340841-26cc-4a0e-adbe-3bbba9e6bb4f","Type":"ContainerStarted","Data":"27a8ce86be642c14cf81ad86dce60ebeb0d7a5a5b8c97c57d1792759ed9e5ac4"}
Oct 31 01:45:57.357156 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:45:57.356690    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-c0daf4-7f578c867c-84zhj"
Oct 31 01:45:57.471552 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:45:57.471494    4964 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-c0daf4-7f578c867c-84zhj" podStartSLOduration=2.66494108 podStartE2EDuration="3.471474124s" totalImagesPullingTime="806.533044ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:45:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:45:57.470616501 +0000 UTC m=+2525.947182882" watchObservedRunningTime="2026-10-31 01:45:57.471474124 +0000 UTC m=+2525.948040505"
Oct 31 01:46:00.176730 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:46:00.176691    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:46:02.536664 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:46:02.536623    4964 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-wgfd6"]
Oct 31 01:46:02.541036 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:46:02.541018    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-wgfd6"
Oct 31 01:46:02.592787 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:46:02.592763    4964 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-a-c116f3-openshift-default-dockercfg-289mh\""
Oct 31 01:46:02.646355 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:46:02.646281    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-wgfd6"]
Oct 31 01:46:02.672154 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:46:02.672106    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7e48bcec-5ffa-4f2b-bb9f-f2fbdc1b3da9-istio-podinfo\") pod \"e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-wgfd6\" (UID: \"7e48bcec-5ffa-4f2b-bb9f-f2fbdc1b3da9\") " pod="openshift-ingress/e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-wgfd6"
Oct 31 01:46:02.672345 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:46:02.672162    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7e48bcec-5ffa-4f2b-bb9f-f2fbdc1b3da9-workload-certs\") pod \"e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-wgfd6\" (UID: \"7e48bcec-5ffa-4f2b-bb9f-f2fbdc1b3da9\") " pod="openshift-ingress/e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-wgfd6"
Oct 31 01:46:02.672345 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:46:02.672239    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7e48bcec-5ffa-4f2b-bb9f-f2fbdc1b3da9-istiod-ca-cert\") pod \"e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-wgfd6\" (UID: \"7e48bcec-5ffa-4f2b-bb9f-f2fbdc1b3da9\") " pod="openshift-ingress/e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-wgfd6"
Oct 31 01:46:02.672474 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:46:02.672404    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7e48bcec-5ffa-4f2b-bb9f-f2fbdc1b3da9-istio-token\") pod \"e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-wgfd6\" (UID: \"7e48bcec-5ffa-4f2b-bb9f-f2fbdc1b3da9\") " pod="openshift-ingress/e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-wgfd6"
Oct 31 01:46:02.672521 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:46:02.672478    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7e48bcec-5ffa-4f2b-bb9f-f2fbdc1b3da9-istio-data\") pod \"e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-wgfd6\" (UID: \"7e48bcec-5ffa-4f2b-bb9f-f2fbdc1b3da9\") " pod="openshift-ingress/e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-wgfd6"
Oct 31 01:46:02.672575 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:46:02.672560    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e48bcec-5ffa-4f2b-bb9f-f2fbdc1b3da9-credential-socket\") pod \"e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-wgfd6\" (UID: \"7e48bcec-5ffa-4f2b-bb9f-f2fbdc1b3da9\") " pod="openshift-ingress/e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-wgfd6"
Oct 31 01:46:02.672614 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:46:02.672599    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7e48bcec-5ffa-4f2b-bb9f-f2fbdc1b3da9-istio-envoy\") pod \"e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-wgfd6\" (UID: \"7e48bcec-5ffa-4f2b-bb9f-f2fbdc1b3da9\") " pod="openshift-ingress/e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-wgfd6"
Oct 31 01:46:02.672672 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:46:02.672633    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ss4j6\" (UniqueName: \"kubernetes.io/projected/7e48bcec-5ffa-4f2b-bb9f-f2fbdc1b3da9-kube-api-access-ss4j6\") pod \"e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-wgfd6\" (UID: \"7e48bcec-5ffa-4f2b-bb9f-f2fbdc1b3da9\") " pod="openshift-ingress/e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-wgfd6"
Oct 31 01:46:02.672750 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:46:02.672731    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e48bcec-5ffa-4f2b-bb9f-f2fbdc1b3da9-workload-socket\") pod \"e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-wgfd6\" (UID: \"7e48bcec-5ffa-4f2b-bb9f-f2fbdc1b3da9\") " pod="openshift-ingress/e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-wgfd6"
Oct 31 01:46:02.773749 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:46:02.773699    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e48bcec-5ffa-4f2b-bb9f-f2fbdc1b3da9-credential-socket\") pod \"e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-wgfd6\" (UID: \"7e48bcec-5ffa-4f2b-bb9f-f2fbdc1b3da9\") " pod="openshift-ingress/e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-wgfd6"
Oct 31 01:46:02.773940 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:46:02.773763    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7e48bcec-5ffa-4f2b-bb9f-f2fbdc1b3da9-istio-envoy\") pod \"e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-wgfd6\" (UID: \"7e48bcec-5ffa-4f2b-bb9f-f2fbdc1b3da9\") " pod="openshift-ingress/e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-wgfd6"
Oct 31 01:46:02.773940 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:46:02.773812    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ss4j6\" (UniqueName: \"kubernetes.io/projected/7e48bcec-5ffa-4f2b-bb9f-f2fbdc1b3da9-kube-api-access-ss4j6\") pod \"e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-wgfd6\" (UID: \"7e48bcec-5ffa-4f2b-bb9f-f2fbdc1b3da9\") " pod="openshift-ingress/e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-wgfd6"
Oct 31 01:46:02.773940 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:46:02.773852    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e48bcec-5ffa-4f2b-bb9f-f2fbdc1b3da9-workload-socket\") pod \"e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-wgfd6\" (UID: \"7e48bcec-5ffa-4f2b-bb9f-f2fbdc1b3da9\") " pod="openshift-ingress/e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-wgfd6"
Oct 31 01:46:02.773940 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:46:02.773914    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7e48bcec-5ffa-4f2b-bb9f-f2fbdc1b3da9-istio-podinfo\") pod \"e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-wgfd6\" (UID: \"7e48bcec-5ffa-4f2b-bb9f-f2fbdc1b3da9\") " pod="openshift-ingress/e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-wgfd6"
Oct 31 01:46:02.773940 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:46:02.773939    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7e48bcec-5ffa-4f2b-bb9f-f2fbdc1b3da9-workload-certs\") pod \"e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-wgfd6\" (UID: \"7e48bcec-5ffa-4f2b-bb9f-f2fbdc1b3da9\") " pod="openshift-ingress/e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-wgfd6"
Oct 31 01:46:02.774217 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:46:02.773965    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7e48bcec-5ffa-4f2b-bb9f-f2fbdc1b3da9-istiod-ca-cert\") pod \"e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-wgfd6\" (UID: \"7e48bcec-5ffa-4f2b-bb9f-f2fbdc1b3da9\") " pod="openshift-ingress/e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-wgfd6"
Oct 31 01:46:02.774217 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:46:02.774037    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7e48bcec-5ffa-4f2b-bb9f-f2fbdc1b3da9-istio-token\") pod \"e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-wgfd6\" (UID: \"7e48bcec-5ffa-4f2b-bb9f-f2fbdc1b3da9\") " pod="openshift-ingress/e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-wgfd6"
Oct 31 01:46:02.774217 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:46:02.774097    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7e48bcec-5ffa-4f2b-bb9f-f2fbdc1b3da9-istio-data\") pod \"e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-wgfd6\" (UID: \"7e48bcec-5ffa-4f2b-bb9f-f2fbdc1b3da9\") " pod="openshift-ingress/e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-wgfd6"
Oct 31 01:46:02.774217 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:46:02.774188    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e48bcec-5ffa-4f2b-bb9f-f2fbdc1b3da9-credential-socket\") pod \"e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-wgfd6\" (UID: \"7e48bcec-5ffa-4f2b-bb9f-f2fbdc1b3da9\") " pod="openshift-ingress/e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-wgfd6"
Oct 31 01:46:02.774484 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:46:02.774300    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e48bcec-5ffa-4f2b-bb9f-f2fbdc1b3da9-workload-socket\") pod \"e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-wgfd6\" (UID: \"7e48bcec-5ffa-4f2b-bb9f-f2fbdc1b3da9\") " pod="openshift-ingress/e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-wgfd6"
Oct 31 01:46:02.774484 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:46:02.774387    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7e48bcec-5ffa-4f2b-bb9f-f2fbdc1b3da9-workload-certs\") pod \"e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-wgfd6\" (UID: \"7e48bcec-5ffa-4f2b-bb9f-f2fbdc1b3da9\") " pod="openshift-ingress/e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-wgfd6"
Oct 31 01:46:02.774601 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:46:02.774552    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7e48bcec-5ffa-4f2b-bb9f-f2fbdc1b3da9-istio-data\") pod \"e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-wgfd6\" (UID: \"7e48bcec-5ffa-4f2b-bb9f-f2fbdc1b3da9\") " pod="openshift-ingress/e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-wgfd6"
Oct 31 01:46:02.774832 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:46:02.774797    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7e48bcec-5ffa-4f2b-bb9f-f2fbdc1b3da9-istiod-ca-cert\") pod \"e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-wgfd6\" (UID: \"7e48bcec-5ffa-4f2b-bb9f-f2fbdc1b3da9\") " pod="openshift-ingress/e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-wgfd6"
Oct 31 01:46:02.776830 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:46:02.776804    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7e48bcec-5ffa-4f2b-bb9f-f2fbdc1b3da9-istio-envoy\") pod \"e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-wgfd6\" (UID: \"7e48bcec-5ffa-4f2b-bb9f-f2fbdc1b3da9\") " pod="openshift-ingress/e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-wgfd6"
Oct 31 01:46:02.777589 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:46:02.777566    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7e48bcec-5ffa-4f2b-bb9f-f2fbdc1b3da9-istio-podinfo\") pod \"e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-wgfd6\" (UID: \"7e48bcec-5ffa-4f2b-bb9f-f2fbdc1b3da9\") " pod="openshift-ingress/e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-wgfd6"
Oct 31 01:46:02.830045 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:46:02.829971    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7e48bcec-5ffa-4f2b-bb9f-f2fbdc1b3da9-istio-token\") pod \"e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-wgfd6\" (UID: \"7e48bcec-5ffa-4f2b-bb9f-f2fbdc1b3da9\") " pod="openshift-ingress/e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-wgfd6"
Oct 31 01:46:02.830282 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:46:02.830261    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ss4j6\" (UniqueName: \"kubernetes.io/projected/7e48bcec-5ffa-4f2b-bb9f-f2fbdc1b3da9-kube-api-access-ss4j6\") pod \"e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-wgfd6\" (UID: \"7e48bcec-5ffa-4f2b-bb9f-f2fbdc1b3da9\") " pod="openshift-ingress/e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-wgfd6"
Oct 31 01:46:02.855117 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:46:02.855086    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-wgfd6"
Oct 31 01:46:02.906233 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:46:02.906188    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:46:03.293920 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:46:03.293887    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-wgfd6"]
Oct 31 01:46:03.294778 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:46:03.294726    4964 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e48bcec_5ffa_4f2b_bb9f_f2fbdc1b3da9.slice/crio-5f700999b29241a845fd934736e4a3491717328555fe88feee7be736b54f69ea WatchSource:0}: Error finding container 5f700999b29241a845fd934736e4a3491717328555fe88feee7be736b54f69ea: Status 404 returned error can't find the container with id 5f700999b29241a845fd934736e4a3491717328555fe88feee7be736b54f69ea
Oct 31 01:46:03.297769 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:46:03.297733    4964 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 01:46:03.297867 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:46:03.297805    4964 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 01:46:03.297867 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:46:03.297860    4964 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 01:46:03.370055 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:46:03.370032    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-c0daf4-7f578c867c-84zhj"
Oct 31 01:46:03.402977 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:46:03.402919    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-wgfd6" event={"ID":"7e48bcec-5ffa-4f2b-bb9f-f2fbdc1b3da9","Type":"ContainerStarted","Data":"61fa9ded14547620fe08064b5a186d18d0caa1c5aedbea191d0941170c3257ce"}
Oct 31 01:46:03.402977 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:46:03.402985    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-wgfd6" event={"ID":"7e48bcec-5ffa-4f2b-bb9f-f2fbdc1b3da9","Type":"ContainerStarted","Data":"5f700999b29241a845fd934736e4a3491717328555fe88feee7be736b54f69ea"}
Oct 31 01:46:03.601949 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:46:03.601873    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:46:03.612071 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:46:03.612024    4964 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-wgfd6" podStartSLOduration=1.612011165 podStartE2EDuration="1.612011165s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:46:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:46:03.610262745 +0000 UTC m=+2532.086829126" watchObservedRunningTime="2026-10-31 01:46:03.612011165 +0000 UTC m=+2532.088577607"
Oct 31 01:46:03.857550 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:46:03.857449    4964 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-wgfd6"
Oct 31 01:46:04.857572 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:46:04.857544    4964 patch_prober.go:29] interesting pod/e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-wgfd6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused" start-of-body=
Oct 31 01:46:04.858031 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:46:04.857592    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-wgfd6" podUID="7e48bcec-5ffa-4f2b-bb9f-f2fbdc1b3da9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused"
Oct 31 01:46:04.858031 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:46:04.857940    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:46:05.036233 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:46:05.036176    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:46:05.216407 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:46:05.216276    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:46:05.855971 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:46:05.855935    4964 patch_prober.go:29] interesting pod/e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-wgfd6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused" start-of-body=
Oct 31 01:46:05.856126 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:46:05.856010    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-wgfd6" podUID="7e48bcec-5ffa-4f2b-bb9f-f2fbdc1b3da9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused"
Oct 31 01:46:06.855848 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:46:06.855815    4964 patch_prober.go:29] interesting pod/e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-wgfd6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused" start-of-body=
Oct 31 01:46:06.856256 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:46:06.855890    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-wgfd6" podUID="7e48bcec-5ffa-4f2b-bb9f-f2fbdc1b3da9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused"
Oct 31 01:46:07.856126 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:46:07.856092    4964 patch_prober.go:29] interesting pod/e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-wgfd6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused" start-of-body=
Oct 31 01:46:07.856559 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:46:07.856157    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-wgfd6" podUID="7e48bcec-5ffa-4f2b-bb9f-f2fbdc1b3da9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused"
Oct 31 01:46:08.855720 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:46:08.855683    4964 patch_prober.go:29] interesting pod/e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-wgfd6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused" start-of-body=
Oct 31 01:46:08.855909 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:46:08.855748    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-wgfd6" podUID="7e48bcec-5ffa-4f2b-bb9f-f2fbdc1b3da9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.149:15021/healthz/ready\": dial tcp 10.132.0.149:15021: connect: connection refused"
Oct 31 01:46:09.860030 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:46:09.859995    4964 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-wgfd6"
Oct 31 01:46:09.860536 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:46:09.860339    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-wgfd6"
Oct 31 01:46:09.861043 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:46:09.861024    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-wgfd6"
Oct 31 01:46:09.890443 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:46:09.890408    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:46:11.243048 ip-10-0-133-197 kubenswrapper[4964]: E1031 01:46:11.243015    4964 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q56x5" containerName="extractor"
Oct 31 01:46:14.279439 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:46:14.279382    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:46:14.675944 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:46:14.675858    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:46:19.468041 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:46:19.468005    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:46:24.418980 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:46:24.418937    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:46:25.813665 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:46:25.813623    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:46:27.512221 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:46:27.512183    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:46:28.757581 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:46:28.757545    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:46:29.368464 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:46:29.368427    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:46:29.589800 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:46:29.589769    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:46:31.418206 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:46:31.415174    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:46:32.768660 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:46:32.768626    4964 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-gg8ls"]
Oct 31 01:46:32.773284 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:46:32.773262    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-gg8ls"
Oct 31 01:46:32.880659 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:46:32.880625    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-gg8ls"]
Oct 31 01:46:32.880890 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:46:32.880630    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dd00f3ae-bb65-4eb4-aada-cce1fab3b363-workload-certs\") pod \"e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-gg8ls\" (UID: \"dd00f3ae-bb65-4eb4-aada-cce1fab3b363\") " pod="openshift-ingress/e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-gg8ls"
Oct 31 01:46:32.880890 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:46:32.880779    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd00f3ae-bb65-4eb4-aada-cce1fab3b363-credential-socket\") pod \"e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-gg8ls\" (UID: \"dd00f3ae-bb65-4eb4-aada-cce1fab3b363\") " pod="openshift-ingress/e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-gg8ls"
Oct 31 01:46:32.880890 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:46:32.880851    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8rl2w\" (UniqueName: \"kubernetes.io/projected/dd00f3ae-bb65-4eb4-aada-cce1fab3b363-kube-api-access-8rl2w\") pod \"e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-gg8ls\" (UID: \"dd00f3ae-bb65-4eb4-aada-cce1fab3b363\") " pod="openshift-ingress/e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-gg8ls"
Oct 31 01:46:32.881082 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:46:32.880924    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dd00f3ae-bb65-4eb4-aada-cce1fab3b363-istiod-ca-cert\") pod \"e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-gg8ls\" (UID: \"dd00f3ae-bb65-4eb4-aada-cce1fab3b363\") " pod="openshift-ingress/e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-gg8ls"
Oct 31 01:46:32.881082 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:46:32.880954    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dd00f3ae-bb65-4eb4-aada-cce1fab3b363-istio-podinfo\") pod \"e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-gg8ls\" (UID: \"dd00f3ae-bb65-4eb4-aada-cce1fab3b363\") " pod="openshift-ingress/e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-gg8ls"
Oct 31 01:46:32.881082 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:46:32.881009    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dd00f3ae-bb65-4eb4-aada-cce1fab3b363-istio-token\") pod \"e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-gg8ls\" (UID: \"dd00f3ae-bb65-4eb4-aada-cce1fab3b363\") " pod="openshift-ingress/e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-gg8ls"
Oct 31 01:46:32.881082 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:46:32.881064    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dd00f3ae-bb65-4eb4-aada-cce1fab3b363-istio-data\") pod \"e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-gg8ls\" (UID: \"dd00f3ae-bb65-4eb4-aada-cce1fab3b363\") " pod="openshift-ingress/e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-gg8ls"
Oct 31 01:46:32.881309 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:46:32.881104    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dd00f3ae-bb65-4eb4-aada-cce1fab3b363-istio-envoy\") pod \"e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-gg8ls\" (UID: \"dd00f3ae-bb65-4eb4-aada-cce1fab3b363\") " pod="openshift-ingress/e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-gg8ls"
Oct 31 01:46:32.881309 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:46:32.881183    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd00f3ae-bb65-4eb4-aada-cce1fab3b363-workload-socket\") pod \"e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-gg8ls\" (UID: \"dd00f3ae-bb65-4eb4-aada-cce1fab3b363\") " pod="openshift-ingress/e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-gg8ls"
Oct 31 01:46:32.983431 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:46:32.983382    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8rl2w\" (UniqueName: \"kubernetes.io/projected/dd00f3ae-bb65-4eb4-aada-cce1fab3b363-kube-api-access-8rl2w\") pod \"e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-gg8ls\" (UID: \"dd00f3ae-bb65-4eb4-aada-cce1fab3b363\") " pod="openshift-ingress/e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-gg8ls"
Oct 31 01:46:32.983777 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:46:32.983756    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dd00f3ae-bb65-4eb4-aada-cce1fab3b363-istiod-ca-cert\") pod \"e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-gg8ls\" (UID: \"dd00f3ae-bb65-4eb4-aada-cce1fab3b363\") " pod="openshift-ingress/e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-gg8ls"
Oct 31 01:46:32.983935 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:46:32.983920    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dd00f3ae-bb65-4eb4-aada-cce1fab3b363-istio-podinfo\") pod \"e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-gg8ls\" (UID: \"dd00f3ae-bb65-4eb4-aada-cce1fab3b363\") " pod="openshift-ingress/e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-gg8ls"
Oct 31 01:46:32.984109 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:46:32.984081    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dd00f3ae-bb65-4eb4-aada-cce1fab3b363-istio-token\") pod \"e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-gg8ls\" (UID: \"dd00f3ae-bb65-4eb4-aada-cce1fab3b363\") " pod="openshift-ingress/e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-gg8ls"
Oct 31 01:46:32.984277 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:46:32.984250    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dd00f3ae-bb65-4eb4-aada-cce1fab3b363-istio-data\") pod \"e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-gg8ls\" (UID: \"dd00f3ae-bb65-4eb4-aada-cce1fab3b363\") " pod="openshift-ingress/e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-gg8ls"
Oct 31 01:46:32.984456 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:46:32.984432    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dd00f3ae-bb65-4eb4-aada-cce1fab3b363-istio-envoy\") pod \"e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-gg8ls\" (UID: \"dd00f3ae-bb65-4eb4-aada-cce1fab3b363\") " pod="openshift-ingress/e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-gg8ls"
Oct 31 01:46:32.984651 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:46:32.984636    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd00f3ae-bb65-4eb4-aada-cce1fab3b363-workload-socket\") pod \"e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-gg8ls\" (UID: \"dd00f3ae-bb65-4eb4-aada-cce1fab3b363\") " pod="openshift-ingress/e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-gg8ls"
Oct 31 01:46:32.984842 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:46:32.984827    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dd00f3ae-bb65-4eb4-aada-cce1fab3b363-workload-certs\") pod \"e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-gg8ls\" (UID: \"dd00f3ae-bb65-4eb4-aada-cce1fab3b363\") " pod="openshift-ingress/e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-gg8ls"
Oct 31 01:46:32.985023 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:46:32.985009    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd00f3ae-bb65-4eb4-aada-cce1fab3b363-credential-socket\") pod \"e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-gg8ls\" (UID: \"dd00f3ae-bb65-4eb4-aada-cce1fab3b363\") " pod="openshift-ingress/e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-gg8ls"
Oct 31 01:46:32.985764 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:46:32.985741    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd00f3ae-bb65-4eb4-aada-cce1fab3b363-credential-socket\") pod \"e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-gg8ls\" (UID: \"dd00f3ae-bb65-4eb4-aada-cce1fab3b363\") " pod="openshift-ingress/e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-gg8ls"
Oct 31 01:46:32.986929 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:46:32.986908    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dd00f3ae-bb65-4eb4-aada-cce1fab3b363-istiod-ca-cert\") pod \"e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-gg8ls\" (UID: \"dd00f3ae-bb65-4eb4-aada-cce1fab3b363\") " pod="openshift-ingress/e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-gg8ls"
Oct 31 01:46:32.990300 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:46:32.990271    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dd00f3ae-bb65-4eb4-aada-cce1fab3b363-istio-podinfo\") pod \"e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-gg8ls\" (UID: \"dd00f3ae-bb65-4eb4-aada-cce1fab3b363\") " pod="openshift-ingress/e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-gg8ls"
Oct 31 01:46:32.990632 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:46:32.990607    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd00f3ae-bb65-4eb4-aada-cce1fab3b363-workload-socket\") pod \"e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-gg8ls\" (UID: \"dd00f3ae-bb65-4eb4-aada-cce1fab3b363\") " pod="openshift-ingress/e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-gg8ls"
Oct 31 01:46:32.990834 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:46:32.990810    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dd00f3ae-bb65-4eb4-aada-cce1fab3b363-workload-certs\") pod \"e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-gg8ls\" (UID: \"dd00f3ae-bb65-4eb4-aada-cce1fab3b363\") " pod="openshift-ingress/e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-gg8ls"
Oct 31 01:46:32.991174 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:46:32.991146    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dd00f3ae-bb65-4eb4-aada-cce1fab3b363-istio-data\") pod \"e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-gg8ls\" (UID: \"dd00f3ae-bb65-4eb4-aada-cce1fab3b363\") " pod="openshift-ingress/e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-gg8ls"
Oct 31 01:46:33.003732 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:46:33.003710    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dd00f3ae-bb65-4eb4-aada-cce1fab3b363-istio-envoy\") pod \"e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-gg8ls\" (UID: \"dd00f3ae-bb65-4eb4-aada-cce1fab3b363\") " pod="openshift-ingress/e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-gg8ls"
Oct 31 01:46:33.042273 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:46:33.042208    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8rl2w\" (UniqueName: \"kubernetes.io/projected/dd00f3ae-bb65-4eb4-aada-cce1fab3b363-kube-api-access-8rl2w\") pod \"e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-gg8ls\" (UID: \"dd00f3ae-bb65-4eb4-aada-cce1fab3b363\") " pod="openshift-ingress/e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-gg8ls"
Oct 31 01:46:33.046290 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:46:33.046261    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dd00f3ae-bb65-4eb4-aada-cce1fab3b363-istio-token\") pod \"e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-gg8ls\" (UID: \"dd00f3ae-bb65-4eb4-aada-cce1fab3b363\") " pod="openshift-ingress/e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-gg8ls"
Oct 31 01:46:33.088376 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:46:33.088352    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-gg8ls"
Oct 31 01:46:33.525969 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:46:33.525941    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-gg8ls"]
Oct 31 01:46:33.526942 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:46:33.526909    4964 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd00f3ae_bb65_4eb4_aada_cce1fab3b363.slice/crio-f19d2779eb7084a6653a3ebe2dffb7da61b6b821ee22a78451ff5eac51327b14 WatchSource:0}: Error finding container f19d2779eb7084a6653a3ebe2dffb7da61b6b821ee22a78451ff5eac51327b14: Status 404 returned error can't find the container with id f19d2779eb7084a6653a3ebe2dffb7da61b6b821ee22a78451ff5eac51327b14
Oct 31 01:46:33.529404 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:46:33.529371    4964 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 01:46:33.529470 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:46:33.529436    4964 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 01:46:33.529470 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:46:33.529466    4964 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 01:46:33.655826 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:46:33.655793    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-gg8ls" event={"ID":"dd00f3ae-bb65-4eb4-aada-cce1fab3b363","Type":"ContainerStarted","Data":"95128752e0c27828e733b6cee2800707f3397b21f024cb5d7899248c2043fea4"}
Oct 31 01:46:33.656005 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:46:33.655847    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-gg8ls" event={"ID":"dd00f3ae-bb65-4eb4-aada-cce1fab3b363","Type":"ContainerStarted","Data":"f19d2779eb7084a6653a3ebe2dffb7da61b6b821ee22a78451ff5eac51327b14"}
Oct 31 01:46:33.712538 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:46:33.712509    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:46:33.777242 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:46:33.777127    4964 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-gg8ls" podStartSLOduration=1.777111628 podStartE2EDuration="1.777111628s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:46:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:46:33.775383098 +0000 UTC m=+2562.251949488" watchObservedRunningTime="2026-10-31 01:46:33.777111628 +0000 UTC m=+2562.253678007"
Oct 31 01:46:33.939127 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:46:33.939081    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:46:34.089167 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:46:34.089089    4964 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-gg8ls"
Oct 31 01:46:34.091104 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:46:34.091076    4964 patch_prober.go:29] interesting pod/e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-gg8ls container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.150:15021/healthz/ready\": dial tcp 10.132.0.150:15021: connect: connection refused" start-of-body=
Oct 31 01:46:34.091336 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:46:34.091291    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-gg8ls" podUID="dd00f3ae-bb65-4eb4-aada-cce1fab3b363" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.150:15021/healthz/ready\": dial tcp 10.132.0.150:15021: connect: connection refused"
Oct 31 01:46:34.130668 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:46:34.130628    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:46:35.089460 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:46:35.089429    4964 patch_prober.go:29] interesting pod/e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-gg8ls container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.150:15021/healthz/ready\": dial tcp 10.132.0.150:15021: connect: connection refused" start-of-body=
Oct 31 01:46:35.089884 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:46:35.089483    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-gg8ls" podUID="dd00f3ae-bb65-4eb4-aada-cce1fab3b363" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.150:15021/healthz/ready\": dial tcp 10.132.0.150:15021: connect: connection refused"
Oct 31 01:46:36.088951 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:46:36.088900    4964 patch_prober.go:29] interesting pod/e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-gg8ls container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.150:15021/healthz/ready\": dial tcp 10.132.0.150:15021: connect: connection refused" start-of-body=
Oct 31 01:46:36.089174 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:46:36.088965    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-gg8ls" podUID="dd00f3ae-bb65-4eb4-aada-cce1fab3b363" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.150:15021/healthz/ready\": dial tcp 10.132.0.150:15021: connect: connection refused"
Oct 31 01:46:36.761470 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:46:36.761428    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:46:37.021398 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:46:37.014894    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:46:37.100345 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:46:37.097671    4964 patch_prober.go:29] interesting pod/e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-gg8ls container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.150:15021/healthz/ready\": dial tcp 10.132.0.150:15021: connect: connection refused" start-of-body=
Oct 31 01:46:37.100345 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:46:37.097731    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-gg8ls" podUID="dd00f3ae-bb65-4eb4-aada-cce1fab3b363" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.150:15021/healthz/ready\": dial tcp 10.132.0.150:15021: connect: connection refused"
Oct 31 01:46:37.216704 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:46:37.216672    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:46:38.088834 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:46:38.088797    4964 patch_prober.go:29] interesting pod/e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-gg8ls container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.150:15021/healthz/ready\": dial tcp 10.132.0.150:15021: connect: connection refused" start-of-body=
Oct 31 01:46:38.089343 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:46:38.088857    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-gg8ls" podUID="dd00f3ae-bb65-4eb4-aada-cce1fab3b363" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.150:15021/healthz/ready\": dial tcp 10.132.0.150:15021: connect: connection refused"
Oct 31 01:46:38.548075 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:46:38.548033    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:46:39.088659 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:46:39.088628    4964 patch_prober.go:29] interesting pod/e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-gg8ls container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.150:15021/healthz/ready\": dial tcp 10.132.0.150:15021: connect: connection refused" start-of-body=
Oct 31 01:46:39.088839 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:46:39.088689    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-gg8ls" podUID="dd00f3ae-bb65-4eb4-aada-cce1fab3b363" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.150:15021/healthz/ready\": dial tcp 10.132.0.150:15021: connect: connection refused"
Oct 31 01:46:39.699239 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:46:39.699200    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:46:40.095043 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:46:40.095012    4964 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-gg8ls"
Oct 31 01:46:40.095458 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:46:40.095434    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-gg8ls"
Oct 31 01:46:40.096140 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:46:40.096118    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-gg8ls"
Oct 31 01:46:40.318101 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:46:40.318058    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:46:40.606782 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:46:40.606735    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:46:41.237582 ip-10-0-133-197 kubenswrapper[4964]: E1031 01:46:41.237548    4964 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q56x5" containerName="extractor"
Oct 31 01:46:42.036649 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:46:42.036603    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:46:44.724075 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:46:44.724041    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:46:44.950873 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:46:44.950844    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:46:47.256781 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:46:47.256744    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:46:51.254481 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:46:51.254421    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:46:51.438557 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:46:51.438512    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:46:51.655429 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:46:51.655346    4964 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-99145d-7d5667799-hfvwn"]
Oct 31 01:46:51.670757 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:46:51.670730    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-99145d-7d5667799-hfvwn"
Oct 31 01:46:51.702625 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:46:51.702596    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-99145d-7d5667799-hfvwn"]
Oct 31 01:46:51.724169 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:46:51.724142    4964 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-b-99145d-dockercfg-jdwp6\""
Oct 31 01:46:51.724522 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:46:51.724506    4964 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-b-99145d\""
Oct 31 01:46:51.803956 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:46:51.802161    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7ca5c214-0290-4db5-9fb1-a9e27cdf9f58-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-99145d-7d5667799-hfvwn\" (UID: \"7ca5c214-0290-4db5-9fb1-a9e27cdf9f58\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-99145d-7d5667799-hfvwn"
Oct 31 01:46:51.803956 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:46:51.802279    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vf9fx\" (UniqueName: \"kubernetes.io/projected/7ca5c214-0290-4db5-9fb1-a9e27cdf9f58-kube-api-access-vf9fx\") pod \"payload-pre-processing-e2e-praxis-b-99145d-7d5667799-hfvwn\" (UID: \"7ca5c214-0290-4db5-9fb1-a9e27cdf9f58\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-99145d-7d5667799-hfvwn"
Oct 31 01:46:51.821091 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:46:51.821057    4964 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-99145d-fc449dc4f-zmf85"]
Oct 31 01:46:51.827524 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:46:51.827500    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-99145d-fc449dc4f-zmf85"
Oct 31 01:46:51.846958 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:46:51.846926    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-99145d-fc449dc4f-zmf85"]
Oct 31 01:46:51.904025 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:46:51.903982    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7ca5c214-0290-4db5-9fb1-a9e27cdf9f58-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-99145d-7d5667799-hfvwn\" (UID: \"7ca5c214-0290-4db5-9fb1-a9e27cdf9f58\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-99145d-7d5667799-hfvwn"
Oct 31 01:46:51.904211 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:46:51.904108    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vf9fx\" (UniqueName: \"kubernetes.io/projected/7ca5c214-0290-4db5-9fb1-a9e27cdf9f58-kube-api-access-vf9fx\") pod \"payload-pre-processing-e2e-praxis-b-99145d-7d5667799-hfvwn\" (UID: \"7ca5c214-0290-4db5-9fb1-a9e27cdf9f58\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-99145d-7d5667799-hfvwn"
Oct 31 01:46:51.904358 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:46:51.904276    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a3a77a68-3698-491d-8220-fa9a2c0f8278-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-99145d-fc449dc4f-zmf85\" (UID: \"a3a77a68-3698-491d-8220-fa9a2c0f8278\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-99145d-fc449dc4f-zmf85"
Oct 31 01:46:51.904358 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:46:51.904337    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lvvvg\" (UniqueName: \"kubernetes.io/projected/a3a77a68-3698-491d-8220-fa9a2c0f8278-kube-api-access-lvvvg\") pod \"payload-processing-e2e-praxis-b-99145d-fc449dc4f-zmf85\" (UID: \"a3a77a68-3698-491d-8220-fa9a2c0f8278\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-99145d-fc449dc4f-zmf85"
Oct 31 01:46:51.905311 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:46:51.905276    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7ca5c214-0290-4db5-9fb1-a9e27cdf9f58-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-99145d-7d5667799-hfvwn\" (UID: \"7ca5c214-0290-4db5-9fb1-a9e27cdf9f58\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-99145d-7d5667799-hfvwn"
Oct 31 01:46:51.962904 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:46:51.962811    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vf9fx\" (UniqueName: \"kubernetes.io/projected/7ca5c214-0290-4db5-9fb1-a9e27cdf9f58-kube-api-access-vf9fx\") pod \"payload-pre-processing-e2e-praxis-b-99145d-7d5667799-hfvwn\" (UID: \"7ca5c214-0290-4db5-9fb1-a9e27cdf9f58\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-99145d-7d5667799-hfvwn"
Oct 31 01:46:51.989536 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:46:51.989503    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-99145d-7d5667799-hfvwn"
Oct 31 01:46:52.005641 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:46:52.005597    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a3a77a68-3698-491d-8220-fa9a2c0f8278-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-99145d-fc449dc4f-zmf85\" (UID: \"a3a77a68-3698-491d-8220-fa9a2c0f8278\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-99145d-fc449dc4f-zmf85"
Oct 31 01:46:52.005794 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:46:52.005662    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lvvvg\" (UniqueName: \"kubernetes.io/projected/a3a77a68-3698-491d-8220-fa9a2c0f8278-kube-api-access-lvvvg\") pod \"payload-processing-e2e-praxis-b-99145d-fc449dc4f-zmf85\" (UID: \"a3a77a68-3698-491d-8220-fa9a2c0f8278\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-99145d-fc449dc4f-zmf85"
Oct 31 01:46:52.006441 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:46:52.006403    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a3a77a68-3698-491d-8220-fa9a2c0f8278-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-99145d-fc449dc4f-zmf85\" (UID: \"a3a77a68-3698-491d-8220-fa9a2c0f8278\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-99145d-fc449dc4f-zmf85"
Oct 31 01:46:52.069832 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:46:52.069804    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lvvvg\" (UniqueName: \"kubernetes.io/projected/a3a77a68-3698-491d-8220-fa9a2c0f8278-kube-api-access-lvvvg\") pod \"payload-processing-e2e-praxis-b-99145d-fc449dc4f-zmf85\" (UID: \"a3a77a68-3698-491d-8220-fa9a2c0f8278\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-99145d-fc449dc4f-zmf85"
Oct 31 01:46:52.147273 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:46:52.146982    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-99145d-fc449dc4f-zmf85"
Oct 31 01:46:52.385059 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:46:52.385032    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-99145d-7d5667799-hfvwn"]
Oct 31 01:46:52.386656 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:46:52.386626    4964 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ca5c214_0290_4db5_9fb1_a9e27cdf9f58.slice/crio-2bda8a820f6064be38527d8e253295151b3d9565f4ee9645c5908489bada2b55 WatchSource:0}: Error finding container 2bda8a820f6064be38527d8e253295151b3d9565f4ee9645c5908489bada2b55: Status 404 returned error can't find the container with id 2bda8a820f6064be38527d8e253295151b3d9565f4ee9645c5908489bada2b55
Oct 31 01:46:52.437022 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:46:52.436988    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:46:52.581387 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:46:52.578062    4964 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda3a77a68_3698_491d_8220_fa9a2c0f8278.slice/crio-5cda17dd4705e49ee8a012b9aa1ea71d7c73b162edcd83648afc105e06220278 WatchSource:0}: Error finding container 5cda17dd4705e49ee8a012b9aa1ea71d7c73b162edcd83648afc105e06220278: Status 404 returned error can't find the container with id 5cda17dd4705e49ee8a012b9aa1ea71d7c73b162edcd83648afc105e06220278
Oct 31 01:46:52.581387 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:46:52.578156    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-99145d-fc449dc4f-zmf85"]
Oct 31 01:46:52.703480 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:46:52.703448    4964 scope.go:122] "RemoveContainer" containerID="a5871846f0d6622ac926109861ae68e725c79d0dc8292fa2e3b809d2be05b5b3"
Oct 31 01:46:52.827413 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:46:52.827377    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-99145d-fc449dc4f-zmf85" event={"ID":"a3a77a68-3698-491d-8220-fa9a2c0f8278","Type":"ContainerStarted","Data":"4066a0916f6a6c8d52cdc173dcd6f5fb5f80f2d2bda4078d1c03482f073bf829"}
Oct 31 01:46:52.827606 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:46:52.827439    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-99145d-fc449dc4f-zmf85" event={"ID":"a3a77a68-3698-491d-8220-fa9a2c0f8278","Type":"ContainerStarted","Data":"5cda17dd4705e49ee8a012b9aa1ea71d7c73b162edcd83648afc105e06220278"}
Oct 31 01:46:52.827606 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:46:52.827593    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-99145d-fc449dc4f-zmf85"
Oct 31 01:46:52.829088 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:46:52.829064    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-99145d-7d5667799-hfvwn" event={"ID":"7ca5c214-0290-4db5-9fb1-a9e27cdf9f58","Type":"ContainerStarted","Data":"d8b22b2b6d7dee80394e27d62ab8a83b4e31f4c2ba691aa3d7b6c1819f75ea0d"}
Oct 31 01:46:52.829197 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:46:52.829112    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-99145d-7d5667799-hfvwn"
Oct 31 01:46:52.829197 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:46:52.829133    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-99145d-7d5667799-hfvwn" event={"ID":"7ca5c214-0290-4db5-9fb1-a9e27cdf9f58","Type":"ContainerStarted","Data":"2bda8a820f6064be38527d8e253295151b3d9565f4ee9645c5908489bada2b55"}
Oct 31 01:46:52.942110 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:46:52.942052    4964 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-b-99145d-fc449dc4f-zmf85" podStartSLOduration=1.9420352950000002 podStartE2EDuration="1.942035295s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:46:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:46:52.940882467 +0000 UTC m=+2581.417448847" watchObservedRunningTime="2026-10-31 01:46:52.942035295 +0000 UTC m=+2581.418601677"
Oct 31 01:46:53.058499 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:46:53.058372    4964 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-99145d-7d5667799-hfvwn" podStartSLOduration=2.058350917 podStartE2EDuration="2.058350917s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:46:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:46:53.053776493 +0000 UTC m=+2581.530342874" watchObservedRunningTime="2026-10-31 01:46:53.058350917 +0000 UTC m=+2581.534917299"
Oct 31 01:46:54.849996 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:46:54.849965    4964 generic.go:356] "Generic (PLEG): container finished" podID="79d0fdfe-95ba-4800-9636-68f7afc6829e" containerID="15e2887cfd42d5b3335dcbf8df443b50617bce918ae92dc6c7c8490d2b2440f6" exitCode=0
Oct 31 01:46:54.850472 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:46:54.850065    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-9d7b2c-openshift-default-76c779b789-tvtzf" event={"ID":"79d0fdfe-95ba-4800-9636-68f7afc6829e","Type":"ContainerDied","Data":"15e2887cfd42d5b3335dcbf8df443b50617bce918ae92dc6c7c8490d2b2440f6"}
Oct 31 01:46:54.850519 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:46:54.850494    4964 scope.go:122] "RemoveContainer" containerID="15e2887cfd42d5b3335dcbf8df443b50617bce918ae92dc6c7c8490d2b2440f6"
Oct 31 01:46:54.851993 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:46:54.851964    4964 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 01:46:54.852120 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:46:54.852042    4964 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 01:46:54.852120 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:46:54.852090    4964 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 01:46:55.859198 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:46:55.859164    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-9d7b2c-openshift-default-76c779b789-tvtzf" event={"ID":"79d0fdfe-95ba-4800-9636-68f7afc6829e","Type":"ContainerStarted","Data":"b76be12f47a569be16bab1f7039252f9201ab57c83606f066ab6a9fa9d96897f"}
Oct 31 01:46:56.283553 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:46:56.283520    4964 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-9d7b2c-openshift-default-76c779b789-tvtzf"
Oct 31 01:46:56.285047 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:46:56.285020    4964 patch_prober.go:29] interesting pod/e2e-praxis-a-9d7b2c-openshift-default-76c779b789-tvtzf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused" start-of-body=
Oct 31 01:46:56.285179 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:46:56.285067    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-9d7b2c-openshift-default-76c779b789-tvtzf" podUID="79d0fdfe-95ba-4800-9636-68f7afc6829e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused"
Oct 31 01:46:57.218343 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:46:57.213035    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:46:57.283524 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:46:57.283493    4964 patch_prober.go:29] interesting pod/e2e-praxis-a-9d7b2c-openshift-default-76c779b789-tvtzf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused" start-of-body=
Oct 31 01:46:57.283695 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:46:57.283543    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-9d7b2c-openshift-default-76c779b789-tvtzf" podUID="79d0fdfe-95ba-4800-9636-68f7afc6829e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused"
Oct 31 01:46:57.858743 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:46:57.858711    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-9d7b2c-openshift-default-76c779b789-tvtzf"
Oct 31 01:46:58.283404 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:46:58.283372    4964 patch_prober.go:29] interesting pod/e2e-praxis-a-9d7b2c-openshift-default-76c779b789-tvtzf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused" start-of-body=
Oct 31 01:46:58.283785 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:46:58.283422    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-9d7b2c-openshift-default-76c779b789-tvtzf" podUID="79d0fdfe-95ba-4800-9636-68f7afc6829e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused"
Oct 31 01:46:59.283230 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:46:59.283188    4964 patch_prober.go:29] interesting pod/e2e-praxis-a-9d7b2c-openshift-default-76c779b789-tvtzf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused" start-of-body=
Oct 31 01:46:59.283533 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:46:59.283248    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-9d7b2c-openshift-default-76c779b789-tvtzf" podUID="79d0fdfe-95ba-4800-9636-68f7afc6829e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused"
Oct 31 01:47:00.283390 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:47:00.283359    4964 patch_prober.go:29] interesting pod/e2e-praxis-a-9d7b2c-openshift-default-76c779b789-tvtzf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused" start-of-body=
Oct 31 01:47:00.283566 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:47:00.283416    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-9d7b2c-openshift-default-76c779b789-tvtzf" podUID="79d0fdfe-95ba-4800-9636-68f7afc6829e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused"
Oct 31 01:47:01.288220 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:47:01.288191    4964 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-9d7b2c-openshift-default-76c779b789-tvtzf"
Oct 31 01:47:01.289430 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:47:01.289411    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-9d7b2c-openshift-default-76c779b789-tvtzf"
Oct 31 01:47:02.218049 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:47:02.218015    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:47:03.843158 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:47:03.843131    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-99145d-7d5667799-hfvwn"
Oct 31 01:47:03.843629 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:47:03.843193    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-99145d-fc449dc4f-zmf85"
Oct 31 01:47:08.610224 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:47:08.610174    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:47:11.238446 ip-10-0-133-197 kubenswrapper[4964]: E1031 01:47:11.238411    4964 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q56x5" containerName="extractor"
Oct 31 01:47:13.495927 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:47:13.495890    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:47:13.714902 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:47:13.714858    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:47:15.653199 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:47:15.653156    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:47:16.881502 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:47:16.881460    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:47:17.539349 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:47:17.531848    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:47:17.742079 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:47:17.742035    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:47:17.923988 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:47:17.923891    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:47:18.529844 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:47:18.529803    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:47:20.073337 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:47:20.073288    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:47:22.498211 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:47:22.498167    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:47:22.731522 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:47:22.731482    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:47:22.916726 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:47:22.916613    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:47:23.718900 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:47:23.718863    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:47:23.955921 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:47:23.955887    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:47:26.696382 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:47:26.696353    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:47:26.912884 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:47:26.912846    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:47:27.063921 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:47:27.063891    4964 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-cd0c98-747b757bc-xf68k"]
Oct 31 01:47:27.069224 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:47:27.069208    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-cd0c98-747b757bc-xf68k"
Oct 31 01:47:27.121205 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:47:27.121178    4964 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-b-cd0c98\""
Oct 31 01:47:27.121738 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:47:27.121698    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jxqdm\" (UniqueName: \"kubernetes.io/projected/24c7c765-3123-429e-ad45-b95b7cf5e988-kube-api-access-jxqdm\") pod \"payload-pre-processing-e2e-praxis-b-cd0c98-747b757bc-xf68k\" (UID: \"24c7c765-3123-429e-ad45-b95b7cf5e988\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-cd0c98-747b757bc-xf68k"
Oct 31 01:47:27.121894 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:47:27.121798    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/24c7c765-3123-429e-ad45-b95b7cf5e988-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-cd0c98-747b757bc-xf68k\" (UID: \"24c7c765-3123-429e-ad45-b95b7cf5e988\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-cd0c98-747b757bc-xf68k"
Oct 31 01:47:27.191391 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:47:27.171087    4964 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-b-cd0c98-dockercfg-k78l7\""
Oct 31 01:47:27.239895 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:47:27.239854    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/24c7c765-3123-429e-ad45-b95b7cf5e988-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-cd0c98-747b757bc-xf68k\" (UID: \"24c7c765-3123-429e-ad45-b95b7cf5e988\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-cd0c98-747b757bc-xf68k"
Oct 31 01:47:27.240478 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:47:27.240396    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jxqdm\" (UniqueName: \"kubernetes.io/projected/24c7c765-3123-429e-ad45-b95b7cf5e988-kube-api-access-jxqdm\") pod \"payload-pre-processing-e2e-praxis-b-cd0c98-747b757bc-xf68k\" (UID: \"24c7c765-3123-429e-ad45-b95b7cf5e988\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-cd0c98-747b757bc-xf68k"
Oct 31 01:47:27.241867 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:47:27.241787    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/24c7c765-3123-429e-ad45-b95b7cf5e988-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-cd0c98-747b757bc-xf68k\" (UID: \"24c7c765-3123-429e-ad45-b95b7cf5e988\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-cd0c98-747b757bc-xf68k"
Oct 31 01:47:27.243495 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:47:27.240527    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-cd0c98-747b757bc-xf68k"]
Oct 31 01:47:27.257632 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:47:27.257605    4964 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-cd0c98-7dd9d94dc-skdst"]
Oct 31 01:47:27.262244 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:47:27.262224    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-cd0c98-7dd9d94dc-skdst"
Oct 31 01:47:27.276639 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:47:27.276620    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-cd0c98-7dd9d94dc-skdst"]
Oct 31 01:47:27.299679 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:47:27.299650    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jxqdm\" (UniqueName: \"kubernetes.io/projected/24c7c765-3123-429e-ad45-b95b7cf5e988-kube-api-access-jxqdm\") pod \"payload-pre-processing-e2e-praxis-b-cd0c98-747b757bc-xf68k\" (UID: \"24c7c765-3123-429e-ad45-b95b7cf5e988\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-cd0c98-747b757bc-xf68k"
Oct 31 01:47:27.343405 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:47:27.343336    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5eba908f-921b-45ce-8676-c027af587013-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-cd0c98-7dd9d94dc-skdst\" (UID: \"5eba908f-921b-45ce-8676-c027af587013\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-cd0c98-7dd9d94dc-skdst"
Oct 31 01:47:27.343514 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:47:27.343452    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x7fgm\" (UniqueName: \"kubernetes.io/projected/5eba908f-921b-45ce-8676-c027af587013-kube-api-access-x7fgm\") pod \"payload-processing-e2e-praxis-b-cd0c98-7dd9d94dc-skdst\" (UID: \"5eba908f-921b-45ce-8676-c027af587013\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-cd0c98-7dd9d94dc-skdst"
Oct 31 01:47:27.383685 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:47:27.383658    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-cd0c98-747b757bc-xf68k"
Oct 31 01:47:27.444897 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:47:27.444860    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5eba908f-921b-45ce-8676-c027af587013-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-cd0c98-7dd9d94dc-skdst\" (UID: \"5eba908f-921b-45ce-8676-c027af587013\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-cd0c98-7dd9d94dc-skdst"
Oct 31 01:47:27.445052 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:47:27.444952    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x7fgm\" (UniqueName: \"kubernetes.io/projected/5eba908f-921b-45ce-8676-c027af587013-kube-api-access-x7fgm\") pod \"payload-processing-e2e-praxis-b-cd0c98-7dd9d94dc-skdst\" (UID: \"5eba908f-921b-45ce-8676-c027af587013\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-cd0c98-7dd9d94dc-skdst"
Oct 31 01:47:27.453408 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:47:27.447777    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5eba908f-921b-45ce-8676-c027af587013-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-cd0c98-7dd9d94dc-skdst\" (UID: \"5eba908f-921b-45ce-8676-c027af587013\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-cd0c98-7dd9d94dc-skdst"
Oct 31 01:47:27.502070 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:47:27.502027    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x7fgm\" (UniqueName: \"kubernetes.io/projected/5eba908f-921b-45ce-8676-c027af587013-kube-api-access-x7fgm\") pod \"payload-processing-e2e-praxis-b-cd0c98-7dd9d94dc-skdst\" (UID: \"5eba908f-921b-45ce-8676-c027af587013\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-cd0c98-7dd9d94dc-skdst"
Oct 31 01:47:27.576638 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:47:27.576596    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-cd0c98-7dd9d94dc-skdst"
Oct 31 01:47:27.761108 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:47:27.761084    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-cd0c98-747b757bc-xf68k"]
Oct 31 01:47:27.762568 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:47:27.762540    4964 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod24c7c765_3123_429e_ad45_b95b7cf5e988.slice/crio-86cb69f4bad4efe77802cd085c356d365cccb46a127bbdc441276dbb0346e03b WatchSource:0}: Error finding container 86cb69f4bad4efe77802cd085c356d365cccb46a127bbdc441276dbb0346e03b: Status 404 returned error can't find the container with id 86cb69f4bad4efe77802cd085c356d365cccb46a127bbdc441276dbb0346e03b
Oct 31 01:47:27.961358 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:47:27.960444    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-cd0c98-7dd9d94dc-skdst"]
Oct 31 01:47:27.963579 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:47:27.962470    4964 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5eba908f_921b_45ce_8676_c027af587013.slice/crio-8e76102bc2629bac41a8bb56f711573b1d11be81b5462aee602b7aabb9cc90eb WatchSource:0}: Error finding container 8e76102bc2629bac41a8bb56f711573b1d11be81b5462aee602b7aabb9cc90eb: Status 404 returned error can't find the container with id 8e76102bc2629bac41a8bb56f711573b1d11be81b5462aee602b7aabb9cc90eb
Oct 31 01:47:28.048281 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:47:28.048251    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:47:28.090715 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:47:28.090672    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-cd0c98-7dd9d94dc-skdst" event={"ID":"5eba908f-921b-45ce-8676-c027af587013","Type":"ContainerStarted","Data":"10c4044bc5faec8e7ecc1969819dd03bbd0145eec16d7323ff277fc029994a60"}
Oct 31 01:47:28.090909 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:47:28.090740    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-cd0c98-7dd9d94dc-skdst"
Oct 31 01:47:28.090909 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:47:28.090768    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-cd0c98-7dd9d94dc-skdst" event={"ID":"5eba908f-921b-45ce-8676-c027af587013","Type":"ContainerStarted","Data":"8e76102bc2629bac41a8bb56f711573b1d11be81b5462aee602b7aabb9cc90eb"}
Oct 31 01:47:28.092235 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:47:28.092210    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-cd0c98-747b757bc-xf68k" event={"ID":"24c7c765-3123-429e-ad45-b95b7cf5e988","Type":"ContainerStarted","Data":"747402714dd7f6b363072cdb9d12517c4fb401db8046289ab2b40298015ccb0e"}
Oct 31 01:47:28.092362 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:47:28.092255    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-cd0c98-747b757bc-xf68k" event={"ID":"24c7c765-3123-429e-ad45-b95b7cf5e988","Type":"ContainerStarted","Data":"86cb69f4bad4efe77802cd085c356d365cccb46a127bbdc441276dbb0346e03b"}
Oct 31 01:47:28.092362 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:47:28.092286    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-cd0c98-747b757bc-xf68k"
Oct 31 01:47:28.208224 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:47:28.208177    4964 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-b-cd0c98-7dd9d94dc-skdst" podStartSLOduration=1.208164482 podStartE2EDuration="1.208164482s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:47:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:47:28.204363722 +0000 UTC m=+2616.680930102" watchObservedRunningTime="2026-10-31 01:47:28.208164482 +0000 UTC m=+2616.684730861"
Oct 31 01:47:28.321636 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:47:28.321572    4964 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-cd0c98-747b757bc-xf68k" podStartSLOduration=1.321551492 podStartE2EDuration="1.321551492s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:47:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:47:28.317490354 +0000 UTC m=+2616.794056737" watchObservedRunningTime="2026-10-31 01:47:28.321551492 +0000 UTC m=+2616.798117873"
Oct 31 01:47:28.757207 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:47:28.757164    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:47:28.901699 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:47:28.901670    4964 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-tlb5m_c8461fdb-12e5-4691-8b95-139d5a901f0d/payload-processing/0.log"
Oct 31 01:47:29.166505 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:47:29.166429    4964 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-tlb5m_c8461fdb-12e5-4691-8b95-139d5a901f0d/payload-processing/0.log"
Oct 31 01:47:33.919917 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:47:33.919877    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:47:39.102583 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:47:39.102549    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-cd0c98-747b757bc-xf68k"
Oct 31 01:47:39.103130 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:47:39.102636    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-cd0c98-7dd9d94dc-skdst"
Oct 31 01:47:39.236367 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:47:39.236018    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:47:41.238824 ip-10-0-133-197 kubenswrapper[4964]: E1031 01:47:41.238781    4964 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q56x5" containerName="extractor"
Oct 31 01:47:45.742779 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:47:45.742741    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:47:49.262751 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:47:49.262714    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:47:49.866284 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:47:49.866251    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:47:50.251076 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:47:50.251043    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:47:50.530681 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:47:50.530595    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:47:50.716903 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:47:50.716867    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:47:54.348018 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:47:54.347979    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:47:54.595512 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:47:54.595479    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:47:55.708782 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:47:55.708744    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:47:58.753133 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:47:58.753095    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:47:58.988802 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:47:58.988767    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:47:59.122757 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:47:59.122667    4964 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-2c0de9-5f57446ff9-wrm5h"]
Oct 31 01:47:59.128287 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:47:59.128261    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-2c0de9-5f57446ff9-wrm5h"
Oct 31 01:47:59.236508 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:47:59.236472    4964 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-b-2c0de9-dockercfg-t5th5\""
Oct 31 01:47:59.236838 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:47:59.236818    4964 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-b-2c0de9\""
Oct 31 01:47:59.281779 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:47:59.281729    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c3c14a55-5462-4545-afd3-aa589cc162c9-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-2c0de9-5f57446ff9-wrm5h\" (UID: \"c3c14a55-5462-4545-afd3-aa589cc162c9\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-2c0de9-5f57446ff9-wrm5h"
Oct 31 01:47:59.282010 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:47:59.281896    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n5cnd\" (UniqueName: \"kubernetes.io/projected/c3c14a55-5462-4545-afd3-aa589cc162c9-kube-api-access-n5cnd\") pod \"payload-pre-processing-e2e-praxis-b-2c0de9-5f57446ff9-wrm5h\" (UID: \"c3c14a55-5462-4545-afd3-aa589cc162c9\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-2c0de9-5f57446ff9-wrm5h"
Oct 31 01:47:59.284065 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:47:59.284038    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-2c0de9-5f57446ff9-wrm5h"]
Oct 31 01:47:59.300019 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:47:59.299982    4964 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-2c0de9-f5cc7596c-j6k5n"]
Oct 31 01:47:59.305804 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:47:59.305767    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-2c0de9-f5cc7596c-j6k5n"
Oct 31 01:47:59.331183 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:47:59.331144    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-2c0de9-f5cc7596c-j6k5n"]
Oct 31 01:47:59.383002 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:47:59.382895    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d7751c9b-c8db-4bf9-8193-49c3fefe90e3-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-2c0de9-f5cc7596c-j6k5n\" (UID: \"d7751c9b-c8db-4bf9-8193-49c3fefe90e3\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-2c0de9-f5cc7596c-j6k5n"
Oct 31 01:47:59.383219 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:47:59.383051    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c3c14a55-5462-4545-afd3-aa589cc162c9-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-2c0de9-5f57446ff9-wrm5h\" (UID: \"c3c14a55-5462-4545-afd3-aa589cc162c9\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-2c0de9-5f57446ff9-wrm5h"
Oct 31 01:47:59.383219 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:47:59.383102    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lvxwb\" (UniqueName: \"kubernetes.io/projected/d7751c9b-c8db-4bf9-8193-49c3fefe90e3-kube-api-access-lvxwb\") pod \"payload-processing-e2e-praxis-b-2c0de9-f5cc7596c-j6k5n\" (UID: \"d7751c9b-c8db-4bf9-8193-49c3fefe90e3\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-2c0de9-f5cc7596c-j6k5n"
Oct 31 01:47:59.383348 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:47:59.383238    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n5cnd\" (UniqueName: \"kubernetes.io/projected/c3c14a55-5462-4545-afd3-aa589cc162c9-kube-api-access-n5cnd\") pod \"payload-pre-processing-e2e-praxis-b-2c0de9-5f57446ff9-wrm5h\" (UID: \"c3c14a55-5462-4545-afd3-aa589cc162c9\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-2c0de9-5f57446ff9-wrm5h"
Oct 31 01:47:59.383812 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:47:59.383791    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c3c14a55-5462-4545-afd3-aa589cc162c9-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-2c0de9-5f57446ff9-wrm5h\" (UID: \"c3c14a55-5462-4545-afd3-aa589cc162c9\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-2c0de9-5f57446ff9-wrm5h"
Oct 31 01:47:59.440425 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:47:59.440393    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n5cnd\" (UniqueName: \"kubernetes.io/projected/c3c14a55-5462-4545-afd3-aa589cc162c9-kube-api-access-n5cnd\") pod \"payload-pre-processing-e2e-praxis-b-2c0de9-5f57446ff9-wrm5h\" (UID: \"c3c14a55-5462-4545-afd3-aa589cc162c9\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-2c0de9-5f57446ff9-wrm5h"
Oct 31 01:47:59.443131 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:47:59.443106    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-2c0de9-5f57446ff9-wrm5h"
Oct 31 01:47:59.485026 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:47:59.484992    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d7751c9b-c8db-4bf9-8193-49c3fefe90e3-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-2c0de9-f5cc7596c-j6k5n\" (UID: \"d7751c9b-c8db-4bf9-8193-49c3fefe90e3\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-2c0de9-f5cc7596c-j6k5n"
Oct 31 01:47:59.485213 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:47:59.485085    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lvxwb\" (UniqueName: \"kubernetes.io/projected/d7751c9b-c8db-4bf9-8193-49c3fefe90e3-kube-api-access-lvxwb\") pod \"payload-processing-e2e-praxis-b-2c0de9-f5cc7596c-j6k5n\" (UID: \"d7751c9b-c8db-4bf9-8193-49c3fefe90e3\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-2c0de9-f5cc7596c-j6k5n"
Oct 31 01:47:59.485776 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:47:59.485755    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d7751c9b-c8db-4bf9-8193-49c3fefe90e3-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-2c0de9-f5cc7596c-j6k5n\" (UID: \"d7751c9b-c8db-4bf9-8193-49c3fefe90e3\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-2c0de9-f5cc7596c-j6k5n"
Oct 31 01:47:59.542779 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:47:59.542748    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lvxwb\" (UniqueName: \"kubernetes.io/projected/d7751c9b-c8db-4bf9-8193-49c3fefe90e3-kube-api-access-lvxwb\") pod \"payload-processing-e2e-praxis-b-2c0de9-f5cc7596c-j6k5n\" (UID: \"d7751c9b-c8db-4bf9-8193-49c3fefe90e3\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-2c0de9-f5cc7596c-j6k5n"
Oct 31 01:47:59.621199 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:47:59.621168    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-2c0de9-f5cc7596c-j6k5n"
Oct 31 01:47:59.829025 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:47:59.828998    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-2c0de9-5f57446ff9-wrm5h"]
Oct 31 01:47:59.831042 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:47:59.831012    4964 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc3c14a55_5462_4545_afd3_aa589cc162c9.slice/crio-5b0d33bdd7809dcffd2e308cf9ad84980eb0b387202ab12d4272efb4b695bac7 WatchSource:0}: Error finding container 5b0d33bdd7809dcffd2e308cf9ad84980eb0b387202ab12d4272efb4b695bac7: Status 404 returned error can't find the container with id 5b0d33bdd7809dcffd2e308cf9ad84980eb0b387202ab12d4272efb4b695bac7
Oct 31 01:48:00.003520 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:48:00.003489    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-2c0de9-f5cc7596c-j6k5n"]
Oct 31 01:48:00.004620 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:48:00.004588    4964 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd7751c9b_c8db_4bf9_8193_49c3fefe90e3.slice/crio-dd8619a30cc35f9ba4bef7a1a98c9be447496bd9bc6e03a56719b7fd7879ca72 WatchSource:0}: Error finding container dd8619a30cc35f9ba4bef7a1a98c9be447496bd9bc6e03a56719b7fd7879ca72: Status 404 returned error can't find the container with id dd8619a30cc35f9ba4bef7a1a98c9be447496bd9bc6e03a56719b7fd7879ca72
Oct 31 01:48:00.334394 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:48:00.334348    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-2c0de9-f5cc7596c-j6k5n" event={"ID":"d7751c9b-c8db-4bf9-8193-49c3fefe90e3","Type":"ContainerStarted","Data":"87036946e9d1a7812a27f2e99faddd8ed33478a12e4f359fb8b16729790917cb"}
Oct 31 01:48:00.334624 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:48:00.334432    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-2c0de9-f5cc7596c-j6k5n"
Oct 31 01:48:00.334624 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:48:00.334458    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-2c0de9-f5cc7596c-j6k5n" event={"ID":"d7751c9b-c8db-4bf9-8193-49c3fefe90e3","Type":"ContainerStarted","Data":"dd8619a30cc35f9ba4bef7a1a98c9be447496bd9bc6e03a56719b7fd7879ca72"}
Oct 31 01:48:00.336230 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:48:00.336203    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-2c0de9-5f57446ff9-wrm5h" event={"ID":"c3c14a55-5462-4545-afd3-aa589cc162c9","Type":"ContainerStarted","Data":"2e44b411cf520ac3b914e0f8a123cfe497d6f9b28e17e05bdb2726524c808162"}
Oct 31 01:48:00.336378 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:48:00.336253    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-2c0de9-5f57446ff9-wrm5h"
Oct 31 01:48:00.336378 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:48:00.336270    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-2c0de9-5f57446ff9-wrm5h" event={"ID":"c3c14a55-5462-4545-afd3-aa589cc162c9","Type":"ContainerStarted","Data":"5b0d33bdd7809dcffd2e308cf9ad84980eb0b387202ab12d4272efb4b695bac7"}
Oct 31 01:48:00.449311 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:48:00.449236    4964 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-b-2c0de9-f5cc7596c-j6k5n" podStartSLOduration=1.449218334 podStartE2EDuration="1.449218334s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:47:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:48:00.44786713 +0000 UTC m=+2648.924433514" watchObservedRunningTime="2026-10-31 01:48:00.449218334 +0000 UTC m=+2648.925784713"
Oct 31 01:48:00.562284 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:48:00.562231    4964 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-2c0de9-5f57446ff9-wrm5h" podStartSLOduration=1.562214978 podStartE2EDuration="1.562214978s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:47:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:48:00.560535339 +0000 UTC m=+2649.037101719" watchObservedRunningTime="2026-10-31 01:48:00.562214978 +0000 UTC m=+2649.038781357"
Oct 31 01:48:00.589808 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:48:00.589723    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:48:05.262978 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:48:05.262935    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:48:05.479932 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:48:05.479896    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:48:09.380736 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:48:09.380691    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:48:09.598556 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:48:09.598522    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:48:10.289188 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:48:10.289137    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:48:10.858094 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:48:10.858058    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:48:11.239648 ip-10-0-133-197 kubenswrapper[4964]: E1031 01:48:11.239605    4964 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q56x5" containerName="extractor"
Oct 31 01:48:11.346806 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:48:11.346776    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-2c0de9-f5cc7596c-j6k5n"
Oct 31 01:48:11.346999 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:48:11.346843    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-2c0de9-5f57446ff9-wrm5h"
Oct 31 01:48:15.205471 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:48:15.205440    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:48:19.929704 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:48:19.929670    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:48:20.406308 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:48:20.406277    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:48:24.852767 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:48:24.852729    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:48:31.290227 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:48:31.290184    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:48:36.028940 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:48:36.028886    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:48:40.732769 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:48:40.732732    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:48:41.238359 ip-10-0-133-197 kubenswrapper[4964]: E1031 01:48:41.238291    4964 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q56x5" containerName="extractor"
Oct 31 01:48:45.439792 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:48:45.439747    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:48:45.658261 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:48:45.658217    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:48:45.850687 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:48:45.850652    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:48:50.383096 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:48:50.383052    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:48:52.185005 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:48:52.184975    4964 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-9257n_f0c0ada7-7688-4dc2-97fc-813e4d0c4ef3/console-operator/2.log"
Oct 31 01:48:52.195525 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:48:52.195499    4964 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-9257n_f0c0ada7-7688-4dc2-97fc-813e4d0c4ef3/console-operator/2.log"
Oct 31 01:48:55.252031 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:48:55.251990    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:49:00.026144 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:49:00.026106    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:49:04.944153 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:49:04.944115    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:49:09.753033 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:49:09.752993    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:49:11.239125 ip-10-0-133-197 kubenswrapper[4964]: E1031 01:49:11.239086    4964 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q56x5" containerName="extractor"
Oct 31 01:49:14.123024 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:49:14.122993    4964 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-2c0de9-5ff79799cc-hj9vt"]
Oct 31 01:49:14.127440 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:49:14.127422    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-2c0de9-5ff79799cc-hj9vt"
Oct 31 01:49:14.194456 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:49:14.194427    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b073d0f5-5121-480d-b530-64568abb47f5-maas-api-tls\") pod \"maas-api-e2e-praxis-b-2c0de9-5ff79799cc-hj9vt\" (UID: \"b073d0f5-5121-480d-b530-64568abb47f5\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-2c0de9-5ff79799cc-hj9vt"
Oct 31 01:49:14.194618 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:49:14.194465    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b073d0f5-5121-480d-b530-64568abb47f5-metrics-certs\") pod \"maas-api-e2e-praxis-b-2c0de9-5ff79799cc-hj9vt\" (UID: \"b073d0f5-5121-480d-b530-64568abb47f5\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-2c0de9-5ff79799cc-hj9vt"
Oct 31 01:49:14.194698 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:49:14.194673    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b073d0f5-5121-480d-b530-64568abb47f5-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-2c0de9-5ff79799cc-hj9vt\" (UID: \"b073d0f5-5121-480d-b530-64568abb47f5\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-2c0de9-5ff79799cc-hj9vt"
Oct 31 01:49:14.194759 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:49:14.194718    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wljhb\" (UniqueName: \"kubernetes.io/projected/b073d0f5-5121-480d-b530-64568abb47f5-kube-api-access-wljhb\") pod \"maas-api-e2e-praxis-b-2c0de9-5ff79799cc-hj9vt\" (UID: \"b073d0f5-5121-480d-b530-64568abb47f5\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-2c0de9-5ff79799cc-hj9vt"
Oct 31 01:49:14.229097 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:49:14.229062    4964 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-b-2c0de9-serving-cert\""
Oct 31 01:49:14.232827 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:49:14.232800    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-2c0de9-5ff79799cc-hj9vt"]
Oct 31 01:49:14.296038 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:49:14.296006    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b073d0f5-5121-480d-b530-64568abb47f5-maas-api-tls\") pod \"maas-api-e2e-praxis-b-2c0de9-5ff79799cc-hj9vt\" (UID: \"b073d0f5-5121-480d-b530-64568abb47f5\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-2c0de9-5ff79799cc-hj9vt"
Oct 31 01:49:14.296199 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:49:14.296047    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b073d0f5-5121-480d-b530-64568abb47f5-metrics-certs\") pod \"maas-api-e2e-praxis-b-2c0de9-5ff79799cc-hj9vt\" (UID: \"b073d0f5-5121-480d-b530-64568abb47f5\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-2c0de9-5ff79799cc-hj9vt"
Oct 31 01:49:14.296199 ip-10-0-133-197 kubenswrapper[4964]: E1031 01:49:14.296143    4964 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-praxis-b-2c0de9-serving-cert: secret "maas-api-e2e-praxis-b-2c0de9-serving-cert" not found
Oct 31 01:49:14.296199 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:49:14.296189    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b073d0f5-5121-480d-b530-64568abb47f5-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-2c0de9-5ff79799cc-hj9vt\" (UID: \"b073d0f5-5121-480d-b530-64568abb47f5\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-2c0de9-5ff79799cc-hj9vt"
Oct 31 01:49:14.296401 ip-10-0-133-197 kubenswrapper[4964]: E1031 01:49:14.296212    4964 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b073d0f5-5121-480d-b530-64568abb47f5-maas-api-tls podName:b073d0f5-5121-480d-b530-64568abb47f5 nodeName:}" failed. No retries permitted until 2026-10-31 01:49:14.796197167 +0000 UTC m=+2723.272763525 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/b073d0f5-5121-480d-b530-64568abb47f5-maas-api-tls") pod "maas-api-e2e-praxis-b-2c0de9-5ff79799cc-hj9vt" (UID: "b073d0f5-5121-480d-b530-64568abb47f5") : secret "maas-api-e2e-praxis-b-2c0de9-serving-cert" not found
Oct 31 01:49:14.296401 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:49:14.296258    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wljhb\" (UniqueName: \"kubernetes.io/projected/b073d0f5-5121-480d-b530-64568abb47f5-kube-api-access-wljhb\") pod \"maas-api-e2e-praxis-b-2c0de9-5ff79799cc-hj9vt\" (UID: \"b073d0f5-5121-480d-b530-64568abb47f5\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-2c0de9-5ff79799cc-hj9vt"
Oct 31 01:49:14.296971 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:49:14.296949    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b073d0f5-5121-480d-b530-64568abb47f5-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-2c0de9-5ff79799cc-hj9vt\" (UID: \"b073d0f5-5121-480d-b530-64568abb47f5\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-2c0de9-5ff79799cc-hj9vt"
Oct 31 01:49:14.298770 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:49:14.298749    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b073d0f5-5121-480d-b530-64568abb47f5-metrics-certs\") pod \"maas-api-e2e-praxis-b-2c0de9-5ff79799cc-hj9vt\" (UID: \"b073d0f5-5121-480d-b530-64568abb47f5\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-2c0de9-5ff79799cc-hj9vt"
Oct 31 01:49:14.353642 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:49:14.353616    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wljhb\" (UniqueName: \"kubernetes.io/projected/b073d0f5-5121-480d-b530-64568abb47f5-kube-api-access-wljhb\") pod \"maas-api-e2e-praxis-b-2c0de9-5ff79799cc-hj9vt\" (UID: \"b073d0f5-5121-480d-b530-64568abb47f5\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-2c0de9-5ff79799cc-hj9vt"
Oct 31 01:49:14.354933 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:49:14.354908    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:49:14.580258 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:49:14.580221    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:49:14.755815 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:49:14.755777    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:49:14.802222 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:49:14.802189    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b073d0f5-5121-480d-b530-64568abb47f5-maas-api-tls\") pod \"maas-api-e2e-praxis-b-2c0de9-5ff79799cc-hj9vt\" (UID: \"b073d0f5-5121-480d-b530-64568abb47f5\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-2c0de9-5ff79799cc-hj9vt"
Oct 31 01:49:14.805225 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:49:14.805194    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b073d0f5-5121-480d-b530-64568abb47f5-maas-api-tls\") pod \"maas-api-e2e-praxis-b-2c0de9-5ff79799cc-hj9vt\" (UID: \"b073d0f5-5121-480d-b530-64568abb47f5\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-2c0de9-5ff79799cc-hj9vt"
Oct 31 01:49:15.041735 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:49:15.041708    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-2c0de9-5ff79799cc-hj9vt"
Oct 31 01:49:15.421044 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:49:15.421012    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-2c0de9-5ff79799cc-hj9vt"]
Oct 31 01:49:15.423017 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:49:15.422986    4964 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb073d0f5_5121_480d_b530_64568abb47f5.slice/crio-322676be770c258a8b8d7484bb60aba76daea42c0df39cc8eacf1ba0be29a125 WatchSource:0}: Error finding container 322676be770c258a8b8d7484bb60aba76daea42c0df39cc8eacf1ba0be29a125: Status 404 returned error can't find the container with id 322676be770c258a8b8d7484bb60aba76daea42c0df39cc8eacf1ba0be29a125
Oct 31 01:49:15.894935 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:49:15.894896    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-2c0de9-5ff79799cc-hj9vt" event={"ID":"b073d0f5-5121-480d-b530-64568abb47f5","Type":"ContainerStarted","Data":"322676be770c258a8b8d7484bb60aba76daea42c0df39cc8eacf1ba0be29a125"}
Oct 31 01:49:16.903842 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:49:16.903798    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-2c0de9-5ff79799cc-hj9vt" event={"ID":"b073d0f5-5121-480d-b530-64568abb47f5","Type":"ContainerStarted","Data":"eacc8fa63a3e54bcad6ebe77bf10e9bd9312d530a72990fd574f9d7cf8ca7096"}
Oct 31 01:49:16.904255 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:49:16.903876    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-2c0de9-5ff79799cc-hj9vt"
Oct 31 01:49:17.020074 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:49:17.020021    4964 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-2c0de9-5ff79799cc-hj9vt" podStartSLOduration=2.160968886 podStartE2EDuration="3.020008281s" totalImagesPullingTime="859.039395ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:49:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:49:17.017562249 +0000 UTC m=+2725.494128633" watchObservedRunningTime="2026-10-31 01:49:17.020008281 +0000 UTC m=+2725.496574659"
Oct 31 01:49:17.467962 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:49:17.467927    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:49:18.391408 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:49:18.391367    4964 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-qt5vs"]
Oct 31 01:49:18.396606 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:49:18.396585    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-qt5vs"
Oct 31 01:49:18.447489 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:49:18.447462    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fb838d78-ff4a-4199-a6e7-d69bd176a022-istiod-ca-cert\") pod \"e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-qt5vs\" (UID: \"fb838d78-ff4a-4199-a6e7-d69bd176a022\") " pod="openshift-ingress/e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-qt5vs"
Oct 31 01:49:18.447607 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:49:18.447577    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fb838d78-ff4a-4199-a6e7-d69bd176a022-istio-podinfo\") pod \"e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-qt5vs\" (UID: \"fb838d78-ff4a-4199-a6e7-d69bd176a022\") " pod="openshift-ingress/e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-qt5vs"
Oct 31 01:49:18.447666 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:49:18.447606    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fb838d78-ff4a-4199-a6e7-d69bd176a022-istio-envoy\") pod \"e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-qt5vs\" (UID: \"fb838d78-ff4a-4199-a6e7-d69bd176a022\") " pod="openshift-ingress/e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-qt5vs"
Oct 31 01:49:18.447666 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:49:18.447635    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb838d78-ff4a-4199-a6e7-d69bd176a022-workload-socket\") pod \"e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-qt5vs\" (UID: \"fb838d78-ff4a-4199-a6e7-d69bd176a022\") " pod="openshift-ingress/e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-qt5vs"
Oct 31 01:49:18.447779 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:49:18.447740    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fb838d78-ff4a-4199-a6e7-d69bd176a022-workload-certs\") pod \"e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-qt5vs\" (UID: \"fb838d78-ff4a-4199-a6e7-d69bd176a022\") " pod="openshift-ingress/e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-qt5vs"
Oct 31 01:49:18.447842 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:49:18.447825    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fb838d78-ff4a-4199-a6e7-d69bd176a022-istio-token\") pod \"e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-qt5vs\" (UID: \"fb838d78-ff4a-4199-a6e7-d69bd176a022\") " pod="openshift-ingress/e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-qt5vs"
Oct 31 01:49:18.447900 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:49:18.447856    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb838d78-ff4a-4199-a6e7-d69bd176a022-credential-socket\") pod \"e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-qt5vs\" (UID: \"fb838d78-ff4a-4199-a6e7-d69bd176a022\") " pod="openshift-ingress/e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-qt5vs"
Oct 31 01:49:18.447967 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:49:18.447902    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rntgc\" (UniqueName: \"kubernetes.io/projected/fb838d78-ff4a-4199-a6e7-d69bd176a022-kube-api-access-rntgc\") pod \"e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-qt5vs\" (UID: \"fb838d78-ff4a-4199-a6e7-d69bd176a022\") " pod="openshift-ingress/e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-qt5vs"
Oct 31 01:49:18.448027 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:49:18.447974    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fb838d78-ff4a-4199-a6e7-d69bd176a022-istio-data\") pod \"e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-qt5vs\" (UID: \"fb838d78-ff4a-4199-a6e7-d69bd176a022\") " pod="openshift-ingress/e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-qt5vs"
Oct 31 01:49:18.448232 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:49:18.448215    4964 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-b-cd0c98-openshift-default-dockercfg-xf86x\""
Oct 31 01:49:18.502951 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:49:18.502919    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-qt5vs"]
Oct 31 01:49:18.549832 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:49:18.549795    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fb838d78-ff4a-4199-a6e7-d69bd176a022-istiod-ca-cert\") pod \"e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-qt5vs\" (UID: \"fb838d78-ff4a-4199-a6e7-d69bd176a022\") " pod="openshift-ingress/e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-qt5vs"
Oct 31 01:49:18.550027 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:49:18.549902    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fb838d78-ff4a-4199-a6e7-d69bd176a022-istio-podinfo\") pod \"e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-qt5vs\" (UID: \"fb838d78-ff4a-4199-a6e7-d69bd176a022\") " pod="openshift-ingress/e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-qt5vs"
Oct 31 01:49:18.550027 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:49:18.549935    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fb838d78-ff4a-4199-a6e7-d69bd176a022-istio-envoy\") pod \"e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-qt5vs\" (UID: \"fb838d78-ff4a-4199-a6e7-d69bd176a022\") " pod="openshift-ingress/e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-qt5vs"
Oct 31 01:49:18.550027 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:49:18.549967    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb838d78-ff4a-4199-a6e7-d69bd176a022-workload-socket\") pod \"e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-qt5vs\" (UID: \"fb838d78-ff4a-4199-a6e7-d69bd176a022\") " pod="openshift-ingress/e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-qt5vs"
Oct 31 01:49:18.550027 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:49:18.549995    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fb838d78-ff4a-4199-a6e7-d69bd176a022-workload-certs\") pod \"e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-qt5vs\" (UID: \"fb838d78-ff4a-4199-a6e7-d69bd176a022\") " pod="openshift-ingress/e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-qt5vs"
Oct 31 01:49:18.550027 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:49:18.550020    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fb838d78-ff4a-4199-a6e7-d69bd176a022-istio-token\") pod \"e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-qt5vs\" (UID: \"fb838d78-ff4a-4199-a6e7-d69bd176a022\") " pod="openshift-ingress/e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-qt5vs"
Oct 31 01:49:18.550294 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:49:18.550054    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb838d78-ff4a-4199-a6e7-d69bd176a022-credential-socket\") pod \"e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-qt5vs\" (UID: \"fb838d78-ff4a-4199-a6e7-d69bd176a022\") " pod="openshift-ingress/e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-qt5vs"
Oct 31 01:49:18.550294 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:49:18.550111    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rntgc\" (UniqueName: \"kubernetes.io/projected/fb838d78-ff4a-4199-a6e7-d69bd176a022-kube-api-access-rntgc\") pod \"e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-qt5vs\" (UID: \"fb838d78-ff4a-4199-a6e7-d69bd176a022\") " pod="openshift-ingress/e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-qt5vs"
Oct 31 01:49:18.550294 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:49:18.550150    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fb838d78-ff4a-4199-a6e7-d69bd176a022-istio-data\") pod \"e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-qt5vs\" (UID: \"fb838d78-ff4a-4199-a6e7-d69bd176a022\") " pod="openshift-ingress/e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-qt5vs"
Oct 31 01:49:18.550495 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:49:18.550434    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fb838d78-ff4a-4199-a6e7-d69bd176a022-workload-certs\") pod \"e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-qt5vs\" (UID: \"fb838d78-ff4a-4199-a6e7-d69bd176a022\") " pod="openshift-ingress/e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-qt5vs"
Oct 31 01:49:18.550554 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:49:18.550503    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb838d78-ff4a-4199-a6e7-d69bd176a022-workload-socket\") pod \"e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-qt5vs\" (UID: \"fb838d78-ff4a-4199-a6e7-d69bd176a022\") " pod="openshift-ingress/e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-qt5vs"
Oct 31 01:49:18.550646 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:49:18.550617    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fb838d78-ff4a-4199-a6e7-d69bd176a022-istio-data\") pod \"e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-qt5vs\" (UID: \"fb838d78-ff4a-4199-a6e7-d69bd176a022\") " pod="openshift-ingress/e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-qt5vs"
Oct 31 01:49:18.550646 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:49:18.550636    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fb838d78-ff4a-4199-a6e7-d69bd176a022-istiod-ca-cert\") pod \"e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-qt5vs\" (UID: \"fb838d78-ff4a-4199-a6e7-d69bd176a022\") " pod="openshift-ingress/e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-qt5vs"
Oct 31 01:49:18.550731 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:49:18.550669    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb838d78-ff4a-4199-a6e7-d69bd176a022-credential-socket\") pod \"e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-qt5vs\" (UID: \"fb838d78-ff4a-4199-a6e7-d69bd176a022\") " pod="openshift-ingress/e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-qt5vs"
Oct 31 01:49:18.552719 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:49:18.552696    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fb838d78-ff4a-4199-a6e7-d69bd176a022-istio-podinfo\") pod \"e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-qt5vs\" (UID: \"fb838d78-ff4a-4199-a6e7-d69bd176a022\") " pod="openshift-ingress/e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-qt5vs"
Oct 31 01:49:18.552818 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:49:18.552722    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fb838d78-ff4a-4199-a6e7-d69bd176a022-istio-envoy\") pod \"e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-qt5vs\" (UID: \"fb838d78-ff4a-4199-a6e7-d69bd176a022\") " pod="openshift-ingress/e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-qt5vs"
Oct 31 01:49:18.610916 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:49:18.610866    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fb838d78-ff4a-4199-a6e7-d69bd176a022-istio-token\") pod \"e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-qt5vs\" (UID: \"fb838d78-ff4a-4199-a6e7-d69bd176a022\") " pod="openshift-ingress/e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-qt5vs"
Oct 31 01:49:18.611693 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:49:18.611663    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rntgc\" (UniqueName: \"kubernetes.io/projected/fb838d78-ff4a-4199-a6e7-d69bd176a022-kube-api-access-rntgc\") pod \"e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-qt5vs\" (UID: \"fb838d78-ff4a-4199-a6e7-d69bd176a022\") " pod="openshift-ingress/e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-qt5vs"
Oct 31 01:49:18.710555 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:49:18.710459    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-qt5vs"
Oct 31 01:49:19.143435 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:49:19.143407    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-qt5vs"]
Oct 31 01:49:19.144758 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:49:19.144733    4964 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb838d78_ff4a_4199_a6e7_d69bd176a022.slice/crio-1c48c5eade4ac2e01916a1c175db136e63e1ed915b7345bdaa2b3e47f2085b5f WatchSource:0}: Error finding container 1c48c5eade4ac2e01916a1c175db136e63e1ed915b7345bdaa2b3e47f2085b5f: Status 404 returned error can't find the container with id 1c48c5eade4ac2e01916a1c175db136e63e1ed915b7345bdaa2b3e47f2085b5f
Oct 31 01:49:19.147762 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:49:19.147730    4964 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 01:49:19.147840 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:49:19.147796    4964 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 01:49:19.147840 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:49:19.147831    4964 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 01:49:19.660378 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:49:19.660254    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:49:19.937052 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:49:19.936951    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-qt5vs" event={"ID":"fb838d78-ff4a-4199-a6e7-d69bd176a022","Type":"ContainerStarted","Data":"ea890ad2297c2b1c42671bdcbf1b3c333d54df0506082ee88e234310a7662fcb"}
Oct 31 01:49:19.937052 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:49:19.937001    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-qt5vs" event={"ID":"fb838d78-ff4a-4199-a6e7-d69bd176a022","Type":"ContainerStarted","Data":"1c48c5eade4ac2e01916a1c175db136e63e1ed915b7345bdaa2b3e47f2085b5f"}
Oct 31 01:49:20.055935 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:49:20.055881    4964 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-qt5vs" podStartSLOduration=2.0558673 podStartE2EDuration="2.0558673s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:49:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:49:20.052427732 +0000 UTC m=+2728.528994111" watchObservedRunningTime="2026-10-31 01:49:20.0558673 +0000 UTC m=+2728.532433679"
Oct 31 01:49:20.713725 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:49:20.712585    4964 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-qt5vs"
Oct 31 01:49:20.719342 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:49:20.718490    4964 patch_prober.go:29] interesting pod/e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-qt5vs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.158:15021/healthz/ready\": dial tcp 10.132.0.158:15021: connect: connection refused" start-of-body=
Oct 31 01:49:20.719342 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:49:20.718552    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-qt5vs" podUID="fb838d78-ff4a-4199-a6e7-d69bd176a022" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.158:15021/healthz/ready\": dial tcp 10.132.0.158:15021: connect: connection refused"
Oct 31 01:49:21.712634 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:49:21.712601    4964 patch_prober.go:29] interesting pod/e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-qt5vs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.158:15021/healthz/ready\": dial tcp 10.132.0.158:15021: connect: connection refused" start-of-body=
Oct 31 01:49:21.712904 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:49:21.712877    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-qt5vs" podUID="fb838d78-ff4a-4199-a6e7-d69bd176a022" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.158:15021/healthz/ready\": dial tcp 10.132.0.158:15021: connect: connection refused"
Oct 31 01:49:22.713669 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:49:22.713632    4964 patch_prober.go:29] interesting pod/e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-qt5vs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.158:15021/healthz/ready\": dial tcp 10.132.0.158:15021: connect: connection refused" start-of-body=
Oct 31 01:49:22.714468 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:49:22.714424    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-qt5vs" podUID="fb838d78-ff4a-4199-a6e7-d69bd176a022" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.158:15021/healthz/ready\": dial tcp 10.132.0.158:15021: connect: connection refused"
Oct 31 01:49:22.920589 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:49:22.920558    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-2c0de9-5ff79799cc-hj9vt"
Oct 31 01:49:23.712003 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:49:23.711966    4964 patch_prober.go:29] interesting pod/e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-qt5vs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.158:15021/healthz/ready\": dial tcp 10.132.0.158:15021: connect: connection refused" start-of-body=
Oct 31 01:49:23.712301 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:49:23.712269    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-qt5vs" podUID="fb838d78-ff4a-4199-a6e7-d69bd176a022" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.158:15021/healthz/ready\": dial tcp 10.132.0.158:15021: connect: connection refused"
Oct 31 01:49:24.717705 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:49:24.717666    4964 patch_prober.go:29] interesting pod/e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-qt5vs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.158:15021/healthz/ready\": dial tcp 10.132.0.158:15021: connect: connection refused" start-of-body=
Oct 31 01:49:24.718308 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:49:24.718276    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-qt5vs" podUID="fb838d78-ff4a-4199-a6e7-d69bd176a022" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.158:15021/healthz/ready\": dial tcp 10.132.0.158:15021: connect: connection refused"
Oct 31 01:49:24.932458 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:49:24.932423    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:49:25.717764 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:49:25.717733    4964 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-qt5vs"
Oct 31 01:49:25.718233 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:49:25.718062    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-qt5vs"
Oct 31 01:49:25.719053 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:49:25.719032    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-qt5vs"
Oct 31 01:49:26.442705 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:49:26.442670    4964 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-klw8t"]
Oct 31 01:49:26.442911 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:49:26.442884    4964 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-klw8t" podUID="c15fa68c-09a2-453a-81c1-687a3ce2b87a" containerName="istio-proxy" containerID="cri-o://2ed0bd1cfed13a4fab558bde12f03065ff30db31105d26f8092c7fdb2935252b" gracePeriod=30
Oct 31 01:49:29.144852 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:49:29.144811    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:49:30.231821 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:49:30.231743    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:49:30.759474 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:49:30.759435    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:49:31.731074 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:49:31.731048    4964 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-klw8t"
Oct 31 01:49:31.743992 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:49:31.743968    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c15fa68c-09a2-453a-81c1-687a3ce2b87a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c15fa68c-09a2-453a-81c1-687a3ce2b87a-workload-socket\") pod \"c15fa68c-09a2-453a-81c1-687a3ce2b87a\" (UID: \"c15fa68c-09a2-453a-81c1-687a3ce2b87a\") "
Oct 31 01:49:31.744119 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:49:31.743998    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c15fa68c-09a2-453a-81c1-687a3ce2b87a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c15fa68c-09a2-453a-81c1-687a3ce2b87a-workload-certs\") pod \"c15fa68c-09a2-453a-81c1-687a3ce2b87a\" (UID: \"c15fa68c-09a2-453a-81c1-687a3ce2b87a\") "
Oct 31 01:49:31.744119 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:49:31.744030    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c15fa68c-09a2-453a-81c1-687a3ce2b87a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c15fa68c-09a2-453a-81c1-687a3ce2b87a-istio-podinfo\") pod \"c15fa68c-09a2-453a-81c1-687a3ce2b87a\" (UID: \"c15fa68c-09a2-453a-81c1-687a3ce2b87a\") "
Oct 31 01:49:31.744216 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:49:31.744134    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c15fa68c-09a2-453a-81c1-687a3ce2b87a-kube-api-access-zxjc6\" (UniqueName: \"kubernetes.io/projected/c15fa68c-09a2-453a-81c1-687a3ce2b87a-kube-api-access-zxjc6\") pod \"c15fa68c-09a2-453a-81c1-687a3ce2b87a\" (UID: \"c15fa68c-09a2-453a-81c1-687a3ce2b87a\") "
Oct 31 01:49:31.744216 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:49:31.744164    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c15fa68c-09a2-453a-81c1-687a3ce2b87a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c15fa68c-09a2-453a-81c1-687a3ce2b87a-istiod-ca-cert\") pod \"c15fa68c-09a2-453a-81c1-687a3ce2b87a\" (UID: \"c15fa68c-09a2-453a-81c1-687a3ce2b87a\") "
Oct 31 01:49:31.744351 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:49:31.744203    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c15fa68c-09a2-453a-81c1-687a3ce2b87a-workload-socket" pod "c15fa68c-09a2-453a-81c1-687a3ce2b87a" (UID: "c15fa68c-09a2-453a-81c1-687a3ce2b87a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:49:31.744351 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:49:31.744213    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c15fa68c-09a2-453a-81c1-687a3ce2b87a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c15fa68c-09a2-453a-81c1-687a3ce2b87a-credential-socket\") pod \"c15fa68c-09a2-453a-81c1-687a3ce2b87a\" (UID: \"c15fa68c-09a2-453a-81c1-687a3ce2b87a\") "
Oct 31 01:49:31.744351 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:49:31.744296    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c15fa68c-09a2-453a-81c1-687a3ce2b87a-workload-certs" pod "c15fa68c-09a2-453a-81c1-687a3ce2b87a" (UID: "c15fa68c-09a2-453a-81c1-687a3ce2b87a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:49:31.744526 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:49:31.744353    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c15fa68c-09a2-453a-81c1-687a3ce2b87a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c15fa68c-09a2-453a-81c1-687a3ce2b87a-istio-data\") pod \"c15fa68c-09a2-453a-81c1-687a3ce2b87a\" (UID: \"c15fa68c-09a2-453a-81c1-687a3ce2b87a\") "
Oct 31 01:49:31.744526 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:49:31.744439    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c15fa68c-09a2-453a-81c1-687a3ce2b87a-credential-socket" pod "c15fa68c-09a2-453a-81c1-687a3ce2b87a" (UID: "c15fa68c-09a2-453a-81c1-687a3ce2b87a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:49:31.744526 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:49:31.744506    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c15fa68c-09a2-453a-81c1-687a3ce2b87a-istio-token\" (UniqueName: \"kubernetes.io/projected/c15fa68c-09a2-453a-81c1-687a3ce2b87a-istio-token\") pod \"c15fa68c-09a2-453a-81c1-687a3ce2b87a\" (UID: \"c15fa68c-09a2-453a-81c1-687a3ce2b87a\") "
Oct 31 01:49:31.744662 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:49:31.744547    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c15fa68c-09a2-453a-81c1-687a3ce2b87a-istiod-ca-cert" pod "c15fa68c-09a2-453a-81c1-687a3ce2b87a" (UID: "c15fa68c-09a2-453a-81c1-687a3ce2b87a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:49:31.744662 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:49:31.744567    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c15fa68c-09a2-453a-81c1-687a3ce2b87a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c15fa68c-09a2-453a-81c1-687a3ce2b87a-istio-envoy\") pod \"c15fa68c-09a2-453a-81c1-687a3ce2b87a\" (UID: \"c15fa68c-09a2-453a-81c1-687a3ce2b87a\") "
Oct 31 01:49:31.744662 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:49:31.744590    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c15fa68c-09a2-453a-81c1-687a3ce2b87a-istio-data" pod "c15fa68c-09a2-453a-81c1-687a3ce2b87a" (UID: "c15fa68c-09a2-453a-81c1-687a3ce2b87a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:49:31.745091 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:49:31.745069    4964 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c15fa68c-09a2-453a-81c1-687a3ce2b87a-workload-socket\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:49:31.745091 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:49:31.745091    4964 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c15fa68c-09a2-453a-81c1-687a3ce2b87a-workload-certs\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:49:31.745478 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:49:31.745101    4964 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c15fa68c-09a2-453a-81c1-687a3ce2b87a-istiod-ca-cert\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:49:31.745478 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:49:31.745109    4964 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c15fa68c-09a2-453a-81c1-687a3ce2b87a-credential-socket\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:49:31.745478 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:49:31.745117    4964 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c15fa68c-09a2-453a-81c1-687a3ce2b87a-istio-data\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:49:31.746689 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:49:31.746661    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c15fa68c-09a2-453a-81c1-687a3ce2b87a-kube-api-access-zxjc6" pod "c15fa68c-09a2-453a-81c1-687a3ce2b87a" (UID: "c15fa68c-09a2-453a-81c1-687a3ce2b87a"). InnerVolumeSpecName "kube-api-access-zxjc6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:49:31.746848 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:49:31.746830    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c15fa68c-09a2-453a-81c1-687a3ce2b87a-istio-envoy" pod "c15fa68c-09a2-453a-81c1-687a3ce2b87a" (UID: "c15fa68c-09a2-453a-81c1-687a3ce2b87a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:49:31.747664 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:49:31.747631    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c15fa68c-09a2-453a-81c1-687a3ce2b87a-istio-podinfo" pod "c15fa68c-09a2-453a-81c1-687a3ce2b87a" (UID: "c15fa68c-09a2-453a-81c1-687a3ce2b87a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 01:49:31.748471 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:49:31.748447    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c15fa68c-09a2-453a-81c1-687a3ce2b87a-istio-token" pod "c15fa68c-09a2-453a-81c1-687a3ce2b87a" (UID: "c15fa68c-09a2-453a-81c1-687a3ce2b87a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:49:31.846784 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:49:31.846703    4964 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c15fa68c-09a2-453a-81c1-687a3ce2b87a-istio-token\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:49:31.846784 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:49:31.846733    4964 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c15fa68c-09a2-453a-81c1-687a3ce2b87a-istio-envoy\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:49:31.846784 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:49:31.846747    4964 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c15fa68c-09a2-453a-81c1-687a3ce2b87a-istio-podinfo\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:49:31.846784 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:49:31.846761    4964 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zxjc6\" (UniqueName: \"kubernetes.io/projected/c15fa68c-09a2-453a-81c1-687a3ce2b87a-kube-api-access-zxjc6\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:49:31.847093 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:49:31.846938    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:49:32.023556 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:49:32.023395    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:49:32.037895 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:49:32.037865    4964 generic.go:356] "Generic (PLEG): container finished" podID="c15fa68c-09a2-453a-81c1-687a3ce2b87a" containerID="2ed0bd1cfed13a4fab558bde12f03065ff30db31105d26f8092c7fdb2935252b" exitCode=0
Oct 31 01:49:32.038068 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:49:32.037920    4964 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-klw8t"
Oct 31 01:49:32.038068 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:49:32.037948    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-klw8t" event={"ID":"c15fa68c-09a2-453a-81c1-687a3ce2b87a","Type":"ContainerDied","Data":"2ed0bd1cfed13a4fab558bde12f03065ff30db31105d26f8092c7fdb2935252b"}
Oct 31 01:49:32.038068 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:49:32.038017    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-klw8t" event={"ID":"c15fa68c-09a2-453a-81c1-687a3ce2b87a","Type":"ContainerDied","Data":"ab45a736141df472f2e05c78d578cd7a8c95c71386fc02ac51abaee6d39949c0"}
Oct 31 01:49:32.038068 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:49:32.038048    4964 scope.go:122] "RemoveContainer" containerID="2ed0bd1cfed13a4fab558bde12f03065ff30db31105d26f8092c7fdb2935252b"
Oct 31 01:49:32.049274 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:49:32.049259    4964 scope.go:122] "RemoveContainer" containerID="2ed0bd1cfed13a4fab558bde12f03065ff30db31105d26f8092c7fdb2935252b"
Oct 31 01:49:32.049625 ip-10-0-133-197 kubenswrapper[4964]: E1031 01:49:32.049602    4964 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2ed0bd1cfed13a4fab558bde12f03065ff30db31105d26f8092c7fdb2935252b\": container with ID starting with 2ed0bd1cfed13a4fab558bde12f03065ff30db31105d26f8092c7fdb2935252b not found: ID does not exist" containerID="2ed0bd1cfed13a4fab558bde12f03065ff30db31105d26f8092c7fdb2935252b"
Oct 31 01:49:32.049722 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:49:32.049632    4964 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2ed0bd1cfed13a4fab558bde12f03065ff30db31105d26f8092c7fdb2935252b"} err="failed to get container status \"2ed0bd1cfed13a4fab558bde12f03065ff30db31105d26f8092c7fdb2935252b\": rpc error: code = NotFound desc = could not find container \"2ed0bd1cfed13a4fab558bde12f03065ff30db31105d26f8092c7fdb2935252b\": container with ID starting with 2ed0bd1cfed13a4fab558bde12f03065ff30db31105d26f8092c7fdb2935252b not found: ID does not exist"
Oct 31 01:49:32.203271 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:49:32.203236    4964 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-klw8t"]
Oct 31 01:49:32.210071 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:49:32.210043    4964 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-klw8t"]
Oct 31 01:49:33.778196 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:49:33.778166    4964 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c15fa68c-09a2-453a-81c1-687a3ce2b87a" path="/var/lib/kubelet/pods/c15fa68c-09a2-453a-81c1-687a3ce2b87a/volumes"
Oct 31 01:49:35.341170 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:49:35.341136    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:49:38.482768 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:49:38.482736    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:49:40.196460 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:49:40.196404    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:49:41.238666 ip-10-0-133-197 kubenswrapper[4964]: E1031 01:49:41.238626    4964 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q56x5" containerName="extractor"
Oct 31 01:49:45.187068 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:49:45.187031    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:49:50.298957 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:49:50.298924    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:49:53.704292 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:49:53.704250    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:49:53.875703 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:49:53.875664    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:49:56.696066 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:49:56.696025    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:50:01.422614 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:50:01.422566    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:50:03.748369 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:50:03.748334    4964 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-8k9j8"]
Oct 31 01:50:03.748981 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:50:03.748968    4964 cpu_manager.go:414] "containerMap: removing container" podUID="c15fa68c-09a2-453a-81c1-687a3ce2b87a" containerName="istio-proxy"
Oct 31 01:50:03.749026 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:50:03.748982    4964 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c15fa68c-09a2-453a-81c1-687a3ce2b87a" containerName="istio-proxy"
Oct 31 01:50:03.749102 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:50:03.749093    4964 memory_manager.go:365] "RemoveStaleState removing state" podUID="c15fa68c-09a2-453a-81c1-687a3ce2b87a" containerName="istio-proxy"
Oct 31 01:50:03.753991 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:50:03.753974    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-8k9j8"
Oct 31 01:50:03.827859 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:50:03.827824    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6252303-98d9-43e9-b6ca-f9087cf79430-workload-socket\") pod \"e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-8k9j8\" (UID: \"c6252303-98d9-43e9-b6ca-f9087cf79430\") " pod="openshift-ingress/e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-8k9j8"
Oct 31 01:50:03.828010 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:50:03.827886    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s6558\" (UniqueName: \"kubernetes.io/projected/c6252303-98d9-43e9-b6ca-f9087cf79430-kube-api-access-s6558\") pod \"e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-8k9j8\" (UID: \"c6252303-98d9-43e9-b6ca-f9087cf79430\") " pod="openshift-ingress/e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-8k9j8"
Oct 31 01:50:03.828110 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:50:03.828089    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6252303-98d9-43e9-b6ca-f9087cf79430-credential-socket\") pod \"e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-8k9j8\" (UID: \"c6252303-98d9-43e9-b6ca-f9087cf79430\") " pod="openshift-ingress/e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-8k9j8"
Oct 31 01:50:03.828154 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:50:03.828135    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c6252303-98d9-43e9-b6ca-f9087cf79430-istiod-ca-cert\") pod \"e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-8k9j8\" (UID: \"c6252303-98d9-43e9-b6ca-f9087cf79430\") " pod="openshift-ingress/e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-8k9j8"
Oct 31 01:50:03.828207 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:50:03.828183    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c6252303-98d9-43e9-b6ca-f9087cf79430-workload-certs\") pod \"e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-8k9j8\" (UID: \"c6252303-98d9-43e9-b6ca-f9087cf79430\") " pod="openshift-ingress/e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-8k9j8"
Oct 31 01:50:03.828283 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:50:03.828263    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c6252303-98d9-43e9-b6ca-f9087cf79430-istio-data\") pod \"e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-8k9j8\" (UID: \"c6252303-98d9-43e9-b6ca-f9087cf79430\") " pod="openshift-ingress/e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-8k9j8"
Oct 31 01:50:03.828420 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:50:03.828399    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c6252303-98d9-43e9-b6ca-f9087cf79430-istio-envoy\") pod \"e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-8k9j8\" (UID: \"c6252303-98d9-43e9-b6ca-f9087cf79430\") " pod="openshift-ingress/e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-8k9j8"
Oct 31 01:50:03.828536 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:50:03.828520    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c6252303-98d9-43e9-b6ca-f9087cf79430-istio-podinfo\") pod \"e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-8k9j8\" (UID: \"c6252303-98d9-43e9-b6ca-f9087cf79430\") " pod="openshift-ingress/e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-8k9j8"
Oct 31 01:50:03.828625 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:50:03.828609    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c6252303-98d9-43e9-b6ca-f9087cf79430-istio-token\") pod \"e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-8k9j8\" (UID: \"c6252303-98d9-43e9-b6ca-f9087cf79430\") " pod="openshift-ingress/e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-8k9j8"
Oct 31 01:50:03.863217 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:50:03.863181    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-8k9j8"]
Oct 31 01:50:03.930280 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:50:03.930251    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c6252303-98d9-43e9-b6ca-f9087cf79430-istio-envoy\") pod \"e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-8k9j8\" (UID: \"c6252303-98d9-43e9-b6ca-f9087cf79430\") " pod="openshift-ingress/e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-8k9j8"
Oct 31 01:50:03.930444 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:50:03.930389    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c6252303-98d9-43e9-b6ca-f9087cf79430-istio-podinfo\") pod \"e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-8k9j8\" (UID: \"c6252303-98d9-43e9-b6ca-f9087cf79430\") " pod="openshift-ingress/e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-8k9j8"
Oct 31 01:50:03.930489 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:50:03.930452    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c6252303-98d9-43e9-b6ca-f9087cf79430-istio-token\") pod \"e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-8k9j8\" (UID: \"c6252303-98d9-43e9-b6ca-f9087cf79430\") " pod="openshift-ingress/e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-8k9j8"
Oct 31 01:50:03.930541 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:50:03.930524    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6252303-98d9-43e9-b6ca-f9087cf79430-workload-socket\") pod \"e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-8k9j8\" (UID: \"c6252303-98d9-43e9-b6ca-f9087cf79430\") " pod="openshift-ingress/e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-8k9j8"
Oct 31 01:50:03.930587 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:50:03.930571    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s6558\" (UniqueName: \"kubernetes.io/projected/c6252303-98d9-43e9-b6ca-f9087cf79430-kube-api-access-s6558\") pod \"e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-8k9j8\" (UID: \"c6252303-98d9-43e9-b6ca-f9087cf79430\") " pod="openshift-ingress/e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-8k9j8"
Oct 31 01:50:03.930672 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:50:03.930651    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6252303-98d9-43e9-b6ca-f9087cf79430-credential-socket\") pod \"e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-8k9j8\" (UID: \"c6252303-98d9-43e9-b6ca-f9087cf79430\") " pod="openshift-ingress/e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-8k9j8"
Oct 31 01:50:03.930823 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:50:03.930682    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c6252303-98d9-43e9-b6ca-f9087cf79430-istiod-ca-cert\") pod \"e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-8k9j8\" (UID: \"c6252303-98d9-43e9-b6ca-f9087cf79430\") " pod="openshift-ingress/e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-8k9j8"
Oct 31 01:50:03.930823 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:50:03.930720    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c6252303-98d9-43e9-b6ca-f9087cf79430-workload-certs\") pod \"e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-8k9j8\" (UID: \"c6252303-98d9-43e9-b6ca-f9087cf79430\") " pod="openshift-ingress/e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-8k9j8"
Oct 31 01:50:03.930823 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:50:03.930752    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c6252303-98d9-43e9-b6ca-f9087cf79430-istio-data\") pod \"e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-8k9j8\" (UID: \"c6252303-98d9-43e9-b6ca-f9087cf79430\") " pod="openshift-ingress/e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-8k9j8"
Oct 31 01:50:03.930993 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:50:03.930913    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6252303-98d9-43e9-b6ca-f9087cf79430-workload-socket\") pod \"e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-8k9j8\" (UID: \"c6252303-98d9-43e9-b6ca-f9087cf79430\") " pod="openshift-ingress/e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-8k9j8"
Oct 31 01:50:03.930993 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:50:03.930973    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6252303-98d9-43e9-b6ca-f9087cf79430-credential-socket\") pod \"e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-8k9j8\" (UID: \"c6252303-98d9-43e9-b6ca-f9087cf79430\") " pod="openshift-ingress/e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-8k9j8"
Oct 31 01:50:03.931308 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:50:03.931285    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c6252303-98d9-43e9-b6ca-f9087cf79430-workload-certs\") pod \"e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-8k9j8\" (UID: \"c6252303-98d9-43e9-b6ca-f9087cf79430\") " pod="openshift-ingress/e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-8k9j8"
Oct 31 01:50:03.931459 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:50:03.931378    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c6252303-98d9-43e9-b6ca-f9087cf79430-istio-data\") pod \"e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-8k9j8\" (UID: \"c6252303-98d9-43e9-b6ca-f9087cf79430\") " pod="openshift-ingress/e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-8k9j8"
Oct 31 01:50:03.931732 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:50:03.931708    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c6252303-98d9-43e9-b6ca-f9087cf79430-istiod-ca-cert\") pod \"e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-8k9j8\" (UID: \"c6252303-98d9-43e9-b6ca-f9087cf79430\") " pod="openshift-ingress/e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-8k9j8"
Oct 31 01:50:03.932894 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:50:03.932870    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c6252303-98d9-43e9-b6ca-f9087cf79430-istio-envoy\") pod \"e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-8k9j8\" (UID: \"c6252303-98d9-43e9-b6ca-f9087cf79430\") " pod="openshift-ingress/e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-8k9j8"
Oct 31 01:50:03.933097 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:50:03.933082    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c6252303-98d9-43e9-b6ca-f9087cf79430-istio-podinfo\") pod \"e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-8k9j8\" (UID: \"c6252303-98d9-43e9-b6ca-f9087cf79430\") " pod="openshift-ingress/e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-8k9j8"
Oct 31 01:50:03.988983 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:50:03.988919    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c6252303-98d9-43e9-b6ca-f9087cf79430-istio-token\") pod \"e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-8k9j8\" (UID: \"c6252303-98d9-43e9-b6ca-f9087cf79430\") " pod="openshift-ingress/e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-8k9j8"
Oct 31 01:50:03.994076 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:50:03.994055    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s6558\" (UniqueName: \"kubernetes.io/projected/c6252303-98d9-43e9-b6ca-f9087cf79430-kube-api-access-s6558\") pod \"e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-8k9j8\" (UID: \"c6252303-98d9-43e9-b6ca-f9087cf79430\") " pod="openshift-ingress/e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-8k9j8"
Oct 31 01:50:04.067284 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:50:04.067212    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-8k9j8"
Oct 31 01:50:04.504026 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:50:04.503998    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-8k9j8"]
Oct 31 01:50:04.504783 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:50:04.504749    4964 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6252303_98d9_43e9_b6ca_f9087cf79430.slice/crio-5c52d9fd41dba24fc6e0a949177ee1f075fbf4032994aa97c41380bb29b38738 WatchSource:0}: Error finding container 5c52d9fd41dba24fc6e0a949177ee1f075fbf4032994aa97c41380bb29b38738: Status 404 returned error can't find the container with id 5c52d9fd41dba24fc6e0a949177ee1f075fbf4032994aa97c41380bb29b38738
Oct 31 01:50:04.507394 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:50:04.507365    4964 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 01:50:04.507463 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:50:04.507429    4964 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 01:50:04.507463 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:50:04.507457    4964 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 01:50:05.288062 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:50:05.288027    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-8k9j8" event={"ID":"c6252303-98d9-43e9-b6ca-f9087cf79430","Type":"ContainerStarted","Data":"a26ab811e35fc62fd8e9068f88b29a50a27e22a65c1974c198be545af1f4f515"}
Oct 31 01:50:05.288532 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:50:05.288108    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-8k9j8" event={"ID":"c6252303-98d9-43e9-b6ca-f9087cf79430","Type":"ContainerStarted","Data":"5c52d9fd41dba24fc6e0a949177ee1f075fbf4032994aa97c41380bb29b38738"}
Oct 31 01:50:05.455359 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:50:05.455288    4964 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-8k9j8" podStartSLOduration=2.455272533 podStartE2EDuration="2.455272533s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:50:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:50:05.454516633 +0000 UTC m=+2773.931083027" watchObservedRunningTime="2026-10-31 01:50:05.455272533 +0000 UTC m=+2773.931838913"
Oct 31 01:50:06.068339 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:50:06.068287    4964 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-8k9j8"
Oct 31 01:50:06.069959 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:50:06.069936    4964 patch_prober.go:29] interesting pod/e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-8k9j8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused" start-of-body=
Oct 31 01:50:06.070045 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:50:06.069993    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-8k9j8" podUID="c6252303-98d9-43e9-b6ca-f9087cf79430" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused"
Oct 31 01:50:06.230016 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:50:06.229976    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:50:07.067734 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:50:07.067699    4964 patch_prober.go:29] interesting pod/e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-8k9j8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused" start-of-body=
Oct 31 01:50:07.068137 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:50:07.067761    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-8k9j8" podUID="c6252303-98d9-43e9-b6ca-f9087cf79430" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused"
Oct 31 01:50:08.067910 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:50:08.067876    4964 patch_prober.go:29] interesting pod/e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-8k9j8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused" start-of-body=
Oct 31 01:50:08.068306 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:50:08.067931    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-8k9j8" podUID="c6252303-98d9-43e9-b6ca-f9087cf79430" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused"
Oct 31 01:50:09.067856 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:50:09.067815    4964 patch_prober.go:29] interesting pod/e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-8k9j8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused" start-of-body=
Oct 31 01:50:09.068072 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:50:09.067868    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-8k9j8" podUID="c6252303-98d9-43e9-b6ca-f9087cf79430" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused"
Oct 31 01:50:10.067889 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:50:10.067849    4964 patch_prober.go:29] interesting pod/e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-8k9j8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused" start-of-body=
Oct 31 01:50:10.068070 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:50:10.067907    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-8k9j8" podUID="c6252303-98d9-43e9-b6ca-f9087cf79430" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused"
Oct 31 01:50:11.072640 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:50:11.072604    4964 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-8k9j8"
Oct 31 01:50:11.073172 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:50:11.073130    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-8k9j8"
Oct 31 01:50:11.074112 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:50:11.074093    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-8k9j8"
Oct 31 01:50:11.112500 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:50:11.112464    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:50:11.239197 ip-10-0-133-197 kubenswrapper[4964]: E1031 01:50:11.239158    4964 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q56x5" containerName="extractor"
Oct 31 01:50:15.919936 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:50:15.919899    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:50:20.548221 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:50:20.548101    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:50:21.014052 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:50:21.014017    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:50:23.877183 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:50:23.877147    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:50:25.382659 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:50:25.382624    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:50:26.536763 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:50:26.536702    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:50:26.554984 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:50:26.554959    4964 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-71bc98-openshift-default-84786b6cdc-lstqn"]
Oct 31 01:50:26.555258 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:50:26.555218    4964 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w1-71bc98-openshift-default-84786b6cdc-lstqn" podUID="c4a07c54-1e5b-4aef-b912-dcbc3930ae43" containerName="istio-proxy" containerID="cri-o://07df99b8ce7681f561f854d5f7b7b5276418b4d71813ef86a1675e153712b383" gracePeriod=30
Oct 31 01:50:26.904267 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:50:26.904175    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:50:27.222717 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:50:27.222677    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:50:27.565685 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:50:27.565599    4964 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-53af08-8495bd4fc6-dxcsr"]
Oct 31 01:50:27.566148 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:50:27.565849    4964 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-53af08-8495bd4fc6-dxcsr" podUID="ceae1723-877f-4b19-9574-da266b18535f" containerName="maas-api" containerID="cri-o://7aef7c90ec66b5af747f3858074459e5a844db36139ba9c675c2cce7af7d6376" gracePeriod=30
Oct 31 01:50:27.704473 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:50:27.704422    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:50:27.861055 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:50:27.861027    4964 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-53af08-8495bd4fc6-dxcsr"
Oct 31 01:50:27.990177 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:50:27.990149    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ceae1723-877f-4b19-9574-da266b18535f-metrics-certs\" (UniqueName: \"kubernetes.io/secret/ceae1723-877f-4b19-9574-da266b18535f-metrics-certs\") pod \"ceae1723-877f-4b19-9574-da266b18535f\" (UID: \"ceae1723-877f-4b19-9574-da266b18535f\") "
Oct 31 01:50:27.990389 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:50:27.990223    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ceae1723-877f-4b19-9574-da266b18535f-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ceae1723-877f-4b19-9574-da266b18535f-maas-api-tls\") pod \"ceae1723-877f-4b19-9574-da266b18535f\" (UID: \"ceae1723-877f-4b19-9574-da266b18535f\") "
Oct 31 01:50:27.990389 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:50:27.990258    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ceae1723-877f-4b19-9574-da266b18535f-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ceae1723-877f-4b19-9574-da266b18535f-openshift-service-ca\") pod \"ceae1723-877f-4b19-9574-da266b18535f\" (UID: \"ceae1723-877f-4b19-9574-da266b18535f\") "
Oct 31 01:50:27.990389 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:50:27.990358    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ceae1723-877f-4b19-9574-da266b18535f-kube-api-access-8j4hx\" (UniqueName: \"kubernetes.io/projected/ceae1723-877f-4b19-9574-da266b18535f-kube-api-access-8j4hx\") pod \"ceae1723-877f-4b19-9574-da266b18535f\" (UID: \"ceae1723-877f-4b19-9574-da266b18535f\") "
Oct 31 01:50:27.991249 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:50:27.991219    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ceae1723-877f-4b19-9574-da266b18535f-openshift-service-ca" pod "ceae1723-877f-4b19-9574-da266b18535f" (UID: "ceae1723-877f-4b19-9574-da266b18535f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:50:27.992691 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:50:27.992661    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ceae1723-877f-4b19-9574-da266b18535f-metrics-certs" pod "ceae1723-877f-4b19-9574-da266b18535f" (UID: "ceae1723-877f-4b19-9574-da266b18535f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:50:27.993267 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:50:27.993242    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ceae1723-877f-4b19-9574-da266b18535f-maas-api-tls" pod "ceae1723-877f-4b19-9574-da266b18535f" (UID: "ceae1723-877f-4b19-9574-da266b18535f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:50:27.993404 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:50:27.993366    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ceae1723-877f-4b19-9574-da266b18535f-kube-api-access-8j4hx" pod "ceae1723-877f-4b19-9574-da266b18535f" (UID: "ceae1723-877f-4b19-9574-da266b18535f"). InnerVolumeSpecName "kube-api-access-8j4hx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:50:28.092488 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:50:28.092413    4964 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ceae1723-877f-4b19-9574-da266b18535f-maas-api-tls\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:50:28.092488 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:50:28.092448    4964 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ceae1723-877f-4b19-9574-da266b18535f-openshift-service-ca\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:50:28.092488 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:50:28.092462    4964 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8j4hx\" (UniqueName: \"kubernetes.io/projected/ceae1723-877f-4b19-9574-da266b18535f-kube-api-access-8j4hx\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:50:28.092488 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:50:28.092477    4964 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ceae1723-877f-4b19-9574-da266b18535f-metrics-certs\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:50:28.114997 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:50:28.114964    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:50:28.340902 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:50:28.340873    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:50:28.486938 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:50:28.486906    4964 generic.go:356] "Generic (PLEG): container finished" podID="ceae1723-877f-4b19-9574-da266b18535f" containerID="7aef7c90ec66b5af747f3858074459e5a844db36139ba9c675c2cce7af7d6376" exitCode=0
Oct 31 01:50:28.487132 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:50:28.486964    4964 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-53af08-8495bd4fc6-dxcsr"
Oct 31 01:50:28.487132 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:50:28.486996    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-53af08-8495bd4fc6-dxcsr" event={"ID":"ceae1723-877f-4b19-9574-da266b18535f","Type":"ContainerDied","Data":"7aef7c90ec66b5af747f3858074459e5a844db36139ba9c675c2cce7af7d6376"}
Oct 31 01:50:28.487132 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:50:28.487058    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-53af08-8495bd4fc6-dxcsr" event={"ID":"ceae1723-877f-4b19-9574-da266b18535f","Type":"ContainerDied","Data":"0fa9d9c768e8a9d373c248bb8314634000679b10329d1dafc0058a1a9ce87306"}
Oct 31 01:50:28.487132 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:50:28.487085    4964 scope.go:122] "RemoveContainer" containerID="7aef7c90ec66b5af747f3858074459e5a844db36139ba9c675c2cce7af7d6376"
Oct 31 01:50:28.498393 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:50:28.498377    4964 scope.go:122] "RemoveContainer" containerID="f91a86d76b374a8b023b07afd8539bdb8b71d26715e16cc4654291328e50e2ea"
Oct 31 01:50:28.507785 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:50:28.507766    4964 scope.go:122] "RemoveContainer" containerID="7aef7c90ec66b5af747f3858074459e5a844db36139ba9c675c2cce7af7d6376"
Oct 31 01:50:28.508043 ip-10-0-133-197 kubenswrapper[4964]: E1031 01:50:28.508022    4964 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7aef7c90ec66b5af747f3858074459e5a844db36139ba9c675c2cce7af7d6376\": container with ID starting with 7aef7c90ec66b5af747f3858074459e5a844db36139ba9c675c2cce7af7d6376 not found: ID does not exist" containerID="7aef7c90ec66b5af747f3858074459e5a844db36139ba9c675c2cce7af7d6376"
Oct 31 01:50:28.508130 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:50:28.508053    4964 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7aef7c90ec66b5af747f3858074459e5a844db36139ba9c675c2cce7af7d6376"} err="failed to get container status \"7aef7c90ec66b5af747f3858074459e5a844db36139ba9c675c2cce7af7d6376\": rpc error: code = NotFound desc = could not find container \"7aef7c90ec66b5af747f3858074459e5a844db36139ba9c675c2cce7af7d6376\": container with ID starting with 7aef7c90ec66b5af747f3858074459e5a844db36139ba9c675c2cce7af7d6376 not found: ID does not exist"
Oct 31 01:50:28.508130 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:50:28.508079    4964 scope.go:122] "RemoveContainer" containerID="f91a86d76b374a8b023b07afd8539bdb8b71d26715e16cc4654291328e50e2ea"
Oct 31 01:50:28.508360 ip-10-0-133-197 kubenswrapper[4964]: E1031 01:50:28.508336    4964 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f91a86d76b374a8b023b07afd8539bdb8b71d26715e16cc4654291328e50e2ea\": container with ID starting with f91a86d76b374a8b023b07afd8539bdb8b71d26715e16cc4654291328e50e2ea not found: ID does not exist" containerID="f91a86d76b374a8b023b07afd8539bdb8b71d26715e16cc4654291328e50e2ea"
Oct 31 01:50:28.508473 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:50:28.508368    4964 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f91a86d76b374a8b023b07afd8539bdb8b71d26715e16cc4654291328e50e2ea"} err="failed to get container status \"f91a86d76b374a8b023b07afd8539bdb8b71d26715e16cc4654291328e50e2ea\": rpc error: code = NotFound desc = could not find container \"f91a86d76b374a8b023b07afd8539bdb8b71d26715e16cc4654291328e50e2ea\": container with ID starting with f91a86d76b374a8b023b07afd8539bdb8b71d26715e16cc4654291328e50e2ea not found: ID does not exist"
Oct 31 01:50:28.514274 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:50:28.514250    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:50:28.669489 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:50:28.669457    4964 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-53af08-8495bd4fc6-dxcsr"]
Oct 31 01:50:28.673468 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:50:28.673443    4964 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-53af08-8495bd4fc6-dxcsr"]
Oct 31 01:50:29.105526 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:50:29.105483    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:50:29.481004 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:50:29.480975    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:50:29.667362 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:50:29.667334    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:50:29.780134 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:50:29.780056    4964 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ceae1723-877f-4b19-9574-da266b18535f" path="/var/lib/kubelet/pods/ceae1723-877f-4b19-9574-da266b18535f/volumes"
Oct 31 01:50:29.847590 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:50:29.847556    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:50:30.462602 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:50:30.462567    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:50:31.821291 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:50:31.821235    4964 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-71bc98-openshift-default-84786b6cdc-lstqn"
Oct 31 01:50:31.950398 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:50:31.950372    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c4a07c54-1e5b-4aef-b912-dcbc3930ae43-kube-api-access-d5ftv\" (UniqueName: \"kubernetes.io/projected/c4a07c54-1e5b-4aef-b912-dcbc3930ae43-kube-api-access-d5ftv\") pod \"c4a07c54-1e5b-4aef-b912-dcbc3930ae43\" (UID: \"c4a07c54-1e5b-4aef-b912-dcbc3930ae43\") "
Oct 31 01:50:31.950582 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:50:31.950414    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c4a07c54-1e5b-4aef-b912-dcbc3930ae43-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c4a07c54-1e5b-4aef-b912-dcbc3930ae43-istio-envoy\") pod \"c4a07c54-1e5b-4aef-b912-dcbc3930ae43\" (UID: \"c4a07c54-1e5b-4aef-b912-dcbc3930ae43\") "
Oct 31 01:50:31.950582 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:50:31.950444    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c4a07c54-1e5b-4aef-b912-dcbc3930ae43-istio-token\" (UniqueName: \"kubernetes.io/projected/c4a07c54-1e5b-4aef-b912-dcbc3930ae43-istio-token\") pod \"c4a07c54-1e5b-4aef-b912-dcbc3930ae43\" (UID: \"c4a07c54-1e5b-4aef-b912-dcbc3930ae43\") "
Oct 31 01:50:31.950582 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:50:31.950500    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c4a07c54-1e5b-4aef-b912-dcbc3930ae43-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4a07c54-1e5b-4aef-b912-dcbc3930ae43-credential-socket\") pod \"c4a07c54-1e5b-4aef-b912-dcbc3930ae43\" (UID: \"c4a07c54-1e5b-4aef-b912-dcbc3930ae43\") "
Oct 31 01:50:31.950759 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:50:31.950593    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c4a07c54-1e5b-4aef-b912-dcbc3930ae43-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c4a07c54-1e5b-4aef-b912-dcbc3930ae43-istiod-ca-cert\") pod \"c4a07c54-1e5b-4aef-b912-dcbc3930ae43\" (UID: \"c4a07c54-1e5b-4aef-b912-dcbc3930ae43\") "
Oct 31 01:50:31.950759 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:50:31.950661    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c4a07c54-1e5b-4aef-b912-dcbc3930ae43-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c4a07c54-1e5b-4aef-b912-dcbc3930ae43-istio-podinfo\") pod \"c4a07c54-1e5b-4aef-b912-dcbc3930ae43\" (UID: \"c4a07c54-1e5b-4aef-b912-dcbc3930ae43\") "
Oct 31 01:50:31.950759 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:50:31.950699    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c4a07c54-1e5b-4aef-b912-dcbc3930ae43-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4a07c54-1e5b-4aef-b912-dcbc3930ae43-workload-socket\") pod \"c4a07c54-1e5b-4aef-b912-dcbc3930ae43\" (UID: \"c4a07c54-1e5b-4aef-b912-dcbc3930ae43\") "
Oct 31 01:50:31.950759 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:50:31.950756    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c4a07c54-1e5b-4aef-b912-dcbc3930ae43-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c4a07c54-1e5b-4aef-b912-dcbc3930ae43-istio-data\") pod \"c4a07c54-1e5b-4aef-b912-dcbc3930ae43\" (UID: \"c4a07c54-1e5b-4aef-b912-dcbc3930ae43\") "
Oct 31 01:50:31.950985 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:50:31.950821    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4a07c54-1e5b-4aef-b912-dcbc3930ae43-credential-socket" pod "c4a07c54-1e5b-4aef-b912-dcbc3930ae43" (UID: "c4a07c54-1e5b-4aef-b912-dcbc3930ae43"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:50:31.950985 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:50:31.950837    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c4a07c54-1e5b-4aef-b912-dcbc3930ae43-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c4a07c54-1e5b-4aef-b912-dcbc3930ae43-workload-certs\") pod \"c4a07c54-1e5b-4aef-b912-dcbc3930ae43\" (UID: \"c4a07c54-1e5b-4aef-b912-dcbc3930ae43\") "
Oct 31 01:50:31.951105 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:50:31.951035    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4a07c54-1e5b-4aef-b912-dcbc3930ae43-istio-data" pod "c4a07c54-1e5b-4aef-b912-dcbc3930ae43" (UID: "c4a07c54-1e5b-4aef-b912-dcbc3930ae43"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:50:31.951105 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:50:31.951045    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4a07c54-1e5b-4aef-b912-dcbc3930ae43-workload-socket" pod "c4a07c54-1e5b-4aef-b912-dcbc3930ae43" (UID: "c4a07c54-1e5b-4aef-b912-dcbc3930ae43"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:50:31.951105 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:50:31.951057    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4a07c54-1e5b-4aef-b912-dcbc3930ae43-workload-certs" pod "c4a07c54-1e5b-4aef-b912-dcbc3930ae43" (UID: "c4a07c54-1e5b-4aef-b912-dcbc3930ae43"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:50:31.951453 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:50:31.951425    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c4a07c54-1e5b-4aef-b912-dcbc3930ae43-istiod-ca-cert" pod "c4a07c54-1e5b-4aef-b912-dcbc3930ae43" (UID: "c4a07c54-1e5b-4aef-b912-dcbc3930ae43"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:50:31.951729 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:50:31.951710    4964 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4a07c54-1e5b-4aef-b912-dcbc3930ae43-credential-socket\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:50:31.951809 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:50:31.951737    4964 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c4a07c54-1e5b-4aef-b912-dcbc3930ae43-istiod-ca-cert\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:50:31.951809 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:50:31.951752    4964 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4a07c54-1e5b-4aef-b912-dcbc3930ae43-workload-socket\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:50:31.951809 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:50:31.951768    4964 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c4a07c54-1e5b-4aef-b912-dcbc3930ae43-istio-data\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:50:31.951809 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:50:31.951781    4964 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c4a07c54-1e5b-4aef-b912-dcbc3930ae43-workload-certs\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:50:31.952904 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:50:31.952880    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4a07c54-1e5b-4aef-b912-dcbc3930ae43-istio-envoy" pod "c4a07c54-1e5b-4aef-b912-dcbc3930ae43" (UID: "c4a07c54-1e5b-4aef-b912-dcbc3930ae43"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:50:31.953013 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:50:31.952968    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c4a07c54-1e5b-4aef-b912-dcbc3930ae43-istio-token" pod "c4a07c54-1e5b-4aef-b912-dcbc3930ae43" (UID: "c4a07c54-1e5b-4aef-b912-dcbc3930ae43"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:50:31.953562 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:50:31.953540    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c4a07c54-1e5b-4aef-b912-dcbc3930ae43-istio-podinfo" pod "c4a07c54-1e5b-4aef-b912-dcbc3930ae43" (UID: "c4a07c54-1e5b-4aef-b912-dcbc3930ae43"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 01:50:31.953639 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:50:31.953566    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c4a07c54-1e5b-4aef-b912-dcbc3930ae43-kube-api-access-d5ftv" pod "c4a07c54-1e5b-4aef-b912-dcbc3930ae43" (UID: "c4a07c54-1e5b-4aef-b912-dcbc3930ae43"). InnerVolumeSpecName "kube-api-access-d5ftv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:50:32.053588 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:50:32.053555    4964 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c4a07c54-1e5b-4aef-b912-dcbc3930ae43-istio-podinfo\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:50:32.053588 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:50:32.053584    4964 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d5ftv\" (UniqueName: \"kubernetes.io/projected/c4a07c54-1e5b-4aef-b912-dcbc3930ae43-kube-api-access-d5ftv\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:50:32.053588 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:50:32.053595    4964 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c4a07c54-1e5b-4aef-b912-dcbc3930ae43-istio-envoy\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:50:32.053831 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:50:32.053605    4964 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c4a07c54-1e5b-4aef-b912-dcbc3930ae43-istio-token\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:50:32.517041 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:50:32.517002    4964 generic.go:356] "Generic (PLEG): container finished" podID="c4a07c54-1e5b-4aef-b912-dcbc3930ae43" containerID="07df99b8ce7681f561f854d5f7b7b5276418b4d71813ef86a1675e153712b383" exitCode=0
Oct 31 01:50:32.517216 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:50:32.517045    4964 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-71bc98-openshift-default-84786b6cdc-lstqn"
Oct 31 01:50:32.517216 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:50:32.517098    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-71bc98-openshift-default-84786b6cdc-lstqn" event={"ID":"c4a07c54-1e5b-4aef-b912-dcbc3930ae43","Type":"ContainerDied","Data":"07df99b8ce7681f561f854d5f7b7b5276418b4d71813ef86a1675e153712b383"}
Oct 31 01:50:32.517216 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:50:32.517152    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-71bc98-openshift-default-84786b6cdc-lstqn" event={"ID":"c4a07c54-1e5b-4aef-b912-dcbc3930ae43","Type":"ContainerDied","Data":"f7907db3eb97c7dc8c99beb190b38c3e13c8408b9eb0440eaf8cec22b732928d"}
Oct 31 01:50:32.517216 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:50:32.517176    4964 scope.go:122] "RemoveContainer" containerID="07df99b8ce7681f561f854d5f7b7b5276418b4d71813ef86a1675e153712b383"
Oct 31 01:50:32.533070 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:50:32.533053    4964 scope.go:122] "RemoveContainer" containerID="07df99b8ce7681f561f854d5f7b7b5276418b4d71813ef86a1675e153712b383"
Oct 31 01:50:32.533379 ip-10-0-133-197 kubenswrapper[4964]: E1031 01:50:32.533357    4964 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"07df99b8ce7681f561f854d5f7b7b5276418b4d71813ef86a1675e153712b383\": container with ID starting with 07df99b8ce7681f561f854d5f7b7b5276418b4d71813ef86a1675e153712b383 not found: ID does not exist" containerID="07df99b8ce7681f561f854d5f7b7b5276418b4d71813ef86a1675e153712b383"
Oct 31 01:50:32.533466 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:50:32.533389    4964 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"07df99b8ce7681f561f854d5f7b7b5276418b4d71813ef86a1675e153712b383"} err="failed to get container status \"07df99b8ce7681f561f854d5f7b7b5276418b4d71813ef86a1675e153712b383\": rpc error: code = NotFound desc = could not find container \"07df99b8ce7681f561f854d5f7b7b5276418b4d71813ef86a1675e153712b383\": container with ID starting with 07df99b8ce7681f561f854d5f7b7b5276418b4d71813ef86a1675e153712b383 not found: ID does not exist"
Oct 31 01:50:32.731499 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:50:32.731460    4964 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-71bc98-openshift-default-84786b6cdc-lstqn"]
Oct 31 01:50:32.735724 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:50:32.735700    4964 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-71bc98-openshift-default-84786b6cdc-lstqn"]
Oct 31 01:50:33.777476 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:50:33.777441    4964 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c4a07c54-1e5b-4aef-b912-dcbc3930ae43" path="/var/lib/kubelet/pods/c4a07c54-1e5b-4aef-b912-dcbc3930ae43/volumes"
Oct 31 01:50:34.761142 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:50:34.761100    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:50:35.020803 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:50:35.020726    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:50:35.136909 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:50:35.136877    4964 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-wgfd6"]
Oct 31 01:50:35.137100 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:50:35.137084    4964 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-wgfd6" podUID="7e48bcec-5ffa-4f2b-bb9f-f2fbdc1b3da9" containerName="istio-proxy" containerID="cri-o://61fa9ded14547620fe08064b5a186d18d0caa1c5aedbea191d0941170c3257ce" gracePeriod=30
Oct 31 01:50:35.143715 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:50:35.143687    4964 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-gg8ls"]
Oct 31 01:50:35.143895 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:50:35.143875    4964 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-gg8ls" podUID="dd00f3ae-bb65-4eb4-aada-cce1fab3b363" containerName="istio-proxy" containerID="cri-o://95128752e0c27828e733b6cee2800707f3397b21f024cb5d7899248c2043fea4" gracePeriod=30
Oct 31 01:50:35.211803 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:50:35.211768    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:50:35.497427 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:50:35.497397    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:50:38.007802 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:50:38.007754    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:50:39.710029 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:50:39.709984    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:50:39.862573 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:50:39.862535    4964 patch_prober.go:29] interesting pod/e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-wgfd6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 01:50:39.862740 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:50:39.862603    4964 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-wgfd6" podUID="7e48bcec-5ffa-4f2b-bb9f-f2fbdc1b3da9" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 01:50:40.099017 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:50:40.098982    4964 patch_prober.go:29] interesting pod/e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-gg8ls container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 01:50:40.099280 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:50:40.099249    4964 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-gg8ls" podUID="dd00f3ae-bb65-4eb4-aada-cce1fab3b363" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 01:50:40.445198 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:50:40.445164    4964 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-wgfd6"
Oct 31 01:50:40.457444 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:50:40.457416    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7e48bcec-5ffa-4f2b-bb9f-f2fbdc1b3da9-kube-api-access-ss4j6\" (UniqueName: \"kubernetes.io/projected/7e48bcec-5ffa-4f2b-bb9f-f2fbdc1b3da9-kube-api-access-ss4j6\") pod \"7e48bcec-5ffa-4f2b-bb9f-f2fbdc1b3da9\" (UID: \"7e48bcec-5ffa-4f2b-bb9f-f2fbdc1b3da9\") "
Oct 31 01:50:40.457605 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:50:40.457477    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7e48bcec-5ffa-4f2b-bb9f-f2fbdc1b3da9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7e48bcec-5ffa-4f2b-bb9f-f2fbdc1b3da9-istiod-ca-cert\") pod \"7e48bcec-5ffa-4f2b-bb9f-f2fbdc1b3da9\" (UID: \"7e48bcec-5ffa-4f2b-bb9f-f2fbdc1b3da9\") "
Oct 31 01:50:40.457605 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:50:40.457584    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7e48bcec-5ffa-4f2b-bb9f-f2fbdc1b3da9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7e48bcec-5ffa-4f2b-bb9f-f2fbdc1b3da9-istio-data\") pod \"7e48bcec-5ffa-4f2b-bb9f-f2fbdc1b3da9\" (UID: \"7e48bcec-5ffa-4f2b-bb9f-f2fbdc1b3da9\") "
Oct 31 01:50:40.457735 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:50:40.457671    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7e48bcec-5ffa-4f2b-bb9f-f2fbdc1b3da9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7e48bcec-5ffa-4f2b-bb9f-f2fbdc1b3da9-istio-podinfo\") pod \"7e48bcec-5ffa-4f2b-bb9f-f2fbdc1b3da9\" (UID: \"7e48bcec-5ffa-4f2b-bb9f-f2fbdc1b3da9\") "
Oct 31 01:50:40.457735 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:50:40.457700    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7e48bcec-5ffa-4f2b-bb9f-f2fbdc1b3da9-istio-token\" (UniqueName: \"kubernetes.io/projected/7e48bcec-5ffa-4f2b-bb9f-f2fbdc1b3da9-istio-token\") pod \"7e48bcec-5ffa-4f2b-bb9f-f2fbdc1b3da9\" (UID: \"7e48bcec-5ffa-4f2b-bb9f-f2fbdc1b3da9\") "
Oct 31 01:50:40.457856 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:50:40.457746    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7e48bcec-5ffa-4f2b-bb9f-f2fbdc1b3da9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7e48bcec-5ffa-4f2b-bb9f-f2fbdc1b3da9-istio-envoy\") pod \"7e48bcec-5ffa-4f2b-bb9f-f2fbdc1b3da9\" (UID: \"7e48bcec-5ffa-4f2b-bb9f-f2fbdc1b3da9\") "
Oct 31 01:50:40.457914 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:50:40.457865    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7e48bcec-5ffa-4f2b-bb9f-f2fbdc1b3da9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7e48bcec-5ffa-4f2b-bb9f-f2fbdc1b3da9-workload-certs\") pod \"7e48bcec-5ffa-4f2b-bb9f-f2fbdc1b3da9\" (UID: \"7e48bcec-5ffa-4f2b-bb9f-f2fbdc1b3da9\") "
Oct 31 01:50:40.457972 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:50:40.457911    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7e48bcec-5ffa-4f2b-bb9f-f2fbdc1b3da9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e48bcec-5ffa-4f2b-bb9f-f2fbdc1b3da9-credential-socket\") pod \"7e48bcec-5ffa-4f2b-bb9f-f2fbdc1b3da9\" (UID: \"7e48bcec-5ffa-4f2b-bb9f-f2fbdc1b3da9\") "
Oct 31 01:50:40.457972 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:50:40.457924    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e48bcec-5ffa-4f2b-bb9f-f2fbdc1b3da9-istio-data" pod "7e48bcec-5ffa-4f2b-bb9f-f2fbdc1b3da9" (UID: "7e48bcec-5ffa-4f2b-bb9f-f2fbdc1b3da9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:50:40.458102 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:50:40.457980    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7e48bcec-5ffa-4f2b-bb9f-f2fbdc1b3da9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e48bcec-5ffa-4f2b-bb9f-f2fbdc1b3da9-workload-socket\") pod \"7e48bcec-5ffa-4f2b-bb9f-f2fbdc1b3da9\" (UID: \"7e48bcec-5ffa-4f2b-bb9f-f2fbdc1b3da9\") "
Oct 31 01:50:40.458172 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:50:40.458099    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7e48bcec-5ffa-4f2b-bb9f-f2fbdc1b3da9-istiod-ca-cert" pod "7e48bcec-5ffa-4f2b-bb9f-f2fbdc1b3da9" (UID: "7e48bcec-5ffa-4f2b-bb9f-f2fbdc1b3da9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:50:40.458483 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:50:40.458453    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e48bcec-5ffa-4f2b-bb9f-f2fbdc1b3da9-workload-socket" pod "7e48bcec-5ffa-4f2b-bb9f-f2fbdc1b3da9" (UID: "7e48bcec-5ffa-4f2b-bb9f-f2fbdc1b3da9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:50:40.458677 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:50:40.458657    4964 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e48bcec-5ffa-4f2b-bb9f-f2fbdc1b3da9-workload-socket\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:50:40.458720 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:50:40.458681    4964 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7e48bcec-5ffa-4f2b-bb9f-f2fbdc1b3da9-istiod-ca-cert\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:50:40.458720 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:50:40.458696    4964 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7e48bcec-5ffa-4f2b-bb9f-f2fbdc1b3da9-istio-data\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:50:40.459110 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:50:40.459046    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e48bcec-5ffa-4f2b-bb9f-f2fbdc1b3da9-credential-socket" pod "7e48bcec-5ffa-4f2b-bb9f-f2fbdc1b3da9" (UID: "7e48bcec-5ffa-4f2b-bb9f-f2fbdc1b3da9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:50:40.459110 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:50:40.459131    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e48bcec-5ffa-4f2b-bb9f-f2fbdc1b3da9-workload-certs" pod "7e48bcec-5ffa-4f2b-bb9f-f2fbdc1b3da9" (UID: "7e48bcec-5ffa-4f2b-bb9f-f2fbdc1b3da9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:50:40.460934 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:50:40.460904    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e48bcec-5ffa-4f2b-bb9f-f2fbdc1b3da9-istio-token" pod "7e48bcec-5ffa-4f2b-bb9f-f2fbdc1b3da9" (UID: "7e48bcec-5ffa-4f2b-bb9f-f2fbdc1b3da9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:50:40.460934 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:50:40.460927    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e48bcec-5ffa-4f2b-bb9f-f2fbdc1b3da9-kube-api-access-ss4j6" pod "7e48bcec-5ffa-4f2b-bb9f-f2fbdc1b3da9" (UID: "7e48bcec-5ffa-4f2b-bb9f-f2fbdc1b3da9"). InnerVolumeSpecName "kube-api-access-ss4j6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:50:40.461178 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:50:40.461154    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7e48bcec-5ffa-4f2b-bb9f-f2fbdc1b3da9-istio-podinfo" pod "7e48bcec-5ffa-4f2b-bb9f-f2fbdc1b3da9" (UID: "7e48bcec-5ffa-4f2b-bb9f-f2fbdc1b3da9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 01:50:40.461855 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:50:40.461832    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e48bcec-5ffa-4f2b-bb9f-f2fbdc1b3da9-istio-envoy" pod "7e48bcec-5ffa-4f2b-bb9f-f2fbdc1b3da9" (UID: "7e48bcec-5ffa-4f2b-bb9f-f2fbdc1b3da9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:50:40.485450 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:50:40.485425    4964 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-gg8ls"
Oct 31 01:50:40.559619 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:50:40.559590    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dd00f3ae-bb65-4eb4-aada-cce1fab3b363-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dd00f3ae-bb65-4eb4-aada-cce1fab3b363-istiod-ca-cert\") pod \"dd00f3ae-bb65-4eb4-aada-cce1fab3b363\" (UID: \"dd00f3ae-bb65-4eb4-aada-cce1fab3b363\") "
Oct 31 01:50:40.559778 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:50:40.559672    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dd00f3ae-bb65-4eb4-aada-cce1fab3b363-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd00f3ae-bb65-4eb4-aada-cce1fab3b363-workload-socket\") pod \"dd00f3ae-bb65-4eb4-aada-cce1fab3b363\" (UID: \"dd00f3ae-bb65-4eb4-aada-cce1fab3b363\") "
Oct 31 01:50:40.559778 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:50:40.559699    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dd00f3ae-bb65-4eb4-aada-cce1fab3b363-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dd00f3ae-bb65-4eb4-aada-cce1fab3b363-istio-envoy\") pod \"dd00f3ae-bb65-4eb4-aada-cce1fab3b363\" (UID: \"dd00f3ae-bb65-4eb4-aada-cce1fab3b363\") "
Oct 31 01:50:40.559893 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:50:40.559787    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/dd00f3ae-bb65-4eb4-aada-cce1fab3b363-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dd00f3ae-bb65-4eb4-aada-cce1fab3b363-istio-podinfo\") pod \"dd00f3ae-bb65-4eb4-aada-cce1fab3b363\" (UID: \"dd00f3ae-bb65-4eb4-aada-cce1fab3b363\") "
Oct 31 01:50:40.559893 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:50:40.559818    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dd00f3ae-bb65-4eb4-aada-cce1fab3b363-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dd00f3ae-bb65-4eb4-aada-cce1fab3b363-workload-certs\") pod \"dd00f3ae-bb65-4eb4-aada-cce1fab3b363\" (UID: \"dd00f3ae-bb65-4eb4-aada-cce1fab3b363\") "
Oct 31 01:50:40.559893 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:50:40.559835    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dd00f3ae-bb65-4eb4-aada-cce1fab3b363-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd00f3ae-bb65-4eb4-aada-cce1fab3b363-credential-socket\") pod \"dd00f3ae-bb65-4eb4-aada-cce1fab3b363\" (UID: \"dd00f3ae-bb65-4eb4-aada-cce1fab3b363\") "
Oct 31 01:50:40.560071 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:50:40.559890    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dd00f3ae-bb65-4eb4-aada-cce1fab3b363-istio-token\" (UniqueName: \"kubernetes.io/projected/dd00f3ae-bb65-4eb4-aada-cce1fab3b363-istio-token\") pod \"dd00f3ae-bb65-4eb4-aada-cce1fab3b363\" (UID: \"dd00f3ae-bb65-4eb4-aada-cce1fab3b363\") "
Oct 31 01:50:40.560071 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:50:40.559916    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dd00f3ae-bb65-4eb4-aada-cce1fab3b363-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dd00f3ae-bb65-4eb4-aada-cce1fab3b363-istio-data\") pod \"dd00f3ae-bb65-4eb4-aada-cce1fab3b363\" (UID: \"dd00f3ae-bb65-4eb4-aada-cce1fab3b363\") "
Oct 31 01:50:40.560071 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:50:40.559949    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dd00f3ae-bb65-4eb4-aada-cce1fab3b363-kube-api-access-8rl2w\" (UniqueName: \"kubernetes.io/projected/dd00f3ae-bb65-4eb4-aada-cce1fab3b363-kube-api-access-8rl2w\") pod \"dd00f3ae-bb65-4eb4-aada-cce1fab3b363\" (UID: \"dd00f3ae-bb65-4eb4-aada-cce1fab3b363\") "
Oct 31 01:50:40.560071 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:50:40.559940    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd00f3ae-bb65-4eb4-aada-cce1fab3b363-workload-socket" pod "dd00f3ae-bb65-4eb4-aada-cce1fab3b363" (UID: "dd00f3ae-bb65-4eb4-aada-cce1fab3b363"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:50:40.560280 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:50:40.560115    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd00f3ae-bb65-4eb4-aada-cce1fab3b363-workload-certs" pod "dd00f3ae-bb65-4eb4-aada-cce1fab3b363" (UID: "dd00f3ae-bb65-4eb4-aada-cce1fab3b363"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:50:40.560280 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:50:40.560014    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dd00f3ae-bb65-4eb4-aada-cce1fab3b363-istiod-ca-cert" pod "dd00f3ae-bb65-4eb4-aada-cce1fab3b363" (UID: "dd00f3ae-bb65-4eb4-aada-cce1fab3b363"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:50:40.560280 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:50:40.560195    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd00f3ae-bb65-4eb4-aada-cce1fab3b363-credential-socket" pod "dd00f3ae-bb65-4eb4-aada-cce1fab3b363" (UID: "dd00f3ae-bb65-4eb4-aada-cce1fab3b363"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:50:40.560458 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:50:40.560308    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd00f3ae-bb65-4eb4-aada-cce1fab3b363-istio-data" pod "dd00f3ae-bb65-4eb4-aada-cce1fab3b363" (UID: "dd00f3ae-bb65-4eb4-aada-cce1fab3b363"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:50:40.560799 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:50:40.560775    4964 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7e48bcec-5ffa-4f2b-bb9f-f2fbdc1b3da9-istio-podinfo\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:50:40.560894 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:50:40.560804    4964 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7e48bcec-5ffa-4f2b-bb9f-f2fbdc1b3da9-istio-token\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:50:40.560894 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:50:40.560821    4964 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7e48bcec-5ffa-4f2b-bb9f-f2fbdc1b3da9-istio-envoy\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:50:40.560894 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:50:40.560836    4964 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dd00f3ae-bb65-4eb4-aada-cce1fab3b363-workload-certs\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:50:40.560894 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:50:40.560850    4964 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd00f3ae-bb65-4eb4-aada-cce1fab3b363-credential-socket\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:50:40.560894 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:50:40.560865    4964 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7e48bcec-5ffa-4f2b-bb9f-f2fbdc1b3da9-workload-certs\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:50:40.560894 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:50:40.560880    4964 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dd00f3ae-bb65-4eb4-aada-cce1fab3b363-istio-data\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:50:40.560894 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:50:40.560894    4964 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e48bcec-5ffa-4f2b-bb9f-f2fbdc1b3da9-credential-socket\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:50:40.561267 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:50:40.560909    4964 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ss4j6\" (UniqueName: \"kubernetes.io/projected/7e48bcec-5ffa-4f2b-bb9f-f2fbdc1b3da9-kube-api-access-ss4j6\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:50:40.561267 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:50:40.560925    4964 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dd00f3ae-bb65-4eb4-aada-cce1fab3b363-istiod-ca-cert\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:50:40.561267 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:50:40.560941    4964 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd00f3ae-bb65-4eb4-aada-cce1fab3b363-workload-socket\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:50:40.562279 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:50:40.562250    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd00f3ae-bb65-4eb4-aada-cce1fab3b363-kube-api-access-8rl2w" pod "dd00f3ae-bb65-4eb4-aada-cce1fab3b363" (UID: "dd00f3ae-bb65-4eb4-aada-cce1fab3b363"). InnerVolumeSpecName "kube-api-access-8rl2w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:50:40.562279 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:50:40.562257    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/dd00f3ae-bb65-4eb4-aada-cce1fab3b363-istio-podinfo" pod "dd00f3ae-bb65-4eb4-aada-cce1fab3b363" (UID: "dd00f3ae-bb65-4eb4-aada-cce1fab3b363"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 01:50:40.562447 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:50:40.562283    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd00f3ae-bb65-4eb4-aada-cce1fab3b363-istio-token" pod "dd00f3ae-bb65-4eb4-aada-cce1fab3b363" (UID: "dd00f3ae-bb65-4eb4-aada-cce1fab3b363"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:50:40.562447 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:50:40.562311    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd00f3ae-bb65-4eb4-aada-cce1fab3b363-istio-envoy" pod "dd00f3ae-bb65-4eb4-aada-cce1fab3b363" (UID: "dd00f3ae-bb65-4eb4-aada-cce1fab3b363"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:50:40.588088 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:50:40.588053    4964 generic.go:356] "Generic (PLEG): container finished" podID="7e48bcec-5ffa-4f2b-bb9f-f2fbdc1b3da9" containerID="61fa9ded14547620fe08064b5a186d18d0caa1c5aedbea191d0941170c3257ce" exitCode=0
Oct 31 01:50:40.588236 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:50:40.588108    4964 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-wgfd6"
Oct 31 01:50:40.588236 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:50:40.588177    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-wgfd6" event={"ID":"7e48bcec-5ffa-4f2b-bb9f-f2fbdc1b3da9","Type":"ContainerDied","Data":"61fa9ded14547620fe08064b5a186d18d0caa1c5aedbea191d0941170c3257ce"}
Oct 31 01:50:40.588357 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:50:40.588246    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-wgfd6" event={"ID":"7e48bcec-5ffa-4f2b-bb9f-f2fbdc1b3da9","Type":"ContainerDied","Data":"5f700999b29241a845fd934736e4a3491717328555fe88feee7be736b54f69ea"}
Oct 31 01:50:40.588357 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:50:40.588279    4964 scope.go:122] "RemoveContainer" containerID="61fa9ded14547620fe08064b5a186d18d0caa1c5aedbea191d0941170c3257ce"
Oct 31 01:50:40.590234 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:50:40.590206    4964 generic.go:356] "Generic (PLEG): container finished" podID="dd00f3ae-bb65-4eb4-aada-cce1fab3b363" containerID="95128752e0c27828e733b6cee2800707f3397b21f024cb5d7899248c2043fea4" exitCode=0
Oct 31 01:50:40.590383 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:50:40.590246    4964 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-gg8ls"
Oct 31 01:50:40.590383 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:50:40.590270    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-gg8ls" event={"ID":"dd00f3ae-bb65-4eb4-aada-cce1fab3b363","Type":"ContainerDied","Data":"95128752e0c27828e733b6cee2800707f3397b21f024cb5d7899248c2043fea4"}
Oct 31 01:50:40.590503 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:50:40.590308    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-gg8ls" event={"ID":"dd00f3ae-bb65-4eb4-aada-cce1fab3b363","Type":"ContainerDied","Data":"f19d2779eb7084a6653a3ebe2dffb7da61b6b821ee22a78451ff5eac51327b14"}
Oct 31 01:50:40.603466 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:50:40.603445    4964 scope.go:122] "RemoveContainer" containerID="61fa9ded14547620fe08064b5a186d18d0caa1c5aedbea191d0941170c3257ce"
Oct 31 01:50:40.603844 ip-10-0-133-197 kubenswrapper[4964]: E1031 01:50:40.603815    4964 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"61fa9ded14547620fe08064b5a186d18d0caa1c5aedbea191d0941170c3257ce\": container with ID starting with 61fa9ded14547620fe08064b5a186d18d0caa1c5aedbea191d0941170c3257ce not found: ID does not exist" containerID="61fa9ded14547620fe08064b5a186d18d0caa1c5aedbea191d0941170c3257ce"
Oct 31 01:50:40.603929 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:50:40.603856    4964 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"61fa9ded14547620fe08064b5a186d18d0caa1c5aedbea191d0941170c3257ce"} err="failed to get container status \"61fa9ded14547620fe08064b5a186d18d0caa1c5aedbea191d0941170c3257ce\": rpc error: code = NotFound desc = could not find container \"61fa9ded14547620fe08064b5a186d18d0caa1c5aedbea191d0941170c3257ce\": container with ID starting with 61fa9ded14547620fe08064b5a186d18d0caa1c5aedbea191d0941170c3257ce not found: ID does not exist"
Oct 31 01:50:40.603929 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:50:40.603879    4964 scope.go:122] "RemoveContainer" containerID="95128752e0c27828e733b6cee2800707f3397b21f024cb5d7899248c2043fea4"
Oct 31 01:50:40.616448 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:50:40.616423    4964 scope.go:122] "RemoveContainer" containerID="95128752e0c27828e733b6cee2800707f3397b21f024cb5d7899248c2043fea4"
Oct 31 01:50:40.620920 ip-10-0-133-197 kubenswrapper[4964]: E1031 01:50:40.618247    4964 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"95128752e0c27828e733b6cee2800707f3397b21f024cb5d7899248c2043fea4\": container with ID starting with 95128752e0c27828e733b6cee2800707f3397b21f024cb5d7899248c2043fea4 not found: ID does not exist" containerID="95128752e0c27828e733b6cee2800707f3397b21f024cb5d7899248c2043fea4"
Oct 31 01:50:40.620920 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:50:40.618279    4964 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"95128752e0c27828e733b6cee2800707f3397b21f024cb5d7899248c2043fea4"} err="failed to get container status \"95128752e0c27828e733b6cee2800707f3397b21f024cb5d7899248c2043fea4\": rpc error: code = NotFound desc = could not find container \"95128752e0c27828e733b6cee2800707f3397b21f024cb5d7899248c2043fea4\": container with ID starting with 95128752e0c27828e733b6cee2800707f3397b21f024cb5d7899248c2043fea4 not found: ID does not exist"
Oct 31 01:50:40.662776 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:50:40.662700    4964 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dd00f3ae-bb65-4eb4-aada-cce1fab3b363-istio-envoy\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:50:40.662776 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:50:40.662726    4964 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dd00f3ae-bb65-4eb4-aada-cce1fab3b363-istio-podinfo\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:50:40.662776 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:50:40.662736    4964 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dd00f3ae-bb65-4eb4-aada-cce1fab3b363-istio-token\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:50:40.662776 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:50:40.662748    4964 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8rl2w\" (UniqueName: \"kubernetes.io/projected/dd00f3ae-bb65-4eb4-aada-cce1fab3b363-kube-api-access-8rl2w\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:50:40.790556 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:50:40.790523    4964 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-gg8ls"]
Oct 31 01:50:40.797488 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:50:40.797462    4964 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-gg8ls"]
Oct 31 01:50:40.957721 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:50:40.957641    4964 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-wgfd6"]
Oct 31 01:50:40.962097 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:50:40.962068    4964 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-wgfd6"]
Oct 31 01:50:41.094903 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:50:41.094862    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:50:41.241240 ip-10-0-133-197 kubenswrapper[4964]: E1031 01:50:41.241181    4964 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q56x5" containerName="extractor"
Oct 31 01:50:41.778145 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:50:41.778114    4964 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7e48bcec-5ffa-4f2b-bb9f-f2fbdc1b3da9" path="/var/lib/kubelet/pods/7e48bcec-5ffa-4f2b-bb9f-f2fbdc1b3da9/volumes"
Oct 31 01:50:41.778591 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:50:41.778577    4964 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dd00f3ae-bb65-4eb4-aada-cce1fab3b363" path="/var/lib/kubelet/pods/dd00f3ae-bb65-4eb4-aada-cce1fab3b363/volumes"
Oct 31 01:50:43.016744 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:50:43.016692    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:50:43.689280 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:50:43.689246    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:50:43.866480 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:50:43.866446    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:50:44.013353 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:50:44.013297    4964 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-8a3796-openshift-default-857c657d9-8vlxd"]
Oct 31 01:50:44.013607 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:50:44.013579    4964 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-a-8a3796-openshift-default-857c657d9-8vlxd" podUID="11d36add-45f3-4bf4-9372-7d7cec791eff" containerName="istio-proxy" containerID="cri-o://7a4a5bfb6732af6a7fb4dd00f1f8e2a9b9b59611cfebc5aabfe6feb8442880bd" gracePeriod=30
Oct 31 01:50:44.058470 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:50:44.058436    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:50:44.972919 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:50:44.972882    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:50:49.304733 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:50:49.304699    4964 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-8a3796-openshift-default-857c657d9-8vlxd"
Oct 31 01:50:49.390534 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:50:49.390500    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/11d36add-45f3-4bf4-9372-7d7cec791eff-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/11d36add-45f3-4bf4-9372-7d7cec791eff-istio-data\") pod \"11d36add-45f3-4bf4-9372-7d7cec791eff\" (UID: \"11d36add-45f3-4bf4-9372-7d7cec791eff\") "
Oct 31 01:50:49.390697 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:50:49.390574    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/11d36add-45f3-4bf4-9372-7d7cec791eff-istio-token\" (UniqueName: \"kubernetes.io/projected/11d36add-45f3-4bf4-9372-7d7cec791eff-istio-token\") pod \"11d36add-45f3-4bf4-9372-7d7cec791eff\" (UID: \"11d36add-45f3-4bf4-9372-7d7cec791eff\") "
Oct 31 01:50:49.390697 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:50:49.390612    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/11d36add-45f3-4bf4-9372-7d7cec791eff-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/11d36add-45f3-4bf4-9372-7d7cec791eff-workload-certs\") pod \"11d36add-45f3-4bf4-9372-7d7cec791eff\" (UID: \"11d36add-45f3-4bf4-9372-7d7cec791eff\") "
Oct 31 01:50:49.390697 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:50:49.390673    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/11d36add-45f3-4bf4-9372-7d7cec791eff-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/11d36add-45f3-4bf4-9372-7d7cec791eff-istiod-ca-cert\") pod \"11d36add-45f3-4bf4-9372-7d7cec791eff\" (UID: \"11d36add-45f3-4bf4-9372-7d7cec791eff\") "
Oct 31 01:50:49.390697 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:50:49.390695    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/11d36add-45f3-4bf4-9372-7d7cec791eff-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/11d36add-45f3-4bf4-9372-7d7cec791eff-workload-socket\") pod \"11d36add-45f3-4bf4-9372-7d7cec791eff\" (UID: \"11d36add-45f3-4bf4-9372-7d7cec791eff\") "
Oct 31 01:50:49.390925 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:50:49.390726    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/11d36add-45f3-4bf4-9372-7d7cec791eff-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/11d36add-45f3-4bf4-9372-7d7cec791eff-istio-envoy\") pod \"11d36add-45f3-4bf4-9372-7d7cec791eff\" (UID: \"11d36add-45f3-4bf4-9372-7d7cec791eff\") "
Oct 31 01:50:49.390925 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:50:49.390750    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/11d36add-45f3-4bf4-9372-7d7cec791eff-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/11d36add-45f3-4bf4-9372-7d7cec791eff-istio-podinfo\") pod \"11d36add-45f3-4bf4-9372-7d7cec791eff\" (UID: \"11d36add-45f3-4bf4-9372-7d7cec791eff\") "
Oct 31 01:50:49.390925 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:50:49.390828    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/11d36add-45f3-4bf4-9372-7d7cec791eff-kube-api-access-9td2s\" (UniqueName: \"kubernetes.io/projected/11d36add-45f3-4bf4-9372-7d7cec791eff-kube-api-access-9td2s\") pod \"11d36add-45f3-4bf4-9372-7d7cec791eff\" (UID: \"11d36add-45f3-4bf4-9372-7d7cec791eff\") "
Oct 31 01:50:49.390925 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:50:49.390831    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11d36add-45f3-4bf4-9372-7d7cec791eff-istio-data" pod "11d36add-45f3-4bf4-9372-7d7cec791eff" (UID: "11d36add-45f3-4bf4-9372-7d7cec791eff"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:50:49.391136 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:50:49.390969    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/11d36add-45f3-4bf4-9372-7d7cec791eff-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/11d36add-45f3-4bf4-9372-7d7cec791eff-credential-socket\") pod \"11d36add-45f3-4bf4-9372-7d7cec791eff\" (UID: \"11d36add-45f3-4bf4-9372-7d7cec791eff\") "
Oct 31 01:50:49.391136 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:50:49.391057    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11d36add-45f3-4bf4-9372-7d7cec791eff-workload-certs" pod "11d36add-45f3-4bf4-9372-7d7cec791eff" (UID: "11d36add-45f3-4bf4-9372-7d7cec791eff"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:50:49.391136 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:50:49.391072    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11d36add-45f3-4bf4-9372-7d7cec791eff-workload-socket" pod "11d36add-45f3-4bf4-9372-7d7cec791eff" (UID: "11d36add-45f3-4bf4-9372-7d7cec791eff"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:50:49.391466 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:50:49.391268    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/11d36add-45f3-4bf4-9372-7d7cec791eff-istiod-ca-cert" pod "11d36add-45f3-4bf4-9372-7d7cec791eff" (UID: "11d36add-45f3-4bf4-9372-7d7cec791eff"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:50:49.391580 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:50:49.391442    4964 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/11d36add-45f3-4bf4-9372-7d7cec791eff-istio-data\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:50:49.391580 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:50:49.391558    4964 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/11d36add-45f3-4bf4-9372-7d7cec791eff-workload-certs\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:50:49.391580 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:50:49.391568    4964 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/11d36add-45f3-4bf4-9372-7d7cec791eff-workload-socket\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:50:49.391580 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:50:49.391383    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11d36add-45f3-4bf4-9372-7d7cec791eff-credential-socket" pod "11d36add-45f3-4bf4-9372-7d7cec791eff" (UID: "11d36add-45f3-4bf4-9372-7d7cec791eff"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:50:49.393156 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:50:49.393131    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/11d36add-45f3-4bf4-9372-7d7cec791eff-istio-podinfo" pod "11d36add-45f3-4bf4-9372-7d7cec791eff" (UID: "11d36add-45f3-4bf4-9372-7d7cec791eff"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 01:50:49.393256 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:50:49.393168    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11d36add-45f3-4bf4-9372-7d7cec791eff-istio-token" pod "11d36add-45f3-4bf4-9372-7d7cec791eff" (UID: "11d36add-45f3-4bf4-9372-7d7cec791eff"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:50:49.393352 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:50:49.393293    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11d36add-45f3-4bf4-9372-7d7cec791eff-istio-envoy" pod "11d36add-45f3-4bf4-9372-7d7cec791eff" (UID: "11d36add-45f3-4bf4-9372-7d7cec791eff"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:50:49.393601 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:50:49.393586    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11d36add-45f3-4bf4-9372-7d7cec791eff-kube-api-access-9td2s" pod "11d36add-45f3-4bf4-9372-7d7cec791eff" (UID: "11d36add-45f3-4bf4-9372-7d7cec791eff"). InnerVolumeSpecName "kube-api-access-9td2s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:50:49.492952 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:50:49.492920    4964 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9td2s\" (UniqueName: \"kubernetes.io/projected/11d36add-45f3-4bf4-9372-7d7cec791eff-kube-api-access-9td2s\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:50:49.492952 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:50:49.492950    4964 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/11d36add-45f3-4bf4-9372-7d7cec791eff-credential-socket\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:50:49.492952 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:50:49.492961    4964 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/11d36add-45f3-4bf4-9372-7d7cec791eff-istio-token\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:50:49.493193 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:50:49.492970    4964 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/11d36add-45f3-4bf4-9372-7d7cec791eff-istiod-ca-cert\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:50:49.493193 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:50:49.492979    4964 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/11d36add-45f3-4bf4-9372-7d7cec791eff-istio-envoy\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:50:49.493193 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:50:49.492988    4964 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/11d36add-45f3-4bf4-9372-7d7cec791eff-istio-podinfo\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:50:49.665268 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:50:49.665175    4964 generic.go:356] "Generic (PLEG): container finished" podID="11d36add-45f3-4bf4-9372-7d7cec791eff" containerID="7a4a5bfb6732af6a7fb4dd00f1f8e2a9b9b59611cfebc5aabfe6feb8442880bd" exitCode=0
Oct 31 01:50:49.665268 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:50:49.665224    4964 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-8a3796-openshift-default-857c657d9-8vlxd"
Oct 31 01:50:49.665268 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:50:49.665258    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-8a3796-openshift-default-857c657d9-8vlxd" event={"ID":"11d36add-45f3-4bf4-9372-7d7cec791eff","Type":"ContainerDied","Data":"7a4a5bfb6732af6a7fb4dd00f1f8e2a9b9b59611cfebc5aabfe6feb8442880bd"}
Oct 31 01:50:49.665537 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:50:49.665312    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-8a3796-openshift-default-857c657d9-8vlxd" event={"ID":"11d36add-45f3-4bf4-9372-7d7cec791eff","Type":"ContainerDied","Data":"1a1945691ffbeea5e4716dbba07dd72da41b79ee8b88832f8ddc352764de53df"}
Oct 31 01:50:49.665537 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:50:49.665369    4964 scope.go:122] "RemoveContainer" containerID="7a4a5bfb6732af6a7fb4dd00f1f8e2a9b9b59611cfebc5aabfe6feb8442880bd"
Oct 31 01:50:49.676433 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:50:49.676406    4964 scope.go:122] "RemoveContainer" containerID="c91c1fe8e115676b727f5825d66641c22ba6c0d04d6c035d7d8bcbafb620504e"
Oct 31 01:50:49.686928 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:50:49.686906    4964 scope.go:122] "RemoveContainer" containerID="7a4a5bfb6732af6a7fb4dd00f1f8e2a9b9b59611cfebc5aabfe6feb8442880bd"
Oct 31 01:50:49.687226 ip-10-0-133-197 kubenswrapper[4964]: E1031 01:50:49.687202    4964 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7a4a5bfb6732af6a7fb4dd00f1f8e2a9b9b59611cfebc5aabfe6feb8442880bd\": container with ID starting with 7a4a5bfb6732af6a7fb4dd00f1f8e2a9b9b59611cfebc5aabfe6feb8442880bd not found: ID does not exist" containerID="7a4a5bfb6732af6a7fb4dd00f1f8e2a9b9b59611cfebc5aabfe6feb8442880bd"
Oct 31 01:50:49.687300 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:50:49.687234    4964 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7a4a5bfb6732af6a7fb4dd00f1f8e2a9b9b59611cfebc5aabfe6feb8442880bd"} err="failed to get container status \"7a4a5bfb6732af6a7fb4dd00f1f8e2a9b9b59611cfebc5aabfe6feb8442880bd\": rpc error: code = NotFound desc = could not find container \"7a4a5bfb6732af6a7fb4dd00f1f8e2a9b9b59611cfebc5aabfe6feb8442880bd\": container with ID starting with 7a4a5bfb6732af6a7fb4dd00f1f8e2a9b9b59611cfebc5aabfe6feb8442880bd not found: ID does not exist"
Oct 31 01:50:49.687300 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:50:49.687253    4964 scope.go:122] "RemoveContainer" containerID="c91c1fe8e115676b727f5825d66641c22ba6c0d04d6c035d7d8bcbafb620504e"
Oct 31 01:50:49.687578 ip-10-0-133-197 kubenswrapper[4964]: E1031 01:50:49.687558    4964 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c91c1fe8e115676b727f5825d66641c22ba6c0d04d6c035d7d8bcbafb620504e\": container with ID starting with c91c1fe8e115676b727f5825d66641c22ba6c0d04d6c035d7d8bcbafb620504e not found: ID does not exist" containerID="c91c1fe8e115676b727f5825d66641c22ba6c0d04d6c035d7d8bcbafb620504e"
Oct 31 01:50:49.687629 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:50:49.687595    4964 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c91c1fe8e115676b727f5825d66641c22ba6c0d04d6c035d7d8bcbafb620504e"} err="failed to get container status \"c91c1fe8e115676b727f5825d66641c22ba6c0d04d6c035d7d8bcbafb620504e\": rpc error: code = NotFound desc = could not find container \"c91c1fe8e115676b727f5825d66641c22ba6c0d04d6c035d7d8bcbafb620504e\": container with ID starting with c91c1fe8e115676b727f5825d66641c22ba6c0d04d6c035d7d8bcbafb620504e not found: ID does not exist"
Oct 31 01:50:49.828701 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:50:49.828659    4964 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-8a3796-openshift-default-857c657d9-8vlxd"]
Oct 31 01:50:49.833616 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:50:49.833585    4964 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-a-8a3796-openshift-default-857c657d9-8vlxd"]
Oct 31 01:50:50.131658 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:50:50.131616    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:50:51.779250 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:50:51.779218    4964 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="11d36add-45f3-4bf4-9372-7d7cec791eff" path="/var/lib/kubelet/pods/11d36add-45f3-4bf4-9372-7d7cec791eff/volumes"
Oct 31 01:50:53.055898 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:50:53.055810    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:50:54.887469 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:50:54.887424    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:50:55.178260 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:50:55.178168    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:50:57.963347 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:50:57.963297    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:50:58.209404 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:50:58.209370    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:50:59.954903 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:50:59.954866    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:51:01.196454 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:51:01.196424    4964 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-99145d-fc449dc4f-zmf85"]
Oct 31 01:51:01.301456 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:51:01.301425    4964 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-99145d-7d5667799-hfvwn"]
Oct 31 01:51:01.596411 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:51:01.596369    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:51:01.766912 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:51:01.766869    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:51:04.146697 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:51:04.146660    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:51:04.726034 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:51:04.725994    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:51:06.197402 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:51:06.197365    4964 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-b-99145d-fc449dc4f-zmf85" podUID="a3a77a68-3698-491d-8220-fa9a2c0f8278" containerName="payload-processing" containerID="cri-o://4066a0916f6a6c8d52cdc173dcd6f5fb5f80f2d2bda4078d1c03482f073bf829" gracePeriod=25
Oct 31 01:51:06.302162 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:51:06.302080    4964 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-99145d-7d5667799-hfvwn" podUID="7ca5c214-0290-4db5-9fb1-a9e27cdf9f58" containerName="payload-pre-processing" containerID="cri-o://d8b22b2b6d7dee80394e27d62ab8a83b4e31f4c2ba691aa3d7b6c1819f75ea0d" gracePeriod=25
Oct 31 01:51:06.461678 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:51:06.461613    4964 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-99145d-fc449dc4f-zmf85"
Oct 31 01:51:06.513361 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:51:06.513309    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a3a77a68-3698-491d-8220-fa9a2c0f8278-kube-api-access-lvvvg\" (UniqueName: \"kubernetes.io/projected/a3a77a68-3698-491d-8220-fa9a2c0f8278-kube-api-access-lvvvg\") pod \"a3a77a68-3698-491d-8220-fa9a2c0f8278\" (UID: \"a3a77a68-3698-491d-8220-fa9a2c0f8278\") "
Oct 31 01:51:06.513684 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:51:06.513661    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a3a77a68-3698-491d-8220-fa9a2c0f8278-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a3a77a68-3698-491d-8220-fa9a2c0f8278-plugins-config-volume\") pod \"a3a77a68-3698-491d-8220-fa9a2c0f8278\" (UID: \"a3a77a68-3698-491d-8220-fa9a2c0f8278\") "
Oct 31 01:51:06.514106 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:51:06.514078    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a3a77a68-3698-491d-8220-fa9a2c0f8278-plugins-config-volume" pod "a3a77a68-3698-491d-8220-fa9a2c0f8278" (UID: "a3a77a68-3698-491d-8220-fa9a2c0f8278"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:51:06.514492 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:51:06.514474    4964 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a3a77a68-3698-491d-8220-fa9a2c0f8278-plugins-config-volume\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:51:06.516136 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:51:06.516110    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a3a77a68-3698-491d-8220-fa9a2c0f8278-kube-api-access-lvvvg" pod "a3a77a68-3698-491d-8220-fa9a2c0f8278" (UID: "a3a77a68-3698-491d-8220-fa9a2c0f8278"). InnerVolumeSpecName "kube-api-access-lvvvg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:51:06.562182 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:51:06.562160    4964 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-99145d-7d5667799-hfvwn"
Oct 31 01:51:06.614844 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:51:06.614816    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7ca5c214-0290-4db5-9fb1-a9e27cdf9f58-kube-api-access-vf9fx\" (UniqueName: \"kubernetes.io/projected/7ca5c214-0290-4db5-9fb1-a9e27cdf9f58-kube-api-access-vf9fx\") pod \"7ca5c214-0290-4db5-9fb1-a9e27cdf9f58\" (UID: \"7ca5c214-0290-4db5-9fb1-a9e27cdf9f58\") "
Oct 31 01:51:06.615047 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:51:06.614872    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7ca5c214-0290-4db5-9fb1-a9e27cdf9f58-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7ca5c214-0290-4db5-9fb1-a9e27cdf9f58-plugins-config-volume\") pod \"7ca5c214-0290-4db5-9fb1-a9e27cdf9f58\" (UID: \"7ca5c214-0290-4db5-9fb1-a9e27cdf9f58\") "
Oct 31 01:51:06.615343 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:51:06.615296    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7ca5c214-0290-4db5-9fb1-a9e27cdf9f58-plugins-config-volume" pod "7ca5c214-0290-4db5-9fb1-a9e27cdf9f58" (UID: "7ca5c214-0290-4db5-9fb1-a9e27cdf9f58"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:51:06.615523 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:51:06.615505    4964 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lvvvg\" (UniqueName: \"kubernetes.io/projected/a3a77a68-3698-491d-8220-fa9a2c0f8278-kube-api-access-lvvvg\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:51:06.615605 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:51:06.615527    4964 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7ca5c214-0290-4db5-9fb1-a9e27cdf9f58-plugins-config-volume\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:51:06.617222 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:51:06.617205    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ca5c214-0290-4db5-9fb1-a9e27cdf9f58-kube-api-access-vf9fx" pod "7ca5c214-0290-4db5-9fb1-a9e27cdf9f58" (UID: "7ca5c214-0290-4db5-9fb1-a9e27cdf9f58"). InnerVolumeSpecName "kube-api-access-vf9fx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:51:06.717002 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:51:06.716929    4964 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vf9fx\" (UniqueName: \"kubernetes.io/projected/7ca5c214-0290-4db5-9fb1-a9e27cdf9f58-kube-api-access-vf9fx\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:51:06.788619 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:51:06.788584    4964 generic.go:356] "Generic (PLEG): container finished" podID="a3a77a68-3698-491d-8220-fa9a2c0f8278" containerID="4066a0916f6a6c8d52cdc173dcd6f5fb5f80f2d2bda4078d1c03482f073bf829" exitCode=0
Oct 31 01:51:06.788803 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:51:06.788629    4964 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-99145d-fc449dc4f-zmf85"
Oct 31 01:51:06.788803 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:51:06.788670    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-99145d-fc449dc4f-zmf85" event={"ID":"a3a77a68-3698-491d-8220-fa9a2c0f8278","Type":"ContainerDied","Data":"4066a0916f6a6c8d52cdc173dcd6f5fb5f80f2d2bda4078d1c03482f073bf829"}
Oct 31 01:51:06.788803 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:51:06.788722    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-99145d-fc449dc4f-zmf85" event={"ID":"a3a77a68-3698-491d-8220-fa9a2c0f8278","Type":"ContainerDied","Data":"5cda17dd4705e49ee8a012b9aa1ea71d7c73b162edcd83648afc105e06220278"}
Oct 31 01:51:06.788803 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:51:06.788751    4964 scope.go:122] "RemoveContainer" containerID="4066a0916f6a6c8d52cdc173dcd6f5fb5f80f2d2bda4078d1c03482f073bf829"
Oct 31 01:51:06.790547 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:51:06.790521    4964 generic.go:356] "Generic (PLEG): container finished" podID="7ca5c214-0290-4db5-9fb1-a9e27cdf9f58" containerID="d8b22b2b6d7dee80394e27d62ab8a83b4e31f4c2ba691aa3d7b6c1819f75ea0d" exitCode=0
Oct 31 01:51:06.790675 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:51:06.790569    4964 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-99145d-7d5667799-hfvwn"
Oct 31 01:51:06.790675 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:51:06.790600    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-99145d-7d5667799-hfvwn" event={"ID":"7ca5c214-0290-4db5-9fb1-a9e27cdf9f58","Type":"ContainerDied","Data":"d8b22b2b6d7dee80394e27d62ab8a83b4e31f4c2ba691aa3d7b6c1819f75ea0d"}
Oct 31 01:51:06.790675 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:51:06.790655    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-99145d-7d5667799-hfvwn" event={"ID":"7ca5c214-0290-4db5-9fb1-a9e27cdf9f58","Type":"ContainerDied","Data":"2bda8a820f6064be38527d8e253295151b3d9565f4ee9645c5908489bada2b55"}
Oct 31 01:51:06.801185 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:51:06.801165    4964 scope.go:122] "RemoveContainer" containerID="4066a0916f6a6c8d52cdc173dcd6f5fb5f80f2d2bda4078d1c03482f073bf829"
Oct 31 01:51:06.801445 ip-10-0-133-197 kubenswrapper[4964]: E1031 01:51:06.801426    4964 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4066a0916f6a6c8d52cdc173dcd6f5fb5f80f2d2bda4078d1c03482f073bf829\": container with ID starting with 4066a0916f6a6c8d52cdc173dcd6f5fb5f80f2d2bda4078d1c03482f073bf829 not found: ID does not exist" containerID="4066a0916f6a6c8d52cdc173dcd6f5fb5f80f2d2bda4078d1c03482f073bf829"
Oct 31 01:51:06.801548 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:51:06.801452    4964 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4066a0916f6a6c8d52cdc173dcd6f5fb5f80f2d2bda4078d1c03482f073bf829"} err="failed to get container status \"4066a0916f6a6c8d52cdc173dcd6f5fb5f80f2d2bda4078d1c03482f073bf829\": rpc error: code = NotFound desc = could not find container \"4066a0916f6a6c8d52cdc173dcd6f5fb5f80f2d2bda4078d1c03482f073bf829\": container with ID starting with 4066a0916f6a6c8d52cdc173dcd6f5fb5f80f2d2bda4078d1c03482f073bf829 not found: ID does not exist"
Oct 31 01:51:06.801548 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:51:06.801467    4964 scope.go:122] "RemoveContainer" containerID="d8b22b2b6d7dee80394e27d62ab8a83b4e31f4c2ba691aa3d7b6c1819f75ea0d"
Oct 31 01:51:06.810236 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:51:06.810213    4964 scope.go:122] "RemoveContainer" containerID="d8b22b2b6d7dee80394e27d62ab8a83b4e31f4c2ba691aa3d7b6c1819f75ea0d"
Oct 31 01:51:06.810497 ip-10-0-133-197 kubenswrapper[4964]: E1031 01:51:06.810478    4964 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d8b22b2b6d7dee80394e27d62ab8a83b4e31f4c2ba691aa3d7b6c1819f75ea0d\": container with ID starting with d8b22b2b6d7dee80394e27d62ab8a83b4e31f4c2ba691aa3d7b6c1819f75ea0d not found: ID does not exist" containerID="d8b22b2b6d7dee80394e27d62ab8a83b4e31f4c2ba691aa3d7b6c1819f75ea0d"
Oct 31 01:51:06.810548 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:51:06.810504    4964 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d8b22b2b6d7dee80394e27d62ab8a83b4e31f4c2ba691aa3d7b6c1819f75ea0d"} err="failed to get container status \"d8b22b2b6d7dee80394e27d62ab8a83b4e31f4c2ba691aa3d7b6c1819f75ea0d\": rpc error: code = NotFound desc = could not find container \"d8b22b2b6d7dee80394e27d62ab8a83b4e31f4c2ba691aa3d7b6c1819f75ea0d\": container with ID starting with d8b22b2b6d7dee80394e27d62ab8a83b4e31f4c2ba691aa3d7b6c1819f75ea0d not found: ID does not exist"
Oct 31 01:51:06.951788 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:51:06.951760    4964 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-99145d-7d5667799-hfvwn"]
Oct 31 01:51:06.954743 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:51:06.954722    4964 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-99145d-7d5667799-hfvwn"]
Oct 31 01:51:07.062535 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:51:07.062505    4964 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-99145d-fc449dc4f-zmf85"]
Oct 31 01:51:07.067909 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:51:07.067884    4964 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-99145d-fc449dc4f-zmf85"]
Oct 31 01:51:07.779708 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:51:07.779666    4964 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7ca5c214-0290-4db5-9fb1-a9e27cdf9f58" path="/var/lib/kubelet/pods/7ca5c214-0290-4db5-9fb1-a9e27cdf9f58/volumes"
Oct 31 01:51:07.780332 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:51:07.780295    4964 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a3a77a68-3698-491d-8220-fa9a2c0f8278" path="/var/lib/kubelet/pods/a3a77a68-3698-491d-8220-fa9a2c0f8278/volumes"
Oct 31 01:51:08.184808 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:51:08.184730    4964 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-9d7b2c-7548d86fb9-djhv5"]
Oct 31 01:51:08.291934 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:51:08.291900    4964 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-9d7b2c-b59bbf96c-7m86r"]
Oct 31 01:51:08.659625 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:51:08.659595    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:51:08.827790 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:51:08.827751    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:51:09.618193 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:51:09.618149    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:51:11.239368 ip-10-0-133-197 kubenswrapper[4964]: E1031 01:51:11.239328    4964 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q56x5" containerName="extractor"
Oct 31 01:51:11.572396 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:51:11.572305    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:51:12.532201 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:51:12.532166    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:51:13.185872 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:51:13.185834    4964 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-a-9d7b2c-7548d86fb9-djhv5" podUID="850c4489-7821-4e37-95f0-258336f5e75e" containerName="payload-processing" containerID="cri-o://87c35bdf46a217bbc1ffef1dea2e93e80da9e1f03f337948024539f3a9871c9f" gracePeriod=25
Oct 31 01:51:13.292858 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:51:13.292796    4964 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-9d7b2c-b59bbf96c-7m86r" podUID="655b500a-1f9c-424f-9bd2-376464e7151f" containerName="payload-pre-processing" containerID="cri-o://8937701057d857935308ef0e65ce7d3c66023a942e4535c3ae7a8805f71d5ce9" gracePeriod=25
Oct 31 01:51:13.479294 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:51:13.479261    4964 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-9d7b2c-7548d86fb9-djhv5"
Oct 31 01:51:13.565752 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:51:13.565730    4964 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-9d7b2c-b59bbf96c-7m86r"
Oct 31 01:51:13.603713 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:51:13.603687    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/850c4489-7821-4e37-95f0-258336f5e75e-kube-api-access-4bkfk\" (UniqueName: \"kubernetes.io/projected/850c4489-7821-4e37-95f0-258336f5e75e-kube-api-access-4bkfk\") pod \"850c4489-7821-4e37-95f0-258336f5e75e\" (UID: \"850c4489-7821-4e37-95f0-258336f5e75e\") "
Oct 31 01:51:13.603861 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:51:13.603768    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/850c4489-7821-4e37-95f0-258336f5e75e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/850c4489-7821-4e37-95f0-258336f5e75e-plugins-config-volume\") pod \"850c4489-7821-4e37-95f0-258336f5e75e\" (UID: \"850c4489-7821-4e37-95f0-258336f5e75e\") "
Oct 31 01:51:13.604438 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:51:13.604410    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/850c4489-7821-4e37-95f0-258336f5e75e-plugins-config-volume" pod "850c4489-7821-4e37-95f0-258336f5e75e" (UID: "850c4489-7821-4e37-95f0-258336f5e75e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:51:13.607845 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:51:13.606294    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/850c4489-7821-4e37-95f0-258336f5e75e-kube-api-access-4bkfk" pod "850c4489-7821-4e37-95f0-258336f5e75e" (UID: "850c4489-7821-4e37-95f0-258336f5e75e"). InnerVolumeSpecName "kube-api-access-4bkfk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:51:13.705581 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:51:13.705503    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/655b500a-1f9c-424f-9bd2-376464e7151f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/655b500a-1f9c-424f-9bd2-376464e7151f-plugins-config-volume\") pod \"655b500a-1f9c-424f-9bd2-376464e7151f\" (UID: \"655b500a-1f9c-424f-9bd2-376464e7151f\") "
Oct 31 01:51:13.705797 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:51:13.705755    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/655b500a-1f9c-424f-9bd2-376464e7151f-kube-api-access-zpw4x\" (UniqueName: \"kubernetes.io/projected/655b500a-1f9c-424f-9bd2-376464e7151f-kube-api-access-zpw4x\") pod \"655b500a-1f9c-424f-9bd2-376464e7151f\" (UID: \"655b500a-1f9c-424f-9bd2-376464e7151f\") "
Oct 31 01:51:13.706239 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:51:13.706211    4964 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4bkfk\" (UniqueName: \"kubernetes.io/projected/850c4489-7821-4e37-95f0-258336f5e75e-kube-api-access-4bkfk\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:51:13.706239 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:51:13.706238    4964 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/850c4489-7821-4e37-95f0-258336f5e75e-plugins-config-volume\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:51:13.707049 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:51:13.707021    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/655b500a-1f9c-424f-9bd2-376464e7151f-plugins-config-volume" pod "655b500a-1f9c-424f-9bd2-376464e7151f" (UID: "655b500a-1f9c-424f-9bd2-376464e7151f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:51:13.709280 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:51:13.709252    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/655b500a-1f9c-424f-9bd2-376464e7151f-kube-api-access-zpw4x" pod "655b500a-1f9c-424f-9bd2-376464e7151f" (UID: "655b500a-1f9c-424f-9bd2-376464e7151f"). InnerVolumeSpecName "kube-api-access-zpw4x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:51:13.807128 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:51:13.807096    4964 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zpw4x\" (UniqueName: \"kubernetes.io/projected/655b500a-1f9c-424f-9bd2-376464e7151f-kube-api-access-zpw4x\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:51:13.807269 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:51:13.807132    4964 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/655b500a-1f9c-424f-9bd2-376464e7151f-plugins-config-volume\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:51:13.843400 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:51:13.843373    4964 generic.go:356] "Generic (PLEG): container finished" podID="850c4489-7821-4e37-95f0-258336f5e75e" containerID="87c35bdf46a217bbc1ffef1dea2e93e80da9e1f03f337948024539f3a9871c9f" exitCode=0
Oct 31 01:51:13.843524 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:51:13.843418    4964 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-9d7b2c-7548d86fb9-djhv5"
Oct 31 01:51:13.843524 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:51:13.843456    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-9d7b2c-7548d86fb9-djhv5" event={"ID":"850c4489-7821-4e37-95f0-258336f5e75e","Type":"ContainerDied","Data":"87c35bdf46a217bbc1ffef1dea2e93e80da9e1f03f337948024539f3a9871c9f"}
Oct 31 01:51:13.843524 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:51:13.843501    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-9d7b2c-7548d86fb9-djhv5" event={"ID":"850c4489-7821-4e37-95f0-258336f5e75e","Type":"ContainerDied","Data":"7550cc0111e5c2324137a9ffb827c5ce4ef5238bc48a6a183418506a6df2897e"}
Oct 31 01:51:13.843689 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:51:13.843530    4964 scope.go:122] "RemoveContainer" containerID="87c35bdf46a217bbc1ffef1dea2e93e80da9e1f03f337948024539f3a9871c9f"
Oct 31 01:51:13.845064 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:51:13.845034    4964 generic.go:356] "Generic (PLEG): container finished" podID="655b500a-1f9c-424f-9bd2-376464e7151f" containerID="8937701057d857935308ef0e65ce7d3c66023a942e4535c3ae7a8805f71d5ce9" exitCode=0
Oct 31 01:51:13.845178 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:51:13.845069    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-9d7b2c-b59bbf96c-7m86r" event={"ID":"655b500a-1f9c-424f-9bd2-376464e7151f","Type":"ContainerDied","Data":"8937701057d857935308ef0e65ce7d3c66023a942e4535c3ae7a8805f71d5ce9"}
Oct 31 01:51:13.845178 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:51:13.845097    4964 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-9d7b2c-b59bbf96c-7m86r"
Oct 31 01:51:13.845296 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:51:13.845105    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-9d7b2c-b59bbf96c-7m86r" event={"ID":"655b500a-1f9c-424f-9bd2-376464e7151f","Type":"ContainerDied","Data":"a55ecab697687bbad1edea50bea78680f8c89512b42cf37b2f087b83cb4ea0a2"}
Oct 31 01:51:13.853796 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:51:13.853773    4964 scope.go:122] "RemoveContainer" containerID="87c35bdf46a217bbc1ffef1dea2e93e80da9e1f03f337948024539f3a9871c9f"
Oct 31 01:51:13.854059 ip-10-0-133-197 kubenswrapper[4964]: E1031 01:51:13.854039    4964 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"87c35bdf46a217bbc1ffef1dea2e93e80da9e1f03f337948024539f3a9871c9f\": container with ID starting with 87c35bdf46a217bbc1ffef1dea2e93e80da9e1f03f337948024539f3a9871c9f not found: ID does not exist" containerID="87c35bdf46a217bbc1ffef1dea2e93e80da9e1f03f337948024539f3a9871c9f"
Oct 31 01:51:13.854129 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:51:13.854067    4964 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"87c35bdf46a217bbc1ffef1dea2e93e80da9e1f03f337948024539f3a9871c9f"} err="failed to get container status \"87c35bdf46a217bbc1ffef1dea2e93e80da9e1f03f337948024539f3a9871c9f\": rpc error: code = NotFound desc = could not find container \"87c35bdf46a217bbc1ffef1dea2e93e80da9e1f03f337948024539f3a9871c9f\": container with ID starting with 87c35bdf46a217bbc1ffef1dea2e93e80da9e1f03f337948024539f3a9871c9f not found: ID does not exist"
Oct 31 01:51:13.854129 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:51:13.854084    4964 scope.go:122] "RemoveContainer" containerID="8937701057d857935308ef0e65ce7d3c66023a942e4535c3ae7a8805f71d5ce9"
Oct 31 01:51:13.863526 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:51:13.863506    4964 scope.go:122] "RemoveContainer" containerID="8937701057d857935308ef0e65ce7d3c66023a942e4535c3ae7a8805f71d5ce9"
Oct 31 01:51:13.863759 ip-10-0-133-197 kubenswrapper[4964]: E1031 01:51:13.863741    4964 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8937701057d857935308ef0e65ce7d3c66023a942e4535c3ae7a8805f71d5ce9\": container with ID starting with 8937701057d857935308ef0e65ce7d3c66023a942e4535c3ae7a8805f71d5ce9 not found: ID does not exist" containerID="8937701057d857935308ef0e65ce7d3c66023a942e4535c3ae7a8805f71d5ce9"
Oct 31 01:51:13.863822 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:51:13.863769    4964 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8937701057d857935308ef0e65ce7d3c66023a942e4535c3ae7a8805f71d5ce9"} err="failed to get container status \"8937701057d857935308ef0e65ce7d3c66023a942e4535c3ae7a8805f71d5ce9\": rpc error: code = NotFound desc = could not find container \"8937701057d857935308ef0e65ce7d3c66023a942e4535c3ae7a8805f71d5ce9\": container with ID starting with 8937701057d857935308ef0e65ce7d3c66023a942e4535c3ae7a8805f71d5ce9 not found: ID does not exist"
Oct 31 01:51:14.008030 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:51:14.008002    4964 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-9d7b2c-7548d86fb9-djhv5"]
Oct 31 01:51:14.012893 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:51:14.012867    4964 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-9d7b2c-7548d86fb9-djhv5"]
Oct 31 01:51:14.119677 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:51:14.119645    4964 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-9d7b2c-b59bbf96c-7m86r"]
Oct 31 01:51:14.123647 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:51:14.123622    4964 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-9d7b2c-b59bbf96c-7m86r"]
Oct 31 01:51:15.777211 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:51:15.777177    4964 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="655b500a-1f9c-424f-9bd2-376464e7151f" path="/var/lib/kubelet/pods/655b500a-1f9c-424f-9bd2-376464e7151f/volumes"
Oct 31 01:51:15.777646 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:51:15.777592    4964 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="850c4489-7821-4e37-95f0-258336f5e75e" path="/var/lib/kubelet/pods/850c4489-7821-4e37-95f0-258336f5e75e/volumes"
Oct 31 01:51:16.103584 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:51:16.103502    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:51:18.148709 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:51:18.148671    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:51:20.916784 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:51:20.916747    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:51:25.115024 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:51:25.114990    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:51:26.597274 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:51:26.597240    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:51:26.812117 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:51:26.812062    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:51:26.995980 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:51:26.995943    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:51:27.542281 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:51:27.542245    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:51:28.469970 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:51:28.469932    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:51:29.825012 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:51:29.824978    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:51:31.895125 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:51:31.895084    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:51:32.243944 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:51:32.243910    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:51:32.425390 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:51:32.425354    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:51:33.753498 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:51:33.753452    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:51:33.935275 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:51:33.935239    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:51:34.051886 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:51:34.051808    4964 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-9d7b2c-openshift-default-76c779b789-tvtzf"]
Oct 31 01:51:34.052050 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:51:34.052007    4964 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-a-9d7b2c-openshift-default-76c779b789-tvtzf" podUID="79d0fdfe-95ba-4800-9636-68f7afc6829e" containerName="istio-proxy" containerID="cri-o://b76be12f47a569be16bab1f7039252f9201ab57c83606f066ab6a9fa9d96897f" gracePeriod=30
Oct 31 01:51:34.118823 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:51:34.118792    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:51:36.925495 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:51:36.925457    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:51:39.329953 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:51:39.329893    4964 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-9d7b2c-openshift-default-76c779b789-tvtzf"
Oct 31 01:51:39.399771 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:51:39.399742    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/79d0fdfe-95ba-4800-9636-68f7afc6829e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/79d0fdfe-95ba-4800-9636-68f7afc6829e-istio-podinfo\") pod \"79d0fdfe-95ba-4800-9636-68f7afc6829e\" (UID: \"79d0fdfe-95ba-4800-9636-68f7afc6829e\") "
Oct 31 01:51:39.399935 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:51:39.399787    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/79d0fdfe-95ba-4800-9636-68f7afc6829e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/79d0fdfe-95ba-4800-9636-68f7afc6829e-credential-socket\") pod \"79d0fdfe-95ba-4800-9636-68f7afc6829e\" (UID: \"79d0fdfe-95ba-4800-9636-68f7afc6829e\") "
Oct 31 01:51:39.399935 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:51:39.399814    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/79d0fdfe-95ba-4800-9636-68f7afc6829e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/79d0fdfe-95ba-4800-9636-68f7afc6829e-istio-envoy\") pod \"79d0fdfe-95ba-4800-9636-68f7afc6829e\" (UID: \"79d0fdfe-95ba-4800-9636-68f7afc6829e\") "
Oct 31 01:51:39.399935 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:51:39.399897    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/79d0fdfe-95ba-4800-9636-68f7afc6829e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/79d0fdfe-95ba-4800-9636-68f7afc6829e-istio-data\") pod \"79d0fdfe-95ba-4800-9636-68f7afc6829e\" (UID: \"79d0fdfe-95ba-4800-9636-68f7afc6829e\") "
Oct 31 01:51:39.399935 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:51:39.399930    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/79d0fdfe-95ba-4800-9636-68f7afc6829e-istio-token\" (UniqueName: \"kubernetes.io/projected/79d0fdfe-95ba-4800-9636-68f7afc6829e-istio-token\") pod \"79d0fdfe-95ba-4800-9636-68f7afc6829e\" (UID: \"79d0fdfe-95ba-4800-9636-68f7afc6829e\") "
Oct 31 01:51:39.400151 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:51:39.399984    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/79d0fdfe-95ba-4800-9636-68f7afc6829e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/79d0fdfe-95ba-4800-9636-68f7afc6829e-istiod-ca-cert\") pod \"79d0fdfe-95ba-4800-9636-68f7afc6829e\" (UID: \"79d0fdfe-95ba-4800-9636-68f7afc6829e\") "
Oct 31 01:51:39.400151 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:51:39.400024    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/79d0fdfe-95ba-4800-9636-68f7afc6829e-kube-api-access-jdmlx\" (UniqueName: \"kubernetes.io/projected/79d0fdfe-95ba-4800-9636-68f7afc6829e-kube-api-access-jdmlx\") pod \"79d0fdfe-95ba-4800-9636-68f7afc6829e\" (UID: \"79d0fdfe-95ba-4800-9636-68f7afc6829e\") "
Oct 31 01:51:39.400151 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:51:39.400059    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79d0fdfe-95ba-4800-9636-68f7afc6829e-credential-socket" pod "79d0fdfe-95ba-4800-9636-68f7afc6829e" (UID: "79d0fdfe-95ba-4800-9636-68f7afc6829e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:51:39.400151 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:51:39.400066    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/79d0fdfe-95ba-4800-9636-68f7afc6829e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/79d0fdfe-95ba-4800-9636-68f7afc6829e-workload-socket\") pod \"79d0fdfe-95ba-4800-9636-68f7afc6829e\" (UID: \"79d0fdfe-95ba-4800-9636-68f7afc6829e\") "
Oct 31 01:51:39.400375 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:51:39.400155    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/79d0fdfe-95ba-4800-9636-68f7afc6829e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/79d0fdfe-95ba-4800-9636-68f7afc6829e-workload-certs\") pod \"79d0fdfe-95ba-4800-9636-68f7afc6829e\" (UID: \"79d0fdfe-95ba-4800-9636-68f7afc6829e\") "
Oct 31 01:51:39.400569 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:51:39.400533    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79d0fdfe-95ba-4800-9636-68f7afc6829e-workload-socket" pod "79d0fdfe-95ba-4800-9636-68f7afc6829e" (UID: "79d0fdfe-95ba-4800-9636-68f7afc6829e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:51:39.400718 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:51:39.400657    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/79d0fdfe-95ba-4800-9636-68f7afc6829e-istiod-ca-cert" pod "79d0fdfe-95ba-4800-9636-68f7afc6829e" (UID: "79d0fdfe-95ba-4800-9636-68f7afc6829e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:51:39.400782 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:51:39.400740    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79d0fdfe-95ba-4800-9636-68f7afc6829e-workload-certs" pod "79d0fdfe-95ba-4800-9636-68f7afc6829e" (UID: "79d0fdfe-95ba-4800-9636-68f7afc6829e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:51:39.400835 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:51:39.400785    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79d0fdfe-95ba-4800-9636-68f7afc6829e-istio-data" pod "79d0fdfe-95ba-4800-9636-68f7afc6829e" (UID: "79d0fdfe-95ba-4800-9636-68f7afc6829e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:51:39.401208 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:51:39.401085    4964 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/79d0fdfe-95ba-4800-9636-68f7afc6829e-credential-socket\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:51:39.401208 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:51:39.401112    4964 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/79d0fdfe-95ba-4800-9636-68f7afc6829e-istio-data\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:51:39.401208 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:51:39.401129    4964 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/79d0fdfe-95ba-4800-9636-68f7afc6829e-istiod-ca-cert\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:51:39.401208 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:51:39.401144    4964 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/79d0fdfe-95ba-4800-9636-68f7afc6829e-workload-socket\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:51:39.401208 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:51:39.401158    4964 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/79d0fdfe-95ba-4800-9636-68f7afc6829e-workload-certs\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:51:39.402410 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:51:39.402381    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79d0fdfe-95ba-4800-9636-68f7afc6829e-istio-envoy" pod "79d0fdfe-95ba-4800-9636-68f7afc6829e" (UID: "79d0fdfe-95ba-4800-9636-68f7afc6829e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:51:39.402509 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:51:39.402491    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/79d0fdfe-95ba-4800-9636-68f7afc6829e-istio-podinfo" pod "79d0fdfe-95ba-4800-9636-68f7afc6829e" (UID: "79d0fdfe-95ba-4800-9636-68f7afc6829e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 01:51:39.402727 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:51:39.402707    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79d0fdfe-95ba-4800-9636-68f7afc6829e-istio-token" pod "79d0fdfe-95ba-4800-9636-68f7afc6829e" (UID: "79d0fdfe-95ba-4800-9636-68f7afc6829e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:51:39.403171 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:51:39.403150    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79d0fdfe-95ba-4800-9636-68f7afc6829e-kube-api-access-jdmlx" pod "79d0fdfe-95ba-4800-9636-68f7afc6829e" (UID: "79d0fdfe-95ba-4800-9636-68f7afc6829e"). InnerVolumeSpecName "kube-api-access-jdmlx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:51:39.502090 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:51:39.502022    4964 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/79d0fdfe-95ba-4800-9636-68f7afc6829e-istio-podinfo\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:51:39.502090 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:51:39.502051    4964 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/79d0fdfe-95ba-4800-9636-68f7afc6829e-istio-envoy\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:51:39.502090 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:51:39.502061    4964 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/79d0fdfe-95ba-4800-9636-68f7afc6829e-istio-token\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:51:39.502090 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:51:39.502069    4964 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jdmlx\" (UniqueName: \"kubernetes.io/projected/79d0fdfe-95ba-4800-9636-68f7afc6829e-kube-api-access-jdmlx\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:51:40.020208 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:51:40.020172    4964 generic.go:356] "Generic (PLEG): container finished" podID="79d0fdfe-95ba-4800-9636-68f7afc6829e" containerID="b76be12f47a569be16bab1f7039252f9201ab57c83606f066ab6a9fa9d96897f" exitCode=0
Oct 31 01:51:40.020443 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:51:40.020224    4964 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-9d7b2c-openshift-default-76c779b789-tvtzf"
Oct 31 01:51:40.020443 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:51:40.020253    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-9d7b2c-openshift-default-76c779b789-tvtzf" event={"ID":"79d0fdfe-95ba-4800-9636-68f7afc6829e","Type":"ContainerDied","Data":"b76be12f47a569be16bab1f7039252f9201ab57c83606f066ab6a9fa9d96897f"}
Oct 31 01:51:40.020443 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:51:40.020341    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-9d7b2c-openshift-default-76c779b789-tvtzf" event={"ID":"79d0fdfe-95ba-4800-9636-68f7afc6829e","Type":"ContainerDied","Data":"6f38be7ae52e46f5e53ecdd3b58779e78a622a0dc70e9d82bf87d3c102f7057f"}
Oct 31 01:51:40.020443 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:51:40.020358    4964 scope.go:122] "RemoveContainer" containerID="b76be12f47a569be16bab1f7039252f9201ab57c83606f066ab6a9fa9d96897f"
Oct 31 01:51:40.031524 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:51:40.031504    4964 scope.go:122] "RemoveContainer" containerID="15e2887cfd42d5b3335dcbf8df443b50617bce918ae92dc6c7c8490d2b2440f6"
Oct 31 01:51:40.042249 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:51:40.042231    4964 scope.go:122] "RemoveContainer" containerID="b76be12f47a569be16bab1f7039252f9201ab57c83606f066ab6a9fa9d96897f"
Oct 31 01:51:40.042557 ip-10-0-133-197 kubenswrapper[4964]: E1031 01:51:40.042538    4964 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b76be12f47a569be16bab1f7039252f9201ab57c83606f066ab6a9fa9d96897f\": container with ID starting with b76be12f47a569be16bab1f7039252f9201ab57c83606f066ab6a9fa9d96897f not found: ID does not exist" containerID="b76be12f47a569be16bab1f7039252f9201ab57c83606f066ab6a9fa9d96897f"
Oct 31 01:51:40.042618 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:51:40.042565    4964 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b76be12f47a569be16bab1f7039252f9201ab57c83606f066ab6a9fa9d96897f"} err="failed to get container status \"b76be12f47a569be16bab1f7039252f9201ab57c83606f066ab6a9fa9d96897f\": rpc error: code = NotFound desc = could not find container \"b76be12f47a569be16bab1f7039252f9201ab57c83606f066ab6a9fa9d96897f\": container with ID starting with b76be12f47a569be16bab1f7039252f9201ab57c83606f066ab6a9fa9d96897f not found: ID does not exist"
Oct 31 01:51:40.042618 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:51:40.042583    4964 scope.go:122] "RemoveContainer" containerID="15e2887cfd42d5b3335dcbf8df443b50617bce918ae92dc6c7c8490d2b2440f6"
Oct 31 01:51:40.042818 ip-10-0-133-197 kubenswrapper[4964]: E1031 01:51:40.042800    4964 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"15e2887cfd42d5b3335dcbf8df443b50617bce918ae92dc6c7c8490d2b2440f6\": container with ID starting with 15e2887cfd42d5b3335dcbf8df443b50617bce918ae92dc6c7c8490d2b2440f6 not found: ID does not exist" containerID="15e2887cfd42d5b3335dcbf8df443b50617bce918ae92dc6c7c8490d2b2440f6"
Oct 31 01:51:40.042880 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:51:40.042829    4964 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"15e2887cfd42d5b3335dcbf8df443b50617bce918ae92dc6c7c8490d2b2440f6"} err="failed to get container status \"15e2887cfd42d5b3335dcbf8df443b50617bce918ae92dc6c7c8490d2b2440f6\": rpc error: code = NotFound desc = could not find container \"15e2887cfd42d5b3335dcbf8df443b50617bce918ae92dc6c7c8490d2b2440f6\": container with ID starting with 15e2887cfd42d5b3335dcbf8df443b50617bce918ae92dc6c7c8490d2b2440f6 not found: ID does not exist"
Oct 31 01:51:40.184192 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:51:40.184162    4964 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-9d7b2c-openshift-default-76c779b789-tvtzf"]
Oct 31 01:51:40.186959 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:51:40.186938    4964 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-a-9d7b2c-openshift-default-76c779b789-tvtzf"]
Oct 31 01:51:41.246802 ip-10-0-133-197 kubenswrapper[4964]: E1031 01:51:41.246750    4964 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q56x5" containerName="extractor"
Oct 31 01:51:41.318862 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:51:41.318828    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:51:41.535392 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:51:41.535274    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:51:41.777550 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:51:41.777522    4964 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="79d0fdfe-95ba-4800-9636-68f7afc6829e" path="/var/lib/kubelet/pods/79d0fdfe-95ba-4800-9636-68f7afc6829e/volumes"
Oct 31 01:51:42.523384 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:51:42.523350    4964 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-53af08-openshift-default-556ff7748f-4nqv7"]
Oct 31 01:51:42.523811 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:51:42.523566    4964 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-53af08-openshift-default-556ff7748f-4nqv7" podUID="778c73f2-648f-4909-a1e9-a685d1d9a2c7" containerName="istio-proxy" containerID="cri-o://b79a849b557c24c90da3d9ff4869534df181b56f2702e63570fa562aa0df24e6" gracePeriod=30
Oct 31 01:51:42.557472 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:51:42.557444    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:51:42.894706 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:51:42.894619    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:51:43.103279 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:51:43.103237    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:51:43.283036 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:51:43.283006    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:51:43.809072 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:51:43.809031    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:51:46.354174 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:51:46.354135    4964 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-2c0de9-f5cc7596c-j6k5n"]
Oct 31 01:51:46.459816 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:51:46.459784    4964 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-2c0de9-5f57446ff9-wrm5h"]
Oct 31 01:51:46.755760 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:51:46.755725    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:51:46.972960 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:51:46.972927    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:51:47.837680 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:51:47.837655    4964 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-53af08-openshift-default-556ff7748f-4nqv7"
Oct 31 01:51:47.900364 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:51:47.900334    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/778c73f2-648f-4909-a1e9-a685d1d9a2c7-istio-token\" (UniqueName: \"kubernetes.io/projected/778c73f2-648f-4909-a1e9-a685d1d9a2c7-istio-token\") pod \"778c73f2-648f-4909-a1e9-a685d1d9a2c7\" (UID: \"778c73f2-648f-4909-a1e9-a685d1d9a2c7\") "
Oct 31 01:51:47.900557 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:51:47.900394    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/778c73f2-648f-4909-a1e9-a685d1d9a2c7-kube-api-access-7pw4z\" (UniqueName: \"kubernetes.io/projected/778c73f2-648f-4909-a1e9-a685d1d9a2c7-kube-api-access-7pw4z\") pod \"778c73f2-648f-4909-a1e9-a685d1d9a2c7\" (UID: \"778c73f2-648f-4909-a1e9-a685d1d9a2c7\") "
Oct 31 01:51:47.900557 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:51:47.900420    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/778c73f2-648f-4909-a1e9-a685d1d9a2c7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/778c73f2-648f-4909-a1e9-a685d1d9a2c7-istiod-ca-cert\") pod \"778c73f2-648f-4909-a1e9-a685d1d9a2c7\" (UID: \"778c73f2-648f-4909-a1e9-a685d1d9a2c7\") "
Oct 31 01:51:47.900557 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:51:47.900464    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/778c73f2-648f-4909-a1e9-a685d1d9a2c7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/778c73f2-648f-4909-a1e9-a685d1d9a2c7-credential-socket\") pod \"778c73f2-648f-4909-a1e9-a685d1d9a2c7\" (UID: \"778c73f2-648f-4909-a1e9-a685d1d9a2c7\") "
Oct 31 01:51:47.900557 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:51:47.900541    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/778c73f2-648f-4909-a1e9-a685d1d9a2c7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/778c73f2-648f-4909-a1e9-a685d1d9a2c7-istio-podinfo\") pod \"778c73f2-648f-4909-a1e9-a685d1d9a2c7\" (UID: \"778c73f2-648f-4909-a1e9-a685d1d9a2c7\") "
Oct 31 01:51:47.900792 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:51:47.900564    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/778c73f2-648f-4909-a1e9-a685d1d9a2c7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/778c73f2-648f-4909-a1e9-a685d1d9a2c7-istio-data\") pod \"778c73f2-648f-4909-a1e9-a685d1d9a2c7\" (UID: \"778c73f2-648f-4909-a1e9-a685d1d9a2c7\") "
Oct 31 01:51:47.900792 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:51:47.900631    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/778c73f2-648f-4909-a1e9-a685d1d9a2c7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/778c73f2-648f-4909-a1e9-a685d1d9a2c7-workload-socket\") pod \"778c73f2-648f-4909-a1e9-a685d1d9a2c7\" (UID: \"778c73f2-648f-4909-a1e9-a685d1d9a2c7\") "
Oct 31 01:51:47.900792 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:51:47.900673    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/778c73f2-648f-4909-a1e9-a685d1d9a2c7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/778c73f2-648f-4909-a1e9-a685d1d9a2c7-workload-certs\") pod \"778c73f2-648f-4909-a1e9-a685d1d9a2c7\" (UID: \"778c73f2-648f-4909-a1e9-a685d1d9a2c7\") "
Oct 31 01:51:47.900792 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:51:47.900726    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/778c73f2-648f-4909-a1e9-a685d1d9a2c7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/778c73f2-648f-4909-a1e9-a685d1d9a2c7-istio-envoy\") pod \"778c73f2-648f-4909-a1e9-a685d1d9a2c7\" (UID: \"778c73f2-648f-4909-a1e9-a685d1d9a2c7\") "
Oct 31 01:51:47.900792 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:51:47.900760    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/778c73f2-648f-4909-a1e9-a685d1d9a2c7-credential-socket" pod "778c73f2-648f-4909-a1e9-a685d1d9a2c7" (UID: "778c73f2-648f-4909-a1e9-a685d1d9a2c7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:51:47.901066 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:51:47.900923    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/778c73f2-648f-4909-a1e9-a685d1d9a2c7-workload-certs" pod "778c73f2-648f-4909-a1e9-a685d1d9a2c7" (UID: "778c73f2-648f-4909-a1e9-a685d1d9a2c7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:51:47.901066 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:51:47.900883    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/778c73f2-648f-4909-a1e9-a685d1d9a2c7-istio-data" pod "778c73f2-648f-4909-a1e9-a685d1d9a2c7" (UID: "778c73f2-648f-4909-a1e9-a685d1d9a2c7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:51:47.901066 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:51:47.900960    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/778c73f2-648f-4909-a1e9-a685d1d9a2c7-istiod-ca-cert" pod "778c73f2-648f-4909-a1e9-a685d1d9a2c7" (UID: "778c73f2-648f-4909-a1e9-a685d1d9a2c7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:51:47.901066 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:51:47.900994    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/778c73f2-648f-4909-a1e9-a685d1d9a2c7-workload-socket" pod "778c73f2-648f-4909-a1e9-a685d1d9a2c7" (UID: "778c73f2-648f-4909-a1e9-a685d1d9a2c7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:51:47.901370 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:51:47.901312    4964 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/778c73f2-648f-4909-a1e9-a685d1d9a2c7-istiod-ca-cert\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:51:47.901573 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:51:47.901374    4964 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/778c73f2-648f-4909-a1e9-a685d1d9a2c7-credential-socket\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:51:47.901573 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:51:47.901383    4964 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/778c73f2-648f-4909-a1e9-a685d1d9a2c7-istio-data\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:51:47.901573 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:51:47.901392    4964 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/778c73f2-648f-4909-a1e9-a685d1d9a2c7-workload-socket\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:51:47.901573 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:51:47.901400    4964 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/778c73f2-648f-4909-a1e9-a685d1d9a2c7-workload-certs\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:51:47.903047 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:51:47.903004    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/778c73f2-648f-4909-a1e9-a685d1d9a2c7-istio-envoy" pod "778c73f2-648f-4909-a1e9-a685d1d9a2c7" (UID: "778c73f2-648f-4909-a1e9-a685d1d9a2c7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:51:47.903369 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:51:47.903344    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/778c73f2-648f-4909-a1e9-a685d1d9a2c7-istio-podinfo" pod "778c73f2-648f-4909-a1e9-a685d1d9a2c7" (UID: "778c73f2-648f-4909-a1e9-a685d1d9a2c7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 01:51:47.903474 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:51:47.903388    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/778c73f2-648f-4909-a1e9-a685d1d9a2c7-kube-api-access-7pw4z" pod "778c73f2-648f-4909-a1e9-a685d1d9a2c7" (UID: "778c73f2-648f-4909-a1e9-a685d1d9a2c7"). InnerVolumeSpecName "kube-api-access-7pw4z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:51:47.903474 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:51:47.903419    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/778c73f2-648f-4909-a1e9-a685d1d9a2c7-istio-token" pod "778c73f2-648f-4909-a1e9-a685d1d9a2c7" (UID: "778c73f2-648f-4909-a1e9-a685d1d9a2c7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:51:48.002159 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:51:48.002128    4964 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/778c73f2-648f-4909-a1e9-a685d1d9a2c7-istio-token\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:51:48.002159 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:51:48.002157    4964 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7pw4z\" (UniqueName: \"kubernetes.io/projected/778c73f2-648f-4909-a1e9-a685d1d9a2c7-kube-api-access-7pw4z\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:51:48.002415 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:51:48.002172    4964 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/778c73f2-648f-4909-a1e9-a685d1d9a2c7-istio-podinfo\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:51:48.002415 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:51:48.002186    4964 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/778c73f2-648f-4909-a1e9-a685d1d9a2c7-istio-envoy\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:51:48.078256 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:51:48.078167    4964 generic.go:356] "Generic (PLEG): container finished" podID="778c73f2-648f-4909-a1e9-a685d1d9a2c7" containerID="b79a849b557c24c90da3d9ff4869534df181b56f2702e63570fa562aa0df24e6" exitCode=0
Oct 31 01:51:48.078256 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:51:48.078224    4964 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-53af08-openshift-default-556ff7748f-4nqv7"
Oct 31 01:51:48.078256 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:51:48.078237    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-53af08-openshift-default-556ff7748f-4nqv7" event={"ID":"778c73f2-648f-4909-a1e9-a685d1d9a2c7","Type":"ContainerDied","Data":"b79a849b557c24c90da3d9ff4869534df181b56f2702e63570fa562aa0df24e6"}
Oct 31 01:51:48.078543 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:51:48.078303    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-53af08-openshift-default-556ff7748f-4nqv7" event={"ID":"778c73f2-648f-4909-a1e9-a685d1d9a2c7","Type":"ContainerDied","Data":"e5e289dbedca40981ad06b1b5f235d7be84f51451f66b56d5dc1b50e1ad7b4cc"}
Oct 31 01:51:48.078543 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:51:48.078373    4964 scope.go:122] "RemoveContainer" containerID="b79a849b557c24c90da3d9ff4869534df181b56f2702e63570fa562aa0df24e6"
Oct 31 01:51:48.089562 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:51:48.089542    4964 scope.go:122] "RemoveContainer" containerID="537884ca81ca127b04c754bf17d7fe77bcbae562cc0dcd4adf9033d91990d433"
Oct 31 01:51:48.099120 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:51:48.099104    4964 scope.go:122] "RemoveContainer" containerID="b79a849b557c24c90da3d9ff4869534df181b56f2702e63570fa562aa0df24e6"
Oct 31 01:51:48.099441 ip-10-0-133-197 kubenswrapper[4964]: E1031 01:51:48.099418    4964 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b79a849b557c24c90da3d9ff4869534df181b56f2702e63570fa562aa0df24e6\": container with ID starting with b79a849b557c24c90da3d9ff4869534df181b56f2702e63570fa562aa0df24e6 not found: ID does not exist" containerID="b79a849b557c24c90da3d9ff4869534df181b56f2702e63570fa562aa0df24e6"
Oct 31 01:51:48.099534 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:51:48.099450    4964 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b79a849b557c24c90da3d9ff4869534df181b56f2702e63570fa562aa0df24e6"} err="failed to get container status \"b79a849b557c24c90da3d9ff4869534df181b56f2702e63570fa562aa0df24e6\": rpc error: code = NotFound desc = could not find container \"b79a849b557c24c90da3d9ff4869534df181b56f2702e63570fa562aa0df24e6\": container with ID starting with b79a849b557c24c90da3d9ff4869534df181b56f2702e63570fa562aa0df24e6 not found: ID does not exist"
Oct 31 01:51:48.099534 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:51:48.099469    4964 scope.go:122] "RemoveContainer" containerID="537884ca81ca127b04c754bf17d7fe77bcbae562cc0dcd4adf9033d91990d433"
Oct 31 01:51:48.099743 ip-10-0-133-197 kubenswrapper[4964]: E1031 01:51:48.099725    4964 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"537884ca81ca127b04c754bf17d7fe77bcbae562cc0dcd4adf9033d91990d433\": container with ID starting with 537884ca81ca127b04c754bf17d7fe77bcbae562cc0dcd4adf9033d91990d433 not found: ID does not exist" containerID="537884ca81ca127b04c754bf17d7fe77bcbae562cc0dcd4adf9033d91990d433"
Oct 31 01:51:48.099793 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:51:48.099749    4964 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"537884ca81ca127b04c754bf17d7fe77bcbae562cc0dcd4adf9033d91990d433"} err="failed to get container status \"537884ca81ca127b04c754bf17d7fe77bcbae562cc0dcd4adf9033d91990d433\": rpc error: code = NotFound desc = could not find container \"537884ca81ca127b04c754bf17d7fe77bcbae562cc0dcd4adf9033d91990d433\": container with ID starting with 537884ca81ca127b04c754bf17d7fe77bcbae562cc0dcd4adf9033d91990d433 not found: ID does not exist"
Oct 31 01:51:48.123600 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:51:48.123577    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:51:48.243567 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:51:48.243531    4964 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-53af08-openshift-default-556ff7748f-4nqv7"]
Oct 31 01:51:48.246506 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:51:48.246480    4964 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-53af08-openshift-default-556ff7748f-4nqv7"]
Oct 31 01:51:49.546248 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:51:49.546206    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:51:49.725591 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:51:49.725541    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:51:49.777804 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:51:49.777770    4964 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="778c73f2-648f-4909-a1e9-a685d1d9a2c7" path="/var/lib/kubelet/pods/778c73f2-648f-4909-a1e9-a685d1d9a2c7/volumes"
Oct 31 01:51:49.913030 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:51:49.912923    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:51:51.354647 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:51:51.354612    4964 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-b-2c0de9-f5cc7596c-j6k5n" podUID="d7751c9b-c8db-4bf9-8193-49c3fefe90e3" containerName="payload-processing" containerID="cri-o://87036946e9d1a7812a27f2e99faddd8ed33478a12e4f359fb8b16729790917cb" gracePeriod=25
Oct 31 01:51:51.461145 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:51:51.461097    4964 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-2c0de9-5f57446ff9-wrm5h" podUID="c3c14a55-5462-4545-afd3-aa589cc162c9" containerName="payload-pre-processing" containerID="cri-o://2e44b411cf520ac3b914e0f8a123cfe497d6f9b28e17e05bdb2726524c808162" gracePeriod=25
Oct 31 01:51:51.631150 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:51:51.630804    4964 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-2c0de9-f5cc7596c-j6k5n"
Oct 31 01:51:51.749571 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:51:51.749542    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d7751c9b-c8db-4bf9-8193-49c3fefe90e3-kube-api-access-lvxwb\" (UniqueName: \"kubernetes.io/projected/d7751c9b-c8db-4bf9-8193-49c3fefe90e3-kube-api-access-lvxwb\") pod \"d7751c9b-c8db-4bf9-8193-49c3fefe90e3\" (UID: \"d7751c9b-c8db-4bf9-8193-49c3fefe90e3\") "
Oct 31 01:51:51.749789 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:51:51.749771    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d7751c9b-c8db-4bf9-8193-49c3fefe90e3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d7751c9b-c8db-4bf9-8193-49c3fefe90e3-plugins-config-volume\") pod \"d7751c9b-c8db-4bf9-8193-49c3fefe90e3\" (UID: \"d7751c9b-c8db-4bf9-8193-49c3fefe90e3\") "
Oct 31 01:51:51.750107 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:51:51.750076    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d7751c9b-c8db-4bf9-8193-49c3fefe90e3-plugins-config-volume" pod "d7751c9b-c8db-4bf9-8193-49c3fefe90e3" (UID: "d7751c9b-c8db-4bf9-8193-49c3fefe90e3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:51:51.750413 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:51:51.750242    4964 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d7751c9b-c8db-4bf9-8193-49c3fefe90e3-plugins-config-volume\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:51:51.750576 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:51:51.750556    4964 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-2c0de9-5f57446ff9-wrm5h"
Oct 31 01:51:51.752499 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:51:51.752470    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d7751c9b-c8db-4bf9-8193-49c3fefe90e3-kube-api-access-lvxwb" pod "d7751c9b-c8db-4bf9-8193-49c3fefe90e3" (UID: "d7751c9b-c8db-4bf9-8193-49c3fefe90e3"). InnerVolumeSpecName "kube-api-access-lvxwb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:51:51.851379 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:51:51.851353    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c3c14a55-5462-4545-afd3-aa589cc162c9-kube-api-access-n5cnd\" (UniqueName: \"kubernetes.io/projected/c3c14a55-5462-4545-afd3-aa589cc162c9-kube-api-access-n5cnd\") pod \"c3c14a55-5462-4545-afd3-aa589cc162c9\" (UID: \"c3c14a55-5462-4545-afd3-aa589cc162c9\") "
Oct 31 01:51:51.851578 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:51:51.851560    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c3c14a55-5462-4545-afd3-aa589cc162c9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c3c14a55-5462-4545-afd3-aa589cc162c9-plugins-config-volume\") pod \"c3c14a55-5462-4545-afd3-aa589cc162c9\" (UID: \"c3c14a55-5462-4545-afd3-aa589cc162c9\") "
Oct 31 01:51:51.851909 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:51:51.851886    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c3c14a55-5462-4545-afd3-aa589cc162c9-plugins-config-volume" pod "c3c14a55-5462-4545-afd3-aa589cc162c9" (UID: "c3c14a55-5462-4545-afd3-aa589cc162c9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:51:51.852040 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:51:51.852028    4964 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lvxwb\" (UniqueName: \"kubernetes.io/projected/d7751c9b-c8db-4bf9-8193-49c3fefe90e3-kube-api-access-lvxwb\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:51:51.852087 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:51:51.852042    4964 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c3c14a55-5462-4545-afd3-aa589cc162c9-plugins-config-volume\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:51:51.853535 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:51:51.853518    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c3c14a55-5462-4545-afd3-aa589cc162c9-kube-api-access-n5cnd" pod "c3c14a55-5462-4545-afd3-aa589cc162c9" (UID: "c3c14a55-5462-4545-afd3-aa589cc162c9"). InnerVolumeSpecName "kube-api-access-n5cnd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:51:51.953856 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:51:51.953773    4964 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n5cnd\" (UniqueName: \"kubernetes.io/projected/c3c14a55-5462-4545-afd3-aa589cc162c9-kube-api-access-n5cnd\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:51:52.105740 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:51:52.105706    4964 generic.go:356] "Generic (PLEG): container finished" podID="d7751c9b-c8db-4bf9-8193-49c3fefe90e3" containerID="87036946e9d1a7812a27f2e99faddd8ed33478a12e4f359fb8b16729790917cb" exitCode=0
Oct 31 01:51:52.105910 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:51:52.105753    4964 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-2c0de9-f5cc7596c-j6k5n"
Oct 31 01:51:52.105910 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:51:52.105797    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-2c0de9-f5cc7596c-j6k5n" event={"ID":"d7751c9b-c8db-4bf9-8193-49c3fefe90e3","Type":"ContainerDied","Data":"87036946e9d1a7812a27f2e99faddd8ed33478a12e4f359fb8b16729790917cb"}
Oct 31 01:51:52.105910 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:51:52.105856    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-2c0de9-f5cc7596c-j6k5n" event={"ID":"d7751c9b-c8db-4bf9-8193-49c3fefe90e3","Type":"ContainerDied","Data":"dd8619a30cc35f9ba4bef7a1a98c9be447496bd9bc6e03a56719b7fd7879ca72"}
Oct 31 01:51:52.105910 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:51:52.105883    4964 scope.go:122] "RemoveContainer" containerID="87036946e9d1a7812a27f2e99faddd8ed33478a12e4f359fb8b16729790917cb"
Oct 31 01:51:52.107457 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:51:52.107434    4964 generic.go:356] "Generic (PLEG): container finished" podID="c3c14a55-5462-4545-afd3-aa589cc162c9" containerID="2e44b411cf520ac3b914e0f8a123cfe497d6f9b28e17e05bdb2726524c808162" exitCode=0
Oct 31 01:51:52.107587 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:51:52.107491    4964 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-2c0de9-5f57446ff9-wrm5h"
Oct 31 01:51:52.107752 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:51:52.107669    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-2c0de9-5f57446ff9-wrm5h" event={"ID":"c3c14a55-5462-4545-afd3-aa589cc162c9","Type":"ContainerDied","Data":"2e44b411cf520ac3b914e0f8a123cfe497d6f9b28e17e05bdb2726524c808162"}
Oct 31 01:51:52.107752 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:51:52.107721    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-2c0de9-5f57446ff9-wrm5h" event={"ID":"c3c14a55-5462-4545-afd3-aa589cc162c9","Type":"ContainerDied","Data":"5b0d33bdd7809dcffd2e308cf9ad84980eb0b387202ab12d4272efb4b695bac7"}
Oct 31 01:51:52.117878 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:51:52.117861    4964 scope.go:122] "RemoveContainer" containerID="87036946e9d1a7812a27f2e99faddd8ed33478a12e4f359fb8b16729790917cb"
Oct 31 01:51:52.118172 ip-10-0-133-197 kubenswrapper[4964]: E1031 01:51:52.118150    4964 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"87036946e9d1a7812a27f2e99faddd8ed33478a12e4f359fb8b16729790917cb\": container with ID starting with 87036946e9d1a7812a27f2e99faddd8ed33478a12e4f359fb8b16729790917cb not found: ID does not exist" containerID="87036946e9d1a7812a27f2e99faddd8ed33478a12e4f359fb8b16729790917cb"
Oct 31 01:51:52.118217 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:51:52.118185    4964 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"87036946e9d1a7812a27f2e99faddd8ed33478a12e4f359fb8b16729790917cb"} err="failed to get container status \"87036946e9d1a7812a27f2e99faddd8ed33478a12e4f359fb8b16729790917cb\": rpc error: code = NotFound desc = could not find container \"87036946e9d1a7812a27f2e99faddd8ed33478a12e4f359fb8b16729790917cb\": container with ID starting with 87036946e9d1a7812a27f2e99faddd8ed33478a12e4f359fb8b16729790917cb not found: ID does not exist"
Oct 31 01:51:52.118217 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:51:52.118206    4964 scope.go:122] "RemoveContainer" containerID="2e44b411cf520ac3b914e0f8a123cfe497d6f9b28e17e05bdb2726524c808162"
Oct 31 01:51:52.128007 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:51:52.127987    4964 scope.go:122] "RemoveContainer" containerID="2e44b411cf520ac3b914e0f8a123cfe497d6f9b28e17e05bdb2726524c808162"
Oct 31 01:51:52.128262 ip-10-0-133-197 kubenswrapper[4964]: E1031 01:51:52.128244    4964 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2e44b411cf520ac3b914e0f8a123cfe497d6f9b28e17e05bdb2726524c808162\": container with ID starting with 2e44b411cf520ac3b914e0f8a123cfe497d6f9b28e17e05bdb2726524c808162 not found: ID does not exist" containerID="2e44b411cf520ac3b914e0f8a123cfe497d6f9b28e17e05bdb2726524c808162"
Oct 31 01:51:52.128343 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:51:52.128270    4964 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2e44b411cf520ac3b914e0f8a123cfe497d6f9b28e17e05bdb2726524c808162"} err="failed to get container status \"2e44b411cf520ac3b914e0f8a123cfe497d6f9b28e17e05bdb2726524c808162\": rpc error: code = NotFound desc = could not find container \"2e44b411cf520ac3b914e0f8a123cfe497d6f9b28e17e05bdb2726524c808162\": container with ID starting with 2e44b411cf520ac3b914e0f8a123cfe497d6f9b28e17e05bdb2726524c808162 not found: ID does not exist"
Oct 31 01:51:52.268729 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:51:52.268697    4964 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-2c0de9-f5cc7596c-j6k5n"]
Oct 31 01:51:52.271559 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:51:52.271540    4964 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-2c0de9-f5cc7596c-j6k5n"]
Oct 31 01:51:52.377364 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:51:52.377330    4964 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-2c0de9-5f57446ff9-wrm5h"]
Oct 31 01:51:52.380037 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:51:52.380015    4964 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-2c0de9-5f57446ff9-wrm5h"]
Oct 31 01:51:53.507807 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:51:53.507771    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:51:53.776853 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:51:53.776775    4964 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c3c14a55-5462-4545-afd3-aa589cc162c9" path="/var/lib/kubelet/pods/c3c14a55-5462-4545-afd3-aa589cc162c9/volumes"
Oct 31 01:51:53.777686 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:51:53.777653    4964 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d7751c9b-c8db-4bf9-8193-49c3fefe90e3" path="/var/lib/kubelet/pods/d7751c9b-c8db-4bf9-8193-49c3fefe90e3/volumes"
Oct 31 01:51:57.930708 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:51:57.930670    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:51:59.580350 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:51:59.580296    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:51:59.797759 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:51:59.797722    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:51:59.976971 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:51:59.976936    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:52:00.486077 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:52:00.486042    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:52:04.992498 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:52:04.992458    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:52:06.360237 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:52:06.360196    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:52:06.584492 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:52:06.584450    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:52:06.765128 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:52:06.765070    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:52:09.806747 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:52:09.806712    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:52:11.239403 ip-10-0-133-197 kubenswrapper[4964]: E1031 01:52:11.239365    4964 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q56x5" containerName="extractor"
Oct 31 01:52:13.947162 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:52:13.947133    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:52:14.458269 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:52:14.458239    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:52:14.626282 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:52:14.626243    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:52:15.567123 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:52:15.567093    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:52:15.803868 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:52:15.803832    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:52:15.983371 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:52:15.983336    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:52:16.481506 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:52:16.481471    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:52:19.513173 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:52:19.513139    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:52:19.681336 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:52:19.681279    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:52:21.008489 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:52:21.008459    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:52:22.289747 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:52:22.289715    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:52:22.472722 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:52:22.472686    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:52:25.803200 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:52:25.803164    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:52:26.478573 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:52:26.478537    4964 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-g5qrb"]
Oct 31 01:52:26.479113 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:52:26.479100    4964 cpu_manager.go:414] "containerMap: removing container" podUID="11d36add-45f3-4bf4-9372-7d7cec791eff" containerName="istio-proxy"
Oct 31 01:52:26.479158 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:52:26.479115    4964 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="11d36add-45f3-4bf4-9372-7d7cec791eff" containerName="istio-proxy"
Oct 31 01:52:26.479158 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:52:26.479130    4964 cpu_manager.go:414] "containerMap: removing container" podUID="c4a07c54-1e5b-4aef-b912-dcbc3930ae43" containerName="istio-proxy"
Oct 31 01:52:26.479158 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:52:26.479134    4964 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c4a07c54-1e5b-4aef-b912-dcbc3930ae43" containerName="istio-proxy"
Oct 31 01:52:26.479158 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:52:26.479147    4964 cpu_manager.go:414] "containerMap: removing container" podUID="7ca5c214-0290-4db5-9fb1-a9e27cdf9f58" containerName="payload-pre-processing"
Oct 31 01:52:26.479158 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:52:26.479152    4964 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7ca5c214-0290-4db5-9fb1-a9e27cdf9f58" containerName="payload-pre-processing"
Oct 31 01:52:26.479158 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:52:26.479160    4964 cpu_manager.go:414] "containerMap: removing container" podUID="79d0fdfe-95ba-4800-9636-68f7afc6829e" containerName="istio-proxy"
Oct 31 01:52:26.479375 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:52:26.479165    4964 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="79d0fdfe-95ba-4800-9636-68f7afc6829e" containerName="istio-proxy"
Oct 31 01:52:26.479375 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:52:26.479179    4964 cpu_manager.go:414] "containerMap: removing container" podUID="850c4489-7821-4e37-95f0-258336f5e75e" containerName="payload-processing"
Oct 31 01:52:26.479375 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:52:26.479184    4964 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="850c4489-7821-4e37-95f0-258336f5e75e" containerName="payload-processing"
Oct 31 01:52:26.479375 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:52:26.479190    4964 cpu_manager.go:414] "containerMap: removing container" podUID="79d0fdfe-95ba-4800-9636-68f7afc6829e" containerName="istio-proxy"
Oct 31 01:52:26.479375 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:52:26.479194    4964 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="79d0fdfe-95ba-4800-9636-68f7afc6829e" containerName="istio-proxy"
Oct 31 01:52:26.479375 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:52:26.479201    4964 cpu_manager.go:414] "containerMap: removing container" podUID="778c73f2-648f-4909-a1e9-a685d1d9a2c7" containerName="istio-proxy"
Oct 31 01:52:26.479375 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:52:26.479206    4964 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="778c73f2-648f-4909-a1e9-a685d1d9a2c7" containerName="istio-proxy"
Oct 31 01:52:26.479375 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:52:26.479211    4964 cpu_manager.go:414] "containerMap: removing container" podUID="c3c14a55-5462-4545-afd3-aa589cc162c9" containerName="payload-pre-processing"
Oct 31 01:52:26.479375 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:52:26.479215    4964 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c3c14a55-5462-4545-afd3-aa589cc162c9" containerName="payload-pre-processing"
Oct 31 01:52:26.479375 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:52:26.479224    4964 cpu_manager.go:414] "containerMap: removing container" podUID="7e48bcec-5ffa-4f2b-bb9f-f2fbdc1b3da9" containerName="istio-proxy"
Oct 31 01:52:26.479375 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:52:26.479229    4964 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7e48bcec-5ffa-4f2b-bb9f-f2fbdc1b3da9" containerName="istio-proxy"
Oct 31 01:52:26.479375 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:52:26.479236    4964 cpu_manager.go:414] "containerMap: removing container" podUID="778c73f2-648f-4909-a1e9-a685d1d9a2c7" containerName="istio-proxy"
Oct 31 01:52:26.479375 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:52:26.479240    4964 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="778c73f2-648f-4909-a1e9-a685d1d9a2c7" containerName="istio-proxy"
Oct 31 01:52:26.479375 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:52:26.479254    4964 cpu_manager.go:414] "containerMap: removing container" podUID="a3a77a68-3698-491d-8220-fa9a2c0f8278" containerName="payload-processing"
Oct 31 01:52:26.479375 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:52:26.479259    4964 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a3a77a68-3698-491d-8220-fa9a2c0f8278" containerName="payload-processing"
Oct 31 01:52:26.479375 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:52:26.479271    4964 cpu_manager.go:414] "containerMap: removing container" podUID="dd00f3ae-bb65-4eb4-aada-cce1fab3b363" containerName="istio-proxy"
Oct 31 01:52:26.479375 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:52:26.479275    4964 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dd00f3ae-bb65-4eb4-aada-cce1fab3b363" containerName="istio-proxy"
Oct 31 01:52:26.479375 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:52:26.479281    4964 cpu_manager.go:414] "containerMap: removing container" podUID="ceae1723-877f-4b19-9574-da266b18535f" containerName="maas-api"
Oct 31 01:52:26.479375 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:52:26.479285    4964 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ceae1723-877f-4b19-9574-da266b18535f" containerName="maas-api"
Oct 31 01:52:26.479375 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:52:26.479294    4964 cpu_manager.go:414] "containerMap: removing container" podUID="655b500a-1f9c-424f-9bd2-376464e7151f" containerName="payload-pre-processing"
Oct 31 01:52:26.479375 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:52:26.479299    4964 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="655b500a-1f9c-424f-9bd2-376464e7151f" containerName="payload-pre-processing"
Oct 31 01:52:26.479375 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:52:26.479307    4964 cpu_manager.go:414] "containerMap: removing container" podUID="ceae1723-877f-4b19-9574-da266b18535f" containerName="maas-api"
Oct 31 01:52:26.479375 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:52:26.479311    4964 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ceae1723-877f-4b19-9574-da266b18535f" containerName="maas-api"
Oct 31 01:52:26.479375 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:52:26.479331    4964 cpu_manager.go:414] "containerMap: removing container" podUID="d7751c9b-c8db-4bf9-8193-49c3fefe90e3" containerName="payload-processing"
Oct 31 01:52:26.479375 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:52:26.479335    4964 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d7751c9b-c8db-4bf9-8193-49c3fefe90e3" containerName="payload-processing"
Oct 31 01:52:26.480136 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:52:26.479406    4964 memory_manager.go:365] "RemoveStaleState removing state" podUID="a3a77a68-3698-491d-8220-fa9a2c0f8278" containerName="payload-processing"
Oct 31 01:52:26.480136 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:52:26.479415    4964 memory_manager.go:365] "RemoveStaleState removing state" podUID="11d36add-45f3-4bf4-9372-7d7cec791eff" containerName="istio-proxy"
Oct 31 01:52:26.480136 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:52:26.479422    4964 memory_manager.go:365] "RemoveStaleState removing state" podUID="dd00f3ae-bb65-4eb4-aada-cce1fab3b363" containerName="istio-proxy"
Oct 31 01:52:26.480136 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:52:26.479428    4964 memory_manager.go:365] "RemoveStaleState removing state" podUID="ceae1723-877f-4b19-9574-da266b18535f" containerName="maas-api"
Oct 31 01:52:26.480136 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:52:26.479434    4964 memory_manager.go:365] "RemoveStaleState removing state" podUID="655b500a-1f9c-424f-9bd2-376464e7151f" containerName="payload-pre-processing"
Oct 31 01:52:26.480136 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:52:26.479440    4964 memory_manager.go:365] "RemoveStaleState removing state" podUID="ceae1723-877f-4b19-9574-da266b18535f" containerName="maas-api"
Oct 31 01:52:26.480136 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:52:26.479444    4964 memory_manager.go:365] "RemoveStaleState removing state" podUID="d7751c9b-c8db-4bf9-8193-49c3fefe90e3" containerName="payload-processing"
Oct 31 01:52:26.480136 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:52:26.479449    4964 memory_manager.go:365] "RemoveStaleState removing state" podUID="c4a07c54-1e5b-4aef-b912-dcbc3930ae43" containerName="istio-proxy"
Oct 31 01:52:26.480136 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:52:26.479455    4964 memory_manager.go:365] "RemoveStaleState removing state" podUID="7ca5c214-0290-4db5-9fb1-a9e27cdf9f58" containerName="payload-pre-processing"
Oct 31 01:52:26.480136 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:52:26.479461    4964 memory_manager.go:365] "RemoveStaleState removing state" podUID="79d0fdfe-95ba-4800-9636-68f7afc6829e" containerName="istio-proxy"
Oct 31 01:52:26.480136 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:52:26.479467    4964 memory_manager.go:365] "RemoveStaleState removing state" podUID="850c4489-7821-4e37-95f0-258336f5e75e" containerName="payload-processing"
Oct 31 01:52:26.480136 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:52:26.479472    4964 memory_manager.go:365] "RemoveStaleState removing state" podUID="79d0fdfe-95ba-4800-9636-68f7afc6829e" containerName="istio-proxy"
Oct 31 01:52:26.480136 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:52:26.479477    4964 memory_manager.go:365] "RemoveStaleState removing state" podUID="778c73f2-648f-4909-a1e9-a685d1d9a2c7" containerName="istio-proxy"
Oct 31 01:52:26.480136 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:52:26.479482    4964 memory_manager.go:365] "RemoveStaleState removing state" podUID="c3c14a55-5462-4545-afd3-aa589cc162c9" containerName="payload-pre-processing"
Oct 31 01:52:26.480136 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:52:26.479487    4964 memory_manager.go:365] "RemoveStaleState removing state" podUID="7e48bcec-5ffa-4f2b-bb9f-f2fbdc1b3da9" containerName="istio-proxy"
Oct 31 01:52:26.480136 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:52:26.479492    4964 memory_manager.go:365] "RemoveStaleState removing state" podUID="778c73f2-648f-4909-a1e9-a685d1d9a2c7" containerName="istio-proxy"
Oct 31 01:52:26.482638 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:52:26.482620    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-g5qrb"
Oct 31 01:52:26.554769 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:52:26.554728    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4fbf3cd6-80b7-4f80-b086-f6596fd2c011-istio-envoy\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-g5qrb\" (UID: \"4fbf3cd6-80b7-4f80-b086-f6596fd2c011\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-g5qrb"
Oct 31 01:52:26.554959 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:52:26.554862    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zcfjz\" (UniqueName: \"kubernetes.io/projected/4fbf3cd6-80b7-4f80-b086-f6596fd2c011-kube-api-access-zcfjz\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-g5qrb\" (UID: \"4fbf3cd6-80b7-4f80-b086-f6596fd2c011\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-g5qrb"
Oct 31 01:52:26.554959 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:52:26.554919    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4fbf3cd6-80b7-4f80-b086-f6596fd2c011-istio-data\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-g5qrb\" (UID: \"4fbf3cd6-80b7-4f80-b086-f6596fd2c011\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-g5qrb"
Oct 31 01:52:26.555082 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:52:26.554976    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4fbf3cd6-80b7-4f80-b086-f6596fd2c011-istio-token\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-g5qrb\" (UID: \"4fbf3cd6-80b7-4f80-b086-f6596fd2c011\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-g5qrb"
Oct 31 01:52:26.555139 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:52:26.555081    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4fbf3cd6-80b7-4f80-b086-f6596fd2c011-istiod-ca-cert\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-g5qrb\" (UID: \"4fbf3cd6-80b7-4f80-b086-f6596fd2c011\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-g5qrb"
Oct 31 01:52:26.555139 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:52:26.555111    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4fbf3cd6-80b7-4f80-b086-f6596fd2c011-istio-podinfo\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-g5qrb\" (UID: \"4fbf3cd6-80b7-4f80-b086-f6596fd2c011\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-g5qrb"
Oct 31 01:52:26.555244 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:52:26.555187    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4fbf3cd6-80b7-4f80-b086-f6596fd2c011-workload-certs\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-g5qrb\" (UID: \"4fbf3cd6-80b7-4f80-b086-f6596fd2c011\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-g5qrb"
Oct 31 01:52:26.555244 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:52:26.555229    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4fbf3cd6-80b7-4f80-b086-f6596fd2c011-workload-socket\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-g5qrb\" (UID: \"4fbf3cd6-80b7-4f80-b086-f6596fd2c011\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-g5qrb"
Oct 31 01:52:26.555402 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:52:26.555265    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4fbf3cd6-80b7-4f80-b086-f6596fd2c011-credential-socket\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-g5qrb\" (UID: \"4fbf3cd6-80b7-4f80-b086-f6596fd2c011\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-g5qrb"
Oct 31 01:52:26.584122 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:52:26.584091    4964 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-00e840-openshift-default-dockercfg-2nfqp\""
Oct 31 01:52:26.588787 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:52:26.588757    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-g5qrb"]
Oct 31 01:52:26.656866 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:52:26.656831    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4fbf3cd6-80b7-4f80-b086-f6596fd2c011-workload-certs\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-g5qrb\" (UID: \"4fbf3cd6-80b7-4f80-b086-f6596fd2c011\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-g5qrb"
Oct 31 01:52:26.657104 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:52:26.656876    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4fbf3cd6-80b7-4f80-b086-f6596fd2c011-workload-socket\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-g5qrb\" (UID: \"4fbf3cd6-80b7-4f80-b086-f6596fd2c011\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-g5qrb"
Oct 31 01:52:26.657104 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:52:26.656900    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4fbf3cd6-80b7-4f80-b086-f6596fd2c011-credential-socket\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-g5qrb\" (UID: \"4fbf3cd6-80b7-4f80-b086-f6596fd2c011\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-g5qrb"
Oct 31 01:52:26.657104 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:52:26.656925    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4fbf3cd6-80b7-4f80-b086-f6596fd2c011-istio-envoy\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-g5qrb\" (UID: \"4fbf3cd6-80b7-4f80-b086-f6596fd2c011\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-g5qrb"
Oct 31 01:52:26.657104 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:52:26.656968    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zcfjz\" (UniqueName: \"kubernetes.io/projected/4fbf3cd6-80b7-4f80-b086-f6596fd2c011-kube-api-access-zcfjz\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-g5qrb\" (UID: \"4fbf3cd6-80b7-4f80-b086-f6596fd2c011\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-g5qrb"
Oct 31 01:52:26.657104 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:52:26.656998    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4fbf3cd6-80b7-4f80-b086-f6596fd2c011-istio-data\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-g5qrb\" (UID: \"4fbf3cd6-80b7-4f80-b086-f6596fd2c011\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-g5qrb"
Oct 31 01:52:26.657104 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:52:26.657046    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4fbf3cd6-80b7-4f80-b086-f6596fd2c011-istio-token\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-g5qrb\" (UID: \"4fbf3cd6-80b7-4f80-b086-f6596fd2c011\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-g5qrb"
Oct 31 01:52:26.657469 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:52:26.657128    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4fbf3cd6-80b7-4f80-b086-f6596fd2c011-istiod-ca-cert\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-g5qrb\" (UID: \"4fbf3cd6-80b7-4f80-b086-f6596fd2c011\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-g5qrb"
Oct 31 01:52:26.657469 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:52:26.657147    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4fbf3cd6-80b7-4f80-b086-f6596fd2c011-istio-podinfo\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-g5qrb\" (UID: \"4fbf3cd6-80b7-4f80-b086-f6596fd2c011\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-g5qrb"
Oct 31 01:52:26.657469 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:52:26.657368    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4fbf3cd6-80b7-4f80-b086-f6596fd2c011-workload-certs\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-g5qrb\" (UID: \"4fbf3cd6-80b7-4f80-b086-f6596fd2c011\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-g5qrb"
Oct 31 01:52:26.657614 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:52:26.657465    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4fbf3cd6-80b7-4f80-b086-f6596fd2c011-credential-socket\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-g5qrb\" (UID: \"4fbf3cd6-80b7-4f80-b086-f6596fd2c011\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-g5qrb"
Oct 31 01:52:26.657614 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:52:26.657561    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4fbf3cd6-80b7-4f80-b086-f6596fd2c011-workload-socket\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-g5qrb\" (UID: \"4fbf3cd6-80b7-4f80-b086-f6596fd2c011\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-g5qrb"
Oct 31 01:52:26.657710 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:52:26.657612    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4fbf3cd6-80b7-4f80-b086-f6596fd2c011-istio-data\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-g5qrb\" (UID: \"4fbf3cd6-80b7-4f80-b086-f6596fd2c011\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-g5qrb"
Oct 31 01:52:26.658030 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:52:26.657991    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4fbf3cd6-80b7-4f80-b086-f6596fd2c011-istiod-ca-cert\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-g5qrb\" (UID: \"4fbf3cd6-80b7-4f80-b086-f6596fd2c011\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-g5qrb"
Oct 31 01:52:26.659705 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:52:26.659677    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4fbf3cd6-80b7-4f80-b086-f6596fd2c011-istio-envoy\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-g5qrb\" (UID: \"4fbf3cd6-80b7-4f80-b086-f6596fd2c011\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-g5qrb"
Oct 31 01:52:26.659854 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:52:26.659839    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4fbf3cd6-80b7-4f80-b086-f6596fd2c011-istio-podinfo\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-g5qrb\" (UID: \"4fbf3cd6-80b7-4f80-b086-f6596fd2c011\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-g5qrb"
Oct 31 01:52:26.715057 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:52:26.715011    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4fbf3cd6-80b7-4f80-b086-f6596fd2c011-istio-token\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-g5qrb\" (UID: \"4fbf3cd6-80b7-4f80-b086-f6596fd2c011\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-g5qrb"
Oct 31 01:52:26.715222 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:52:26.715072    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zcfjz\" (UniqueName: \"kubernetes.io/projected/4fbf3cd6-80b7-4f80-b086-f6596fd2c011-kube-api-access-zcfjz\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-g5qrb\" (UID: \"4fbf3cd6-80b7-4f80-b086-f6596fd2c011\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-g5qrb"
Oct 31 01:52:26.797375 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:52:26.797243    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-g5qrb"
Oct 31 01:52:27.237010 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:52:27.236980    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-g5qrb"]
Oct 31 01:52:27.238149 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:52:27.238110    4964 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4fbf3cd6_80b7_4f80_b086_f6596fd2c011.slice/crio-48505fa47db066eb720946ad14644693f4a2314db8e14a1617c4437031867f03 WatchSource:0}: Error finding container 48505fa47db066eb720946ad14644693f4a2314db8e14a1617c4437031867f03: Status 404 returned error can't find the container with id 48505fa47db066eb720946ad14644693f4a2314db8e14a1617c4437031867f03
Oct 31 01:52:27.240707 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:52:27.240677    4964 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 01:52:27.240801 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:52:27.240743    4964 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 01:52:27.240801 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:52:27.240772    4964 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 01:52:27.353675 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:52:27.353629    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-g5qrb" event={"ID":"4fbf3cd6-80b7-4f80-b086-f6596fd2c011","Type":"ContainerStarted","Data":"7f988f23592cf5bba8883479cb6cd27449d15df86ed32d13f015e5662e2c1ed5"}
Oct 31 01:52:27.353863 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:52:27.353696    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-g5qrb" event={"ID":"4fbf3cd6-80b7-4f80-b086-f6596fd2c011","Type":"ContainerStarted","Data":"48505fa47db066eb720946ad14644693f4a2314db8e14a1617c4437031867f03"}
Oct 31 01:52:27.473454 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:52:27.473380    4964 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-g5qrb" podStartSLOduration=1.473362382 podStartE2EDuration="1.473362382s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:52:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:52:27.470296386 +0000 UTC m=+2915.946862779" watchObservedRunningTime="2026-10-31 01:52:27.473362382 +0000 UTC m=+2915.949928765"
Oct 31 01:52:27.810340 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:52:27.801567    4964 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-g5qrb"
Oct 31 01:52:28.813350 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:52:28.808415    4964 patch_prober.go:29] interesting pod/e2e-worker-w2-00e840-openshift-default-59b96fb879-g5qrb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" start-of-body=
Oct 31 01:52:28.813350 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:52:28.808485    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-g5qrb" podUID="4fbf3cd6-80b7-4f80-b086-f6596fd2c011" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused"
Oct 31 01:52:28.813350 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:52:28.809302    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:52:29.814356 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:52:29.813731    4964 patch_prober.go:29] interesting pod/e2e-worker-w2-00e840-openshift-default-59b96fb879-g5qrb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" start-of-body=
Oct 31 01:52:29.814356 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:52:29.813813    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-g5qrb" podUID="4fbf3cd6-80b7-4f80-b086-f6596fd2c011" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused"
Oct 31 01:52:30.176711 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:52:30.176599    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:52:30.634203 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:52:30.634164    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:52:30.799458 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:52:30.799408    4964 patch_prober.go:29] interesting pod/e2e-worker-w2-00e840-openshift-default-59b96fb879-g5qrb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" start-of-body=
Oct 31 01:52:30.799742 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:52:30.799712    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-g5qrb" podUID="4fbf3cd6-80b7-4f80-b086-f6596fd2c011" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused"
Oct 31 01:52:31.812775 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:52:31.812738    4964 patch_prober.go:29] interesting pod/e2e-worker-w2-00e840-openshift-default-59b96fb879-g5qrb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" start-of-body=
Oct 31 01:52:31.813435 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:52:31.813401    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-g5qrb" podUID="4fbf3cd6-80b7-4f80-b086-f6596fd2c011" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused"
Oct 31 01:52:32.644514 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:52:32.644474    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:52:32.649798 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:52:32.649770    4964 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-b2d49b-openshift-default-d7d8667b7-69jgz"]
Oct 31 01:52:32.649974 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:52:32.649956    4964 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-b2d49b-openshift-default-d7d8667b7-69jgz" podUID="710b2ea2-bffd-45b2-b0fc-0e5a90defed1" containerName="istio-proxy" containerID="cri-o://572ce9e7a635c5c4063309cc43dbe0dc8a8d201675a8db0c67b5c4e6237af0d0" gracePeriod=30
Oct 31 01:52:32.804877 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:52:32.804842    4964 patch_prober.go:29] interesting pod/e2e-worker-w2-00e840-openshift-default-59b96fb879-g5qrb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" start-of-body=
Oct 31 01:52:32.805187 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:52:32.805151    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-g5qrb" podUID="4fbf3cd6-80b7-4f80-b086-f6596fd2c011" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused"
Oct 31 01:52:33.797657 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:52:33.797623    4964 patch_prober.go:29] interesting pod/e2e-worker-w2-00e840-openshift-default-59b96fb879-g5qrb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" start-of-body=
Oct 31 01:52:33.798081 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:52:33.797685    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-g5qrb" podUID="4fbf3cd6-80b7-4f80-b086-f6596fd2c011" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused"
Oct 31 01:52:34.193596 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:52:34.193490    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:52:34.822955 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:52:34.822923    4964 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-g5qrb"
Oct 31 01:52:34.826077 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:52:34.826049    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-g5qrb"
Oct 31 01:52:34.826426 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:52:34.826407    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-g5qrb"
Oct 31 01:52:34.838177 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:52:34.838060    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:52:35.178970 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:52:35.178864    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:52:35.519722 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:52:35.519685    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:52:37.921385 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:52:37.921285    4964 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-b2d49b-openshift-default-d7d8667b7-69jgz"
Oct 31 01:52:38.025297 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:52:38.025259    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/710b2ea2-bffd-45b2-b0fc-0e5a90defed1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/710b2ea2-bffd-45b2-b0fc-0e5a90defed1-istio-envoy\") pod \"710b2ea2-bffd-45b2-b0fc-0e5a90defed1\" (UID: \"710b2ea2-bffd-45b2-b0fc-0e5a90defed1\") "
Oct 31 01:52:38.025549 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:52:38.025354    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/710b2ea2-bffd-45b2-b0fc-0e5a90defed1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/710b2ea2-bffd-45b2-b0fc-0e5a90defed1-istiod-ca-cert\") pod \"710b2ea2-bffd-45b2-b0fc-0e5a90defed1\" (UID: \"710b2ea2-bffd-45b2-b0fc-0e5a90defed1\") "
Oct 31 01:52:38.025549 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:52:38.025429    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/710b2ea2-bffd-45b2-b0fc-0e5a90defed1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/710b2ea2-bffd-45b2-b0fc-0e5a90defed1-credential-socket\") pod \"710b2ea2-bffd-45b2-b0fc-0e5a90defed1\" (UID: \"710b2ea2-bffd-45b2-b0fc-0e5a90defed1\") "
Oct 31 01:52:38.025549 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:52:38.025457    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/710b2ea2-bffd-45b2-b0fc-0e5a90defed1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/710b2ea2-bffd-45b2-b0fc-0e5a90defed1-istio-data\") pod \"710b2ea2-bffd-45b2-b0fc-0e5a90defed1\" (UID: \"710b2ea2-bffd-45b2-b0fc-0e5a90defed1\") "
Oct 31 01:52:38.025549 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:52:38.025530    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/710b2ea2-bffd-45b2-b0fc-0e5a90defed1-istio-token\" (UniqueName: \"kubernetes.io/projected/710b2ea2-bffd-45b2-b0fc-0e5a90defed1-istio-token\") pod \"710b2ea2-bffd-45b2-b0fc-0e5a90defed1\" (UID: \"710b2ea2-bffd-45b2-b0fc-0e5a90defed1\") "
Oct 31 01:52:38.025844 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:52:38.025558    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/710b2ea2-bffd-45b2-b0fc-0e5a90defed1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/710b2ea2-bffd-45b2-b0fc-0e5a90defed1-workload-certs\") pod \"710b2ea2-bffd-45b2-b0fc-0e5a90defed1\" (UID: \"710b2ea2-bffd-45b2-b0fc-0e5a90defed1\") "
Oct 31 01:52:38.025844 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:52:38.025591    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/710b2ea2-bffd-45b2-b0fc-0e5a90defed1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/710b2ea2-bffd-45b2-b0fc-0e5a90defed1-workload-socket\") pod \"710b2ea2-bffd-45b2-b0fc-0e5a90defed1\" (UID: \"710b2ea2-bffd-45b2-b0fc-0e5a90defed1\") "
Oct 31 01:52:38.025844 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:52:38.025648    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/710b2ea2-bffd-45b2-b0fc-0e5a90defed1-kube-api-access-2mxkl\" (UniqueName: \"kubernetes.io/projected/710b2ea2-bffd-45b2-b0fc-0e5a90defed1-kube-api-access-2mxkl\") pod \"710b2ea2-bffd-45b2-b0fc-0e5a90defed1\" (UID: \"710b2ea2-bffd-45b2-b0fc-0e5a90defed1\") "
Oct 31 01:52:38.025844 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:52:38.025742    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/710b2ea2-bffd-45b2-b0fc-0e5a90defed1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/710b2ea2-bffd-45b2-b0fc-0e5a90defed1-istio-podinfo\") pod \"710b2ea2-bffd-45b2-b0fc-0e5a90defed1\" (UID: \"710b2ea2-bffd-45b2-b0fc-0e5a90defed1\") "
Oct 31 01:52:38.025844 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:52:38.025793    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/710b2ea2-bffd-45b2-b0fc-0e5a90defed1-credential-socket" pod "710b2ea2-bffd-45b2-b0fc-0e5a90defed1" (UID: "710b2ea2-bffd-45b2-b0fc-0e5a90defed1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:52:38.026120 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:52:38.025869    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/710b2ea2-bffd-45b2-b0fc-0e5a90defed1-istiod-ca-cert" pod "710b2ea2-bffd-45b2-b0fc-0e5a90defed1" (UID: "710b2ea2-bffd-45b2-b0fc-0e5a90defed1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:52:38.026120 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:52:38.026100    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/710b2ea2-bffd-45b2-b0fc-0e5a90defed1-istio-data" pod "710b2ea2-bffd-45b2-b0fc-0e5a90defed1" (UID: "710b2ea2-bffd-45b2-b0fc-0e5a90defed1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:52:38.026242 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:52:38.026194    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/710b2ea2-bffd-45b2-b0fc-0e5a90defed1-workload-certs" pod "710b2ea2-bffd-45b2-b0fc-0e5a90defed1" (UID: "710b2ea2-bffd-45b2-b0fc-0e5a90defed1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:52:38.026491 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:52:38.026463    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/710b2ea2-bffd-45b2-b0fc-0e5a90defed1-workload-socket" pod "710b2ea2-bffd-45b2-b0fc-0e5a90defed1" (UID: "710b2ea2-bffd-45b2-b0fc-0e5a90defed1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:52:38.026491 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:52:38.026481    4964 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/710b2ea2-bffd-45b2-b0fc-0e5a90defed1-istiod-ca-cert\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:52:38.026778 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:52:38.026508    4964 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/710b2ea2-bffd-45b2-b0fc-0e5a90defed1-credential-socket\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:52:38.026778 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:52:38.026524    4964 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/710b2ea2-bffd-45b2-b0fc-0e5a90defed1-istio-data\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:52:38.026778 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:52:38.026537    4964 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/710b2ea2-bffd-45b2-b0fc-0e5a90defed1-workload-certs\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:52:38.028110 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:52:38.028069    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/710b2ea2-bffd-45b2-b0fc-0e5a90defed1-istio-envoy" pod "710b2ea2-bffd-45b2-b0fc-0e5a90defed1" (UID: "710b2ea2-bffd-45b2-b0fc-0e5a90defed1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:52:38.028230 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:52:38.028171    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/710b2ea2-bffd-45b2-b0fc-0e5a90defed1-kube-api-access-2mxkl" pod "710b2ea2-bffd-45b2-b0fc-0e5a90defed1" (UID: "710b2ea2-bffd-45b2-b0fc-0e5a90defed1"). InnerVolumeSpecName "kube-api-access-2mxkl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:52:38.028294 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:52:38.028241    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/710b2ea2-bffd-45b2-b0fc-0e5a90defed1-istio-token" pod "710b2ea2-bffd-45b2-b0fc-0e5a90defed1" (UID: "710b2ea2-bffd-45b2-b0fc-0e5a90defed1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:52:38.028468 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:52:38.028445    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/710b2ea2-bffd-45b2-b0fc-0e5a90defed1-istio-podinfo" pod "710b2ea2-bffd-45b2-b0fc-0e5a90defed1" (UID: "710b2ea2-bffd-45b2-b0fc-0e5a90defed1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 01:52:38.127296 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:52:38.127262    4964 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/710b2ea2-bffd-45b2-b0fc-0e5a90defed1-istio-token\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:52:38.127296 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:52:38.127290    4964 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/710b2ea2-bffd-45b2-b0fc-0e5a90defed1-workload-socket\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:52:38.127296 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:52:38.127300    4964 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2mxkl\" (UniqueName: \"kubernetes.io/projected/710b2ea2-bffd-45b2-b0fc-0e5a90defed1-kube-api-access-2mxkl\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:52:38.127558 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:52:38.127309    4964 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/710b2ea2-bffd-45b2-b0fc-0e5a90defed1-istio-podinfo\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:52:38.127558 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:52:38.127339    4964 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/710b2ea2-bffd-45b2-b0fc-0e5a90defed1-istio-envoy\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:52:38.477480 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:52:38.477440    4964 generic.go:356] "Generic (PLEG): container finished" podID="710b2ea2-bffd-45b2-b0fc-0e5a90defed1" containerID="572ce9e7a635c5c4063309cc43dbe0dc8a8d201675a8db0c67b5c4e6237af0d0" exitCode=0
Oct 31 01:52:38.477718 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:52:38.477513    4964 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-b2d49b-openshift-default-d7d8667b7-69jgz"
Oct 31 01:52:38.477718 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:52:38.477537    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-b2d49b-openshift-default-d7d8667b7-69jgz" event={"ID":"710b2ea2-bffd-45b2-b0fc-0e5a90defed1","Type":"ContainerDied","Data":"572ce9e7a635c5c4063309cc43dbe0dc8a8d201675a8db0c67b5c4e6237af0d0"}
Oct 31 01:52:38.477718 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:52:38.477597    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-b2d49b-openshift-default-d7d8667b7-69jgz" event={"ID":"710b2ea2-bffd-45b2-b0fc-0e5a90defed1","Type":"ContainerDied","Data":"767bc2238c8567081847853833d623827f63f6bdf23efeb09e280f03c63877ca"}
Oct 31 01:52:38.477718 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:52:38.477624    4964 scope.go:122] "RemoveContainer" containerID="572ce9e7a635c5c4063309cc43dbe0dc8a8d201675a8db0c67b5c4e6237af0d0"
Oct 31 01:52:38.489081 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:52:38.489058    4964 scope.go:122] "RemoveContainer" containerID="f974e97c15eabf759974ed24a834e0f80d2dc04c22051c94739c2369d33357fb"
Oct 31 01:52:38.499263 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:52:38.499243    4964 scope.go:122] "RemoveContainer" containerID="572ce9e7a635c5c4063309cc43dbe0dc8a8d201675a8db0c67b5c4e6237af0d0"
Oct 31 01:52:38.499633 ip-10-0-133-197 kubenswrapper[4964]: E1031 01:52:38.499613    4964 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"572ce9e7a635c5c4063309cc43dbe0dc8a8d201675a8db0c67b5c4e6237af0d0\": container with ID starting with 572ce9e7a635c5c4063309cc43dbe0dc8a8d201675a8db0c67b5c4e6237af0d0 not found: ID does not exist" containerID="572ce9e7a635c5c4063309cc43dbe0dc8a8d201675a8db0c67b5c4e6237af0d0"
Oct 31 01:52:38.499691 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:52:38.499643    4964 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"572ce9e7a635c5c4063309cc43dbe0dc8a8d201675a8db0c67b5c4e6237af0d0"} err="failed to get container status \"572ce9e7a635c5c4063309cc43dbe0dc8a8d201675a8db0c67b5c4e6237af0d0\": rpc error: code = NotFound desc = could not find container \"572ce9e7a635c5c4063309cc43dbe0dc8a8d201675a8db0c67b5c4e6237af0d0\": container with ID starting with 572ce9e7a635c5c4063309cc43dbe0dc8a8d201675a8db0c67b5c4e6237af0d0 not found: ID does not exist"
Oct 31 01:52:38.499691 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:52:38.499662    4964 scope.go:122] "RemoveContainer" containerID="f974e97c15eabf759974ed24a834e0f80d2dc04c22051c94739c2369d33357fb"
Oct 31 01:52:38.500012 ip-10-0-133-197 kubenswrapper[4964]: E1031 01:52:38.499979    4964 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f974e97c15eabf759974ed24a834e0f80d2dc04c22051c94739c2369d33357fb\": container with ID starting with f974e97c15eabf759974ed24a834e0f80d2dc04c22051c94739c2369d33357fb not found: ID does not exist" containerID="f974e97c15eabf759974ed24a834e0f80d2dc04c22051c94739c2369d33357fb"
Oct 31 01:52:38.500111 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:52:38.500020    4964 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f974e97c15eabf759974ed24a834e0f80d2dc04c22051c94739c2369d33357fb"} err="failed to get container status \"f974e97c15eabf759974ed24a834e0f80d2dc04c22051c94739c2369d33357fb\": rpc error: code = NotFound desc = could not find container \"f974e97c15eabf759974ed24a834e0f80d2dc04c22051c94739c2369d33357fb\": container with ID starting with f974e97c15eabf759974ed24a834e0f80d2dc04c22051c94739c2369d33357fb not found: ID does not exist"
Oct 31 01:52:38.644938 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:52:38.644909    4964 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-b2d49b-openshift-default-d7d8667b7-69jgz"]
Oct 31 01:52:38.649762 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:52:38.649736    4964 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-b2d49b-openshift-default-d7d8667b7-69jgz"]
Oct 31 01:52:39.433349 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:52:39.432836    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:52:39.621068 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:52:39.621030    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:52:39.777768 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:52:39.777733    4964 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="710b2ea2-bffd-45b2-b0fc-0e5a90defed1" path="/var/lib/kubelet/pods/710b2ea2-bffd-45b2-b0fc-0e5a90defed1/volumes"
Oct 31 01:52:39.786833 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:52:39.786805    4964 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-2c0de9-5ff79799cc-hj9vt"]
Oct 31 01:52:39.787038 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:52:39.787017    4964 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-2c0de9-5ff79799cc-hj9vt" podUID="b073d0f5-5121-480d-b530-64568abb47f5" containerName="maas-api" containerID="cri-o://eacc8fa63a3e54bcad6ebe77bf10e9bd9312d530a72990fd574f9d7cf8ca7096" gracePeriod=30
Oct 31 01:52:39.807451 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:52:39.807423    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:52:39.995230 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:52:39.995189    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:52:40.081240 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:52:40.081214    4964 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-2c0de9-5ff79799cc-hj9vt"
Oct 31 01:52:40.157486 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:52:40.157440    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b073d0f5-5121-480d-b530-64568abb47f5-kube-api-access-wljhb\" (UniqueName: \"kubernetes.io/projected/b073d0f5-5121-480d-b530-64568abb47f5-kube-api-access-wljhb\") pod \"b073d0f5-5121-480d-b530-64568abb47f5\" (UID: \"b073d0f5-5121-480d-b530-64568abb47f5\") "
Oct 31 01:52:40.157701 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:52:40.157508    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b073d0f5-5121-480d-b530-64568abb47f5-metrics-certs\" (UniqueName: \"kubernetes.io/secret/b073d0f5-5121-480d-b530-64568abb47f5-metrics-certs\") pod \"b073d0f5-5121-480d-b530-64568abb47f5\" (UID: \"b073d0f5-5121-480d-b530-64568abb47f5\") "
Oct 31 01:52:40.157701 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:52:40.157530    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b073d0f5-5121-480d-b530-64568abb47f5-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b073d0f5-5121-480d-b530-64568abb47f5-maas-api-tls\") pod \"b073d0f5-5121-480d-b530-64568abb47f5\" (UID: \"b073d0f5-5121-480d-b530-64568abb47f5\") "
Oct 31 01:52:40.157701 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:52:40.157642    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b073d0f5-5121-480d-b530-64568abb47f5-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b073d0f5-5121-480d-b530-64568abb47f5-openshift-service-ca\") pod \"b073d0f5-5121-480d-b530-64568abb47f5\" (UID: \"b073d0f5-5121-480d-b530-64568abb47f5\") "
Oct 31 01:52:40.158094 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:52:40.158067    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b073d0f5-5121-480d-b530-64568abb47f5-openshift-service-ca" pod "b073d0f5-5121-480d-b530-64568abb47f5" (UID: "b073d0f5-5121-480d-b530-64568abb47f5"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:52:40.159870 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:52:40.159839    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b073d0f5-5121-480d-b530-64568abb47f5-kube-api-access-wljhb" pod "b073d0f5-5121-480d-b530-64568abb47f5" (UID: "b073d0f5-5121-480d-b530-64568abb47f5"). InnerVolumeSpecName "kube-api-access-wljhb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:52:40.160242 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:52:40.160218    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b073d0f5-5121-480d-b530-64568abb47f5-metrics-certs" pod "b073d0f5-5121-480d-b530-64568abb47f5" (UID: "b073d0f5-5121-480d-b530-64568abb47f5"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:52:40.160242 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:52:40.160236    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b073d0f5-5121-480d-b530-64568abb47f5-maas-api-tls" pod "b073d0f5-5121-480d-b530-64568abb47f5" (UID: "b073d0f5-5121-480d-b530-64568abb47f5"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:52:40.259034 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:52:40.259000    4964 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b073d0f5-5121-480d-b530-64568abb47f5-openshift-service-ca\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:52:40.259034 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:52:40.259034    4964 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wljhb\" (UniqueName: \"kubernetes.io/projected/b073d0f5-5121-480d-b530-64568abb47f5-kube-api-access-wljhb\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:52:40.259304 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:52:40.259048    4964 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b073d0f5-5121-480d-b530-64568abb47f5-metrics-certs\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:52:40.259304 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:52:40.259058    4964 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b073d0f5-5121-480d-b530-64568abb47f5-maas-api-tls\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:52:40.493292 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:52:40.493253    4964 generic.go:356] "Generic (PLEG): container finished" podID="b073d0f5-5121-480d-b530-64568abb47f5" containerID="eacc8fa63a3e54bcad6ebe77bf10e9bd9312d530a72990fd574f9d7cf8ca7096" exitCode=0
Oct 31 01:52:40.493854 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:52:40.493358    4964 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-2c0de9-5ff79799cc-hj9vt"
Oct 31 01:52:40.493854 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:52:40.493357    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-2c0de9-5ff79799cc-hj9vt" event={"ID":"b073d0f5-5121-480d-b530-64568abb47f5","Type":"ContainerDied","Data":"eacc8fa63a3e54bcad6ebe77bf10e9bd9312d530a72990fd574f9d7cf8ca7096"}
Oct 31 01:52:40.493854 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:52:40.493447    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-2c0de9-5ff79799cc-hj9vt" event={"ID":"b073d0f5-5121-480d-b530-64568abb47f5","Type":"ContainerDied","Data":"322676be770c258a8b8d7484bb60aba76daea42c0df39cc8eacf1ba0be29a125"}
Oct 31 01:52:40.493854 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:52:40.493478    4964 scope.go:122] "RemoveContainer" containerID="eacc8fa63a3e54bcad6ebe77bf10e9bd9312d530a72990fd574f9d7cf8ca7096"
Oct 31 01:52:40.504641 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:52:40.504618    4964 scope.go:122] "RemoveContainer" containerID="eacc8fa63a3e54bcad6ebe77bf10e9bd9312d530a72990fd574f9d7cf8ca7096"
Oct 31 01:52:40.504967 ip-10-0-133-197 kubenswrapper[4964]: E1031 01:52:40.504946    4964 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eacc8fa63a3e54bcad6ebe77bf10e9bd9312d530a72990fd574f9d7cf8ca7096\": container with ID starting with eacc8fa63a3e54bcad6ebe77bf10e9bd9312d530a72990fd574f9d7cf8ca7096 not found: ID does not exist" containerID="eacc8fa63a3e54bcad6ebe77bf10e9bd9312d530a72990fd574f9d7cf8ca7096"
Oct 31 01:52:40.505030 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:52:40.504979    4964 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eacc8fa63a3e54bcad6ebe77bf10e9bd9312d530a72990fd574f9d7cf8ca7096"} err="failed to get container status \"eacc8fa63a3e54bcad6ebe77bf10e9bd9312d530a72990fd574f9d7cf8ca7096\": rpc error: code = NotFound desc = could not find container \"eacc8fa63a3e54bcad6ebe77bf10e9bd9312d530a72990fd574f9d7cf8ca7096\": container with ID starting with eacc8fa63a3e54bcad6ebe77bf10e9bd9312d530a72990fd574f9d7cf8ca7096 not found: ID does not exist"
Oct 31 01:52:40.657485 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:52:40.657453    4964 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-2c0de9-5ff79799cc-hj9vt"]
Oct 31 01:52:40.660401 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:52:40.660372    4964 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-2c0de9-5ff79799cc-hj9vt"]
Oct 31 01:52:41.241709 ip-10-0-133-197 kubenswrapper[4964]: E1031 01:52:41.241665    4964 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q56x5" containerName="extractor"
Oct 31 01:52:41.778868 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:52:41.778831    4964 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b073d0f5-5121-480d-b530-64568abb47f5" path="/var/lib/kubelet/pods/b073d0f5-5121-480d-b530-64568abb47f5/volumes"
Oct 31 01:52:41.817477 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:52:41.817442    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:52:42.302872 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:52:42.302836    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:52:42.574143 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:52:42.574056    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:52:43.242558 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:52:43.242520    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:52:46.120097 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:52:46.120065    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:52:46.290241 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:52:46.290206    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:52:46.495751 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:52:46.495712    4964 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-3b7279-6884d85fbc-fhmqm"]
Oct 31 01:52:46.496398 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:52:46.496303    4964 cpu_manager.go:414] "containerMap: removing container" podUID="11d36add-45f3-4bf4-9372-7d7cec791eff" containerName="istio-proxy"
Oct 31 01:52:46.496398 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:52:46.496341    4964 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="11d36add-45f3-4bf4-9372-7d7cec791eff" containerName="istio-proxy"
Oct 31 01:52:46.496583 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:52:46.496406    4964 cpu_manager.go:414] "containerMap: removing container" podUID="b073d0f5-5121-480d-b530-64568abb47f5" containerName="maas-api"
Oct 31 01:52:46.496583 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:52:46.496414    4964 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b073d0f5-5121-480d-b530-64568abb47f5" containerName="maas-api"
Oct 31 01:52:46.496583 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:52:46.496432    4964 cpu_manager.go:414] "containerMap: removing container" podUID="710b2ea2-bffd-45b2-b0fc-0e5a90defed1" containerName="istio-proxy"
Oct 31 01:52:46.496583 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:52:46.496439    4964 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="710b2ea2-bffd-45b2-b0fc-0e5a90defed1" containerName="istio-proxy"
Oct 31 01:52:46.496583 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:52:46.496465    4964 cpu_manager.go:414] "containerMap: removing container" podUID="710b2ea2-bffd-45b2-b0fc-0e5a90defed1" containerName="istio-proxy"
Oct 31 01:52:46.496583 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:52:46.496473    4964 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="710b2ea2-bffd-45b2-b0fc-0e5a90defed1" containerName="istio-proxy"
Oct 31 01:52:46.496873 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:52:46.496632    4964 memory_manager.go:365] "RemoveStaleState removing state" podUID="710b2ea2-bffd-45b2-b0fc-0e5a90defed1" containerName="istio-proxy"
Oct 31 01:52:46.496873 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:52:46.496647    4964 memory_manager.go:365] "RemoveStaleState removing state" podUID="11d36add-45f3-4bf4-9372-7d7cec791eff" containerName="istio-proxy"
Oct 31 01:52:46.496873 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:52:46.496660    4964 memory_manager.go:365] "RemoveStaleState removing state" podUID="710b2ea2-bffd-45b2-b0fc-0e5a90defed1" containerName="istio-proxy"
Oct 31 01:52:46.496873 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:52:46.496679    4964 memory_manager.go:365] "RemoveStaleState removing state" podUID="b073d0f5-5121-480d-b530-64568abb47f5" containerName="maas-api"
Oct 31 01:52:46.501674 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:52:46.501654    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-3b7279-6884d85fbc-fhmqm"
Oct 31 01:52:46.559516 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:52:46.559487    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-3b7279-6884d85fbc-fhmqm"]
Oct 31 01:52:46.604253 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:52:46.604227    4964 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-a-3b7279\""
Oct 31 01:52:46.604519 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:52:46.604232    4964 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-a-3b7279-dockercfg-zw2hq\""
Oct 31 01:52:46.633649 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:52:46.633621    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b1297c54-10e4-4992-9ae3-4d8d628dd0a2-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-3b7279-6884d85fbc-fhmqm\" (UID: \"b1297c54-10e4-4992-9ae3-4d8d628dd0a2\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-3b7279-6884d85fbc-fhmqm"
Oct 31 01:52:46.633802 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:52:46.633666    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2j222\" (UniqueName: \"kubernetes.io/projected/b1297c54-10e4-4992-9ae3-4d8d628dd0a2-kube-api-access-2j222\") pod \"payload-pre-processing-e2e-praxis-a-3b7279-6884d85fbc-fhmqm\" (UID: \"b1297c54-10e4-4992-9ae3-4d8d628dd0a2\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-3b7279-6884d85fbc-fhmqm"
Oct 31 01:52:46.657326 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:52:46.657296    4964 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-3b7279-667b9df887-n49j9"]
Oct 31 01:52:46.661356 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:52:46.661293    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-3b7279-667b9df887-n49j9"
Oct 31 01:52:46.707534 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:52:46.707507    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-3b7279-667b9df887-n49j9"]
Oct 31 01:52:46.735138 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:52:46.735113    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2j222\" (UniqueName: \"kubernetes.io/projected/b1297c54-10e4-4992-9ae3-4d8d628dd0a2-kube-api-access-2j222\") pod \"payload-pre-processing-e2e-praxis-a-3b7279-6884d85fbc-fhmqm\" (UID: \"b1297c54-10e4-4992-9ae3-4d8d628dd0a2\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-3b7279-6884d85fbc-fhmqm"
Oct 31 01:52:46.735367 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:52:46.735349    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b1297c54-10e4-4992-9ae3-4d8d628dd0a2-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-3b7279-6884d85fbc-fhmqm\" (UID: \"b1297c54-10e4-4992-9ae3-4d8d628dd0a2\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-3b7279-6884d85fbc-fhmqm"
Oct 31 01:52:46.736223 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:52:46.736201    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b1297c54-10e4-4992-9ae3-4d8d628dd0a2-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-3b7279-6884d85fbc-fhmqm\" (UID: \"b1297c54-10e4-4992-9ae3-4d8d628dd0a2\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-3b7279-6884d85fbc-fhmqm"
Oct 31 01:52:46.792358 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:52:46.792263    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2j222\" (UniqueName: \"kubernetes.io/projected/b1297c54-10e4-4992-9ae3-4d8d628dd0a2-kube-api-access-2j222\") pod \"payload-pre-processing-e2e-praxis-a-3b7279-6884d85fbc-fhmqm\" (UID: \"b1297c54-10e4-4992-9ae3-4d8d628dd0a2\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-3b7279-6884d85fbc-fhmqm"
Oct 31 01:52:46.815721 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:52:46.815696    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-3b7279-6884d85fbc-fhmqm"
Oct 31 01:52:46.836656 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:52:46.836631    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b6ecd194-e6e7-4b66-bdc3-b6166efd0045-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-3b7279-667b9df887-n49j9\" (UID: \"b6ecd194-e6e7-4b66-bdc3-b6166efd0045\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-3b7279-667b9df887-n49j9"
Oct 31 01:52:46.836807 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:52:46.836784    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6fb2f\" (UniqueName: \"kubernetes.io/projected/b6ecd194-e6e7-4b66-bdc3-b6166efd0045-kube-api-access-6fb2f\") pod \"payload-processing-e2e-praxis-a-3b7279-667b9df887-n49j9\" (UID: \"b6ecd194-e6e7-4b66-bdc3-b6166efd0045\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-3b7279-667b9df887-n49j9"
Oct 31 01:52:46.938565 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:52:46.938531    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6fb2f\" (UniqueName: \"kubernetes.io/projected/b6ecd194-e6e7-4b66-bdc3-b6166efd0045-kube-api-access-6fb2f\") pod \"payload-processing-e2e-praxis-a-3b7279-667b9df887-n49j9\" (UID: \"b6ecd194-e6e7-4b66-bdc3-b6166efd0045\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-3b7279-667b9df887-n49j9"
Oct 31 01:52:46.938816 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:52:46.938791    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b6ecd194-e6e7-4b66-bdc3-b6166efd0045-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-3b7279-667b9df887-n49j9\" (UID: \"b6ecd194-e6e7-4b66-bdc3-b6166efd0045\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-3b7279-667b9df887-n49j9"
Oct 31 01:52:46.939591 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:52:46.939541    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b6ecd194-e6e7-4b66-bdc3-b6166efd0045-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-3b7279-667b9df887-n49j9\" (UID: \"b6ecd194-e6e7-4b66-bdc3-b6166efd0045\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-3b7279-667b9df887-n49j9"
Oct 31 01:52:46.995029 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:52:46.994996    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6fb2f\" (UniqueName: \"kubernetes.io/projected/b6ecd194-e6e7-4b66-bdc3-b6166efd0045-kube-api-access-6fb2f\") pod \"payload-processing-e2e-praxis-a-3b7279-667b9df887-n49j9\" (UID: \"b6ecd194-e6e7-4b66-bdc3-b6166efd0045\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-3b7279-667b9df887-n49j9"
Oct 31 01:52:46.997956 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:52:46.997928    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:52:47.190350 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:52:47.190295    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-3b7279-6884d85fbc-fhmqm"]
Oct 31 01:52:47.192015 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:52:47.191986    4964 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1297c54_10e4_4992_9ae3_4d8d628dd0a2.slice/crio-7f1f02dcb0a2b057a56b3c4dc84afd371bbf338ebf966008e6d0ea5a20c8cc32 WatchSource:0}: Error finding container 7f1f02dcb0a2b057a56b3c4dc84afd371bbf338ebf966008e6d0ea5a20c8cc32: Status 404 returned error can't find the container with id 7f1f02dcb0a2b057a56b3c4dc84afd371bbf338ebf966008e6d0ea5a20c8cc32
Oct 31 01:52:47.275284 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:52:47.275253    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-3b7279-667b9df887-n49j9"
Oct 31 01:52:47.541405 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:52:47.541361    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-3b7279-6884d85fbc-fhmqm" event={"ID":"b1297c54-10e4-4992-9ae3-4d8d628dd0a2","Type":"ContainerStarted","Data":"f0ae72b88c1eefa98fc31fa16c5ff8bcec8cada3a1c5ee2d945763e41f028afc"}
Oct 31 01:52:47.541577 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:52:47.541428    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-3b7279-6884d85fbc-fhmqm" event={"ID":"b1297c54-10e4-4992-9ae3-4d8d628dd0a2","Type":"ContainerStarted","Data":"7f1f02dcb0a2b057a56b3c4dc84afd371bbf338ebf966008e6d0ea5a20c8cc32"}
Oct 31 01:52:47.541577 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:52:47.541485    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-3b7279-6884d85fbc-fhmqm"
Oct 31 01:52:47.655817 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:52:47.655792    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-3b7279-667b9df887-n49j9"]
Oct 31 01:52:47.657632 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:52:47.657609    4964 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6ecd194_e6e7_4b66_bdc3_b6166efd0045.slice/crio-b6be67d466b7ae6000f4f31ce46c098d2400b0301307252aeb377f816e8ef44d WatchSource:0}: Error finding container b6be67d466b7ae6000f4f31ce46c098d2400b0301307252aeb377f816e8ef44d: Status 404 returned error can't find the container with id b6be67d466b7ae6000f4f31ce46c098d2400b0301307252aeb377f816e8ef44d
Oct 31 01:52:47.658374 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:52:47.658328    4964 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-3b7279-6884d85fbc-fhmqm" podStartSLOduration=1.658297604 podStartE2EDuration="1.658297604s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:52:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:52:47.654626606 +0000 UTC m=+2936.131192986" watchObservedRunningTime="2026-10-31 01:52:47.658297604 +0000 UTC m=+2936.134863985"
Oct 31 01:52:48.548934 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:52:48.548894    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-3b7279-667b9df887-n49j9" event={"ID":"b6ecd194-e6e7-4b66-bdc3-b6166efd0045","Type":"ContainerStarted","Data":"e0f96b8b6beb9cc8d29d68774840a3506ee664529628efe0cb47af80b0274ca0"}
Oct 31 01:52:48.549413 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:52:48.548995    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-3b7279-667b9df887-n49j9"
Oct 31 01:52:48.549413 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:52:48.549021    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-3b7279-667b9df887-n49j9" event={"ID":"b6ecd194-e6e7-4b66-bdc3-b6166efd0045","Type":"ContainerStarted","Data":"b6be67d466b7ae6000f4f31ce46c098d2400b0301307252aeb377f816e8ef44d"}
Oct 31 01:52:48.712689 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:52:48.712633    4964 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-a-3b7279-667b9df887-n49j9" podStartSLOduration=2.712617636 podStartE2EDuration="2.712617636s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:52:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:52:48.661058969 +0000 UTC m=+2937.137625350" watchObservedRunningTime="2026-10-31 01:52:48.712617636 +0000 UTC m=+2937.189184018"
Oct 31 01:52:49.050688 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:52:49.050649    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:52:49.271906 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:52:49.271876    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:52:49.443879 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:52:49.443789    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:52:51.779736 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:52:51.779694    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:52:52.049910 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:52:52.049804    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:52:56.813149 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:52:56.813113    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:52:58.552671 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:52:58.552645    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-3b7279-6884d85fbc-fhmqm"
Oct 31 01:52:59.559109 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:52:59.559078    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-3b7279-667b9df887-n49j9"
Oct 31 01:53:01.611238 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:53:01.611198    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:53:06.398946 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:53:06.398897    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:53:11.243656 ip-10-0-133-197 kubenswrapper[4964]: E1031 01:53:11.243561    4964 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q56x5" containerName="extractor"
Oct 31 01:53:11.244262 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:53:11.244188    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:53:11.488281 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:53:11.488245    4964 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-bz562"]
Oct 31 01:53:11.493850 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:53:11.493792    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-bz562"
Oct 31 01:53:11.633717 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:53:11.633681    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6c374822-4758-4cc7-b71f-716c77fd5157-istio-token\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-bz562\" (UID: \"6c374822-4758-4cc7-b71f-716c77fd5157\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-bz562"
Oct 31 01:53:11.633909 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:53:11.633750    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c374822-4758-4cc7-b71f-716c77fd5157-credential-socket\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-bz562\" (UID: \"6c374822-4758-4cc7-b71f-716c77fd5157\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-bz562"
Oct 31 01:53:11.633909 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:53:11.633866    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6c374822-4758-4cc7-b71f-716c77fd5157-istio-podinfo\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-bz562\" (UID: \"6c374822-4758-4cc7-b71f-716c77fd5157\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-bz562"
Oct 31 01:53:11.634038 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:53:11.633912    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6c374822-4758-4cc7-b71f-716c77fd5157-istiod-ca-cert\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-bz562\" (UID: \"6c374822-4758-4cc7-b71f-716c77fd5157\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-bz562"
Oct 31 01:53:11.634038 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:53:11.633945    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6c374822-4758-4cc7-b71f-716c77fd5157-istio-data\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-bz562\" (UID: \"6c374822-4758-4cc7-b71f-716c77fd5157\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-bz562"
Oct 31 01:53:11.634126 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:53:11.634109    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6c374822-4758-4cc7-b71f-716c77fd5157-workload-certs\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-bz562\" (UID: \"6c374822-4758-4cc7-b71f-716c77fd5157\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-bz562"
Oct 31 01:53:11.634181 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:53:11.634148    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6c374822-4758-4cc7-b71f-716c77fd5157-istio-envoy\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-bz562\" (UID: \"6c374822-4758-4cc7-b71f-716c77fd5157\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-bz562"
Oct 31 01:53:11.634290 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:53:11.634249    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c374822-4758-4cc7-b71f-716c77fd5157-workload-socket\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-bz562\" (UID: \"6c374822-4758-4cc7-b71f-716c77fd5157\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-bz562"
Oct 31 01:53:11.634366 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:53:11.634305    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fn64x\" (UniqueName: \"kubernetes.io/projected/6c374822-4758-4cc7-b71f-716c77fd5157-kube-api-access-fn64x\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-bz562\" (UID: \"6c374822-4758-4cc7-b71f-716c77fd5157\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-bz562"
Oct 31 01:53:11.690021 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:53:11.689993    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-bz562"]
Oct 31 01:53:11.735686 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:53:11.735651    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c374822-4758-4cc7-b71f-716c77fd5157-credential-socket\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-bz562\" (UID: \"6c374822-4758-4cc7-b71f-716c77fd5157\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-bz562"
Oct 31 01:53:11.735879 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:53:11.735713    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6c374822-4758-4cc7-b71f-716c77fd5157-istio-podinfo\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-bz562\" (UID: \"6c374822-4758-4cc7-b71f-716c77fd5157\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-bz562"
Oct 31 01:53:11.735879 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:53:11.735758    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6c374822-4758-4cc7-b71f-716c77fd5157-istiod-ca-cert\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-bz562\" (UID: \"6c374822-4758-4cc7-b71f-716c77fd5157\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-bz562"
Oct 31 01:53:11.735879 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:53:11.735793    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6c374822-4758-4cc7-b71f-716c77fd5157-istio-data\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-bz562\" (UID: \"6c374822-4758-4cc7-b71f-716c77fd5157\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-bz562"
Oct 31 01:53:11.735879 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:53:11.735853    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6c374822-4758-4cc7-b71f-716c77fd5157-workload-certs\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-bz562\" (UID: \"6c374822-4758-4cc7-b71f-716c77fd5157\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-bz562"
Oct 31 01:53:11.736134 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:53:11.735882    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6c374822-4758-4cc7-b71f-716c77fd5157-istio-envoy\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-bz562\" (UID: \"6c374822-4758-4cc7-b71f-716c77fd5157\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-bz562"
Oct 31 01:53:11.736134 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:53:11.735922    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c374822-4758-4cc7-b71f-716c77fd5157-workload-socket\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-bz562\" (UID: \"6c374822-4758-4cc7-b71f-716c77fd5157\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-bz562"
Oct 31 01:53:11.736134 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:53:11.735947    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fn64x\" (UniqueName: \"kubernetes.io/projected/6c374822-4758-4cc7-b71f-716c77fd5157-kube-api-access-fn64x\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-bz562\" (UID: \"6c374822-4758-4cc7-b71f-716c77fd5157\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-bz562"
Oct 31 01:53:11.736134 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:53:11.736098    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6c374822-4758-4cc7-b71f-716c77fd5157-istio-token\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-bz562\" (UID: \"6c374822-4758-4cc7-b71f-716c77fd5157\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-bz562"
Oct 31 01:53:11.736806 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:53:11.736776    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c374822-4758-4cc7-b71f-716c77fd5157-credential-socket\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-bz562\" (UID: \"6c374822-4758-4cc7-b71f-716c77fd5157\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-bz562"
Oct 31 01:53:11.737517 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:53:11.737240    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6c374822-4758-4cc7-b71f-716c77fd5157-istiod-ca-cert\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-bz562\" (UID: \"6c374822-4758-4cc7-b71f-716c77fd5157\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-bz562"
Oct 31 01:53:11.737517 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:53:11.737249    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c374822-4758-4cc7-b71f-716c77fd5157-workload-socket\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-bz562\" (UID: \"6c374822-4758-4cc7-b71f-716c77fd5157\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-bz562"
Oct 31 01:53:11.738068 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:53:11.738046    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6c374822-4758-4cc7-b71f-716c77fd5157-istio-data\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-bz562\" (UID: \"6c374822-4758-4cc7-b71f-716c77fd5157\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-bz562"
Oct 31 01:53:11.738543 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:53:11.738522    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6c374822-4758-4cc7-b71f-716c77fd5157-workload-certs\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-bz562\" (UID: \"6c374822-4758-4cc7-b71f-716c77fd5157\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-bz562"
Oct 31 01:53:11.739934 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:53:11.739909    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6c374822-4758-4cc7-b71f-716c77fd5157-istio-envoy\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-bz562\" (UID: \"6c374822-4758-4cc7-b71f-716c77fd5157\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-bz562"
Oct 31 01:53:11.740306 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:53:11.740280    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6c374822-4758-4cc7-b71f-716c77fd5157-istio-podinfo\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-bz562\" (UID: \"6c374822-4758-4cc7-b71f-716c77fd5157\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-bz562"
Oct 31 01:53:11.792923 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:53:11.792862    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6c374822-4758-4cc7-b71f-716c77fd5157-istio-token\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-bz562\" (UID: \"6c374822-4758-4cc7-b71f-716c77fd5157\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-bz562"
Oct 31 01:53:11.793607 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:53:11.793586    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fn64x\" (UniqueName: \"kubernetes.io/projected/6c374822-4758-4cc7-b71f-716c77fd5157-kube-api-access-fn64x\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-bz562\" (UID: \"6c374822-4758-4cc7-b71f-716c77fd5157\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-bz562"
Oct 31 01:53:11.807782 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:53:11.807759    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-bz562"
Oct 31 01:53:12.241114 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:53:12.241086    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-bz562"]
Oct 31 01:53:12.242783 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:53:12.242736    4964 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c374822_4758_4cc7_b71f_716c77fd5157.slice/crio-f257878a53ce5eb07637535afebec3bdde2e8419d2643465006d0f02ec560e3e WatchSource:0}: Error finding container f257878a53ce5eb07637535afebec3bdde2e8419d2643465006d0f02ec560e3e: Status 404 returned error can't find the container with id f257878a53ce5eb07637535afebec3bdde2e8419d2643465006d0f02ec560e3e
Oct 31 01:53:12.244997 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:53:12.244971    4964 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 01:53:12.245284 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:53:12.245031    4964 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 01:53:12.245284 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:53:12.245059    4964 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 01:53:12.724570 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:53:12.724538    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-bz562" event={"ID":"6c374822-4758-4cc7-b71f-716c77fd5157","Type":"ContainerStarted","Data":"be9bea42ca15ca82dfc94ffa747bea66fe2ae5ff558cba1a069c2602b0099f19"}
Oct 31 01:53:12.724741 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:53:12.724592    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-bz562" event={"ID":"6c374822-4758-4cc7-b71f-716c77fd5157","Type":"ContainerStarted","Data":"f257878a53ce5eb07637535afebec3bdde2e8419d2643465006d0f02ec560e3e"}
Oct 31 01:53:12.809548 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:53:12.808089    4964 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-bz562"
Oct 31 01:53:12.890254 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:53:12.890204    4964 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-bz562" podStartSLOduration=1.890186001 podStartE2EDuration="1.890186001s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:53:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:53:12.889000029 +0000 UTC m=+2961.365566429" watchObservedRunningTime="2026-10-31 01:53:12.890186001 +0000 UTC m=+2961.366752381"
Oct 31 01:53:13.821424 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:53:13.815878    4964 patch_prober.go:29] interesting pod/e2e-worker-w2-00e840-openshift-default-59b96fb879-bz562 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused" start-of-body=
Oct 31 01:53:13.821424 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:53:13.815931    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-bz562" podUID="6c374822-4758-4cc7-b71f-716c77fd5157" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused"
Oct 31 01:53:14.816348 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:53:14.808654    4964 patch_prober.go:29] interesting pod/e2e-worker-w2-00e840-openshift-default-59b96fb879-bz562 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused" start-of-body=
Oct 31 01:53:14.816348 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:53:14.808709    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-bz562" podUID="6c374822-4758-4cc7-b71f-716c77fd5157" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused"
Oct 31 01:53:15.809665 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:53:15.808618    4964 patch_prober.go:29] interesting pod/e2e-worker-w2-00e840-openshift-default-59b96fb879-bz562 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused" start-of-body=
Oct 31 01:53:15.809665 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:53:15.808694    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-bz562" podUID="6c374822-4758-4cc7-b71f-716c77fd5157" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused"
Oct 31 01:53:16.129484 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:53:16.129399    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:53:16.816338 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:53:16.815636    4964 patch_prober.go:29] interesting pod/e2e-worker-w2-00e840-openshift-default-59b96fb879-bz562 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused" start-of-body=
Oct 31 01:53:16.816338 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:53:16.815692    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-bz562" podUID="6c374822-4758-4cc7-b71f-716c77fd5157" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused"
Oct 31 01:53:17.808517 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:53:17.808486    4964 patch_prober.go:29] interesting pod/e2e-worker-w2-00e840-openshift-default-59b96fb879-bz562 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused" start-of-body=
Oct 31 01:53:17.821443 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:53:17.821404    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-bz562" podUID="6c374822-4758-4cc7-b71f-716c77fd5157" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused"
Oct 31 01:53:18.809161 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:53:18.809123    4964 patch_prober.go:29] interesting pod/e2e-worker-w2-00e840-openshift-default-59b96fb879-bz562 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused" start-of-body=
Oct 31 01:53:18.809383 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:53:18.809189    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-bz562" podUID="6c374822-4758-4cc7-b71f-716c77fd5157" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused"
Oct 31 01:53:20.853448 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:53:20.853409    4964 patch_prober.go:29] interesting pod/e2e-worker-w2-00e840-openshift-default-59b96fb879-bz562 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.163:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 01:53:20.853966 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:53:20.853462    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-bz562" podUID="6c374822-4758-4cc7-b71f-716c77fd5157" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.163:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:53:21.155761 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:53:21.155682    4964 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-bz562"
Oct 31 01:53:21.156254 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:53:21.156238    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-bz562"
Oct 31 01:53:21.156961 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:53:21.156938    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-bz562"
Oct 31 01:53:21.253273 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:53:21.253234    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:53:24.027917 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:53:24.027886    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:53:26.117561 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:53:26.117524    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:53:29.149308 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:53:29.149278    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:53:30.511078 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:53:30.511038    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:53:30.634665 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:53:30.634628    4964 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-3b7279-7f85db968c-9qfcc"]
Oct 31 01:53:30.638888 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:53:30.638867    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-3b7279-7f85db968c-9qfcc"
Oct 31 01:53:30.740999 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:53:30.740966    4964 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-a-3b7279-serving-cert\""
Oct 31 01:53:30.744407 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:53:30.744377    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-3b7279-7f85db968c-9qfcc"]
Oct 31 01:53:30.787254 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:53:30.787170    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3527b972-c696-4103-a8ce-434fddd6e167-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-3b7279-7f85db968c-9qfcc\" (UID: \"3527b972-c696-4103-a8ce-434fddd6e167\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-3b7279-7f85db968c-9qfcc"
Oct 31 01:53:30.787447 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:53:30.787287    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mmb9d\" (UniqueName: \"kubernetes.io/projected/3527b972-c696-4103-a8ce-434fddd6e167-kube-api-access-mmb9d\") pod \"maas-api-e2e-praxis-a-3b7279-7f85db968c-9qfcc\" (UID: \"3527b972-c696-4103-a8ce-434fddd6e167\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-3b7279-7f85db968c-9qfcc"
Oct 31 01:53:30.787447 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:53:30.787361    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3527b972-c696-4103-a8ce-434fddd6e167-metrics-certs\") pod \"maas-api-e2e-praxis-a-3b7279-7f85db968c-9qfcc\" (UID: \"3527b972-c696-4103-a8ce-434fddd6e167\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-3b7279-7f85db968c-9qfcc"
Oct 31 01:53:30.787447 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:53:30.787438    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3527b972-c696-4103-a8ce-434fddd6e167-maas-api-tls\") pod \"maas-api-e2e-praxis-a-3b7279-7f85db968c-9qfcc\" (UID: \"3527b972-c696-4103-a8ce-434fddd6e167\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-3b7279-7f85db968c-9qfcc"
Oct 31 01:53:30.871455 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:53:30.871412    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:53:30.888514 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:53:30.888479    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3527b972-c696-4103-a8ce-434fddd6e167-metrics-certs\") pod \"maas-api-e2e-praxis-a-3b7279-7f85db968c-9qfcc\" (UID: \"3527b972-c696-4103-a8ce-434fddd6e167\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-3b7279-7f85db968c-9qfcc"
Oct 31 01:53:30.888670 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:53:30.888538    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3527b972-c696-4103-a8ce-434fddd6e167-maas-api-tls\") pod \"maas-api-e2e-praxis-a-3b7279-7f85db968c-9qfcc\" (UID: \"3527b972-c696-4103-a8ce-434fddd6e167\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-3b7279-7f85db968c-9qfcc"
Oct 31 01:53:30.888743 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:53:30.888666    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3527b972-c696-4103-a8ce-434fddd6e167-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-3b7279-7f85db968c-9qfcc\" (UID: \"3527b972-c696-4103-a8ce-434fddd6e167\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-3b7279-7f85db968c-9qfcc"
Oct 31 01:53:30.888743 ip-10-0-133-197 kubenswrapper[4964]: E1031 01:53:30.888690    4964 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-praxis-a-3b7279-serving-cert: secret "maas-api-e2e-praxis-a-3b7279-serving-cert" not found
Oct 31 01:53:30.888859 ip-10-0-133-197 kubenswrapper[4964]: E1031 01:53:30.888760    4964 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3527b972-c696-4103-a8ce-434fddd6e167-maas-api-tls podName:3527b972-c696-4103-a8ce-434fddd6e167 nodeName:}" failed. No retries permitted until 2026-10-31 01:53:31.388739074 +0000 UTC m=+2979.865305438 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/3527b972-c696-4103-a8ce-434fddd6e167-maas-api-tls") pod "maas-api-e2e-praxis-a-3b7279-7f85db968c-9qfcc" (UID: "3527b972-c696-4103-a8ce-434fddd6e167") : secret "maas-api-e2e-praxis-a-3b7279-serving-cert" not found
Oct 31 01:53:30.888924 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:53:30.888853    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mmb9d\" (UniqueName: \"kubernetes.io/projected/3527b972-c696-4103-a8ce-434fddd6e167-kube-api-access-mmb9d\") pod \"maas-api-e2e-praxis-a-3b7279-7f85db968c-9qfcc\" (UID: \"3527b972-c696-4103-a8ce-434fddd6e167\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-3b7279-7f85db968c-9qfcc"
Oct 31 01:53:30.889442 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:53:30.889421    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3527b972-c696-4103-a8ce-434fddd6e167-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-3b7279-7f85db968c-9qfcc\" (UID: \"3527b972-c696-4103-a8ce-434fddd6e167\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-3b7279-7f85db968c-9qfcc"
Oct 31 01:53:30.891165 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:53:30.891147    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3527b972-c696-4103-a8ce-434fddd6e167-metrics-certs\") pod \"maas-api-e2e-praxis-a-3b7279-7f85db968c-9qfcc\" (UID: \"3527b972-c696-4103-a8ce-434fddd6e167\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-3b7279-7f85db968c-9qfcc"
Oct 31 01:53:30.947638 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:53:30.947609    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mmb9d\" (UniqueName: \"kubernetes.io/projected/3527b972-c696-4103-a8ce-434fddd6e167-kube-api-access-mmb9d\") pod \"maas-api-e2e-praxis-a-3b7279-7f85db968c-9qfcc\" (UID: \"3527b972-c696-4103-a8ce-434fddd6e167\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-3b7279-7f85db968c-9qfcc"
Oct 31 01:53:31.094860 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:53:31.094739    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:53:31.263597 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:53:31.263544    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:53:31.394893 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:53:31.394817    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3527b972-c696-4103-a8ce-434fddd6e167-maas-api-tls\") pod \"maas-api-e2e-praxis-a-3b7279-7f85db968c-9qfcc\" (UID: \"3527b972-c696-4103-a8ce-434fddd6e167\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-3b7279-7f85db968c-9qfcc"
Oct 31 01:53:31.397719 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:53:31.397689    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3527b972-c696-4103-a8ce-434fddd6e167-maas-api-tls\") pod \"maas-api-e2e-praxis-a-3b7279-7f85db968c-9qfcc\" (UID: \"3527b972-c696-4103-a8ce-434fddd6e167\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-3b7279-7f85db968c-9qfcc"
Oct 31 01:53:31.553670 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:53:31.553642    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-3b7279-7f85db968c-9qfcc"
Oct 31 01:53:31.934404 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:53:31.934378    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-3b7279-7f85db968c-9qfcc"]
Oct 31 01:53:31.935779 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:53:31.935749    4964 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3527b972_c696_4103_a8ce_434fddd6e167.slice/crio-5ec927a29b4b7f56ee7bb2de918432c21e47ec3a1e8fff3ba43b7e4f2dba4eda WatchSource:0}: Error finding container 5ec927a29b4b7f56ee7bb2de918432c21e47ec3a1e8fff3ba43b7e4f2dba4eda: Status 404 returned error can't find the container with id 5ec927a29b4b7f56ee7bb2de918432c21e47ec3a1e8fff3ba43b7e4f2dba4eda
Oct 31 01:53:31.937272 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:53:31.937248    4964 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 31 01:53:32.890514 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:53:32.890450    4964 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-c0daf4-fb8556c8c-k4ggk"]
Oct 31 01:53:32.900226 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:53:32.900183    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-3b7279-7f85db968c-9qfcc" event={"ID":"3527b972-c696-4103-a8ce-434fddd6e167","Type":"ContainerStarted","Data":"5ec927a29b4b7f56ee7bb2de918432c21e47ec3a1e8fff3ba43b7e4f2dba4eda"}
Oct 31 01:53:33.172977 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:53:33.172895    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:53:33.352091 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:53:33.352055    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:53:33.910222 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:53:33.910186    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-3b7279-7f85db968c-9qfcc" event={"ID":"3527b972-c696-4103-a8ce-434fddd6e167","Type":"ContainerStarted","Data":"8996522284a6957f7ba467e16dc2ea4b9602994f4868448d5e3058145d08625f"}
Oct 31 01:53:33.910667 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:53:33.910306    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-3b7279-7f85db968c-9qfcc"
Oct 31 01:53:34.026660 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:53:34.026599    4964 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-3b7279-7f85db968c-9qfcc" podStartSLOduration=3.035571081 podStartE2EDuration="4.026582068s" totalImagesPullingTime="991.010987ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:53:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:53:34.024964247 +0000 UTC m=+2982.501530651" watchObservedRunningTime="2026-10-31 01:53:34.026582068 +0000 UTC m=+2982.503148449"
Oct 31 01:53:35.798129 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:53:35.798085    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:53:37.891576 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:53:37.891536    4964 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-c0daf4-fb8556c8c-k4ggk" podUID="2e38f4c2-b630-4a05-abb0-49fce8594ec3" containerName="payload-pre-processing" containerID="cri-o://6fed9f9efa0d238b8bfe654095d28e168804e352e9bb7846329132442452e608" gracePeriod=25
Oct 31 01:53:39.922916 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:53:39.922885    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-3b7279-7f85db968c-9qfcc"
Oct 31 01:53:40.617982 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:53:40.617935    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:53:41.237415 ip-10-0-133-197 kubenswrapper[4964]: E1031 01:53:41.237381    4964 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q56x5" containerName="extractor"
Oct 31 01:53:44.830720 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:53:44.830686    4964 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-c0daf4-fb8556c8c-k4ggk" podUID="2e38f4c2-b630-4a05-abb0-49fce8594ec3" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 01:53:45.350227 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:53:45.350194    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:53:50.130541 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:53:50.130507    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:53:52.124076 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:53:52.124039    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:53:52.234492 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:53:52.234464    4964 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-9257n_f0c0ada7-7688-4dc2-97fc-813e4d0c4ef3/console-operator/2.log"
Oct 31 01:53:52.243777 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:53:52.243754    4964 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-9257n_f0c0ada7-7688-4dc2-97fc-813e4d0c4ef3/console-operator/2.log"
Oct 31 01:53:54.835627 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:53:54.835587    4964 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-c0daf4-fb8556c8c-k4ggk" podUID="2e38f4c2-b630-4a05-abb0-49fce8594ec3" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 01:53:55.046343 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:53:55.045856    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:53:58.100631 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:53:58.100590    4964 generic.go:356] "Generic (PLEG): container finished" podID="2e38f4c2-b630-4a05-abb0-49fce8594ec3" containerID="6fed9f9efa0d238b8bfe654095d28e168804e352e9bb7846329132442452e608" exitCode=0
Oct 31 01:53:58.101086 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:53:58.100658    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-c0daf4-fb8556c8c-k4ggk" event={"ID":"2e38f4c2-b630-4a05-abb0-49fce8594ec3","Type":"ContainerDied","Data":"6fed9f9efa0d238b8bfe654095d28e168804e352e9bb7846329132442452e608"}
Oct 31 01:53:58.188422 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:53:58.188392    4964 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-c0daf4-fb8556c8c-k4ggk"
Oct 31 01:53:58.295534 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:53:58.295502    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2e38f4c2-b630-4a05-abb0-49fce8594ec3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2e38f4c2-b630-4a05-abb0-49fce8594ec3-plugins-config-volume\") pod \"2e38f4c2-b630-4a05-abb0-49fce8594ec3\" (UID: \"2e38f4c2-b630-4a05-abb0-49fce8594ec3\") "
Oct 31 01:53:58.295741 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:53:58.295725    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2e38f4c2-b630-4a05-abb0-49fce8594ec3-kube-api-access-q9lxw\" (UniqueName: \"kubernetes.io/projected/2e38f4c2-b630-4a05-abb0-49fce8594ec3-kube-api-access-q9lxw\") pod \"2e38f4c2-b630-4a05-abb0-49fce8594ec3\" (UID: \"2e38f4c2-b630-4a05-abb0-49fce8594ec3\") "
Oct 31 01:53:58.295861 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:53:58.295840    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2e38f4c2-b630-4a05-abb0-49fce8594ec3-plugins-config-volume" pod "2e38f4c2-b630-4a05-abb0-49fce8594ec3" (UID: "2e38f4c2-b630-4a05-abb0-49fce8594ec3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:53:58.296246 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:53:58.296221    4964 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2e38f4c2-b630-4a05-abb0-49fce8594ec3-plugins-config-volume\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:53:58.298104 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:53:58.298087    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e38f4c2-b630-4a05-abb0-49fce8594ec3-kube-api-access-q9lxw" pod "2e38f4c2-b630-4a05-abb0-49fce8594ec3" (UID: "2e38f4c2-b630-4a05-abb0-49fce8594ec3"). InnerVolumeSpecName "kube-api-access-q9lxw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:53:58.397673 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:53:58.397641    4964 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q9lxw\" (UniqueName: \"kubernetes.io/projected/2e38f4c2-b630-4a05-abb0-49fce8594ec3-kube-api-access-q9lxw\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:53:59.108188 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:53:59.108161    4964 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-c0daf4-fb8556c8c-k4ggk"
Oct 31 01:53:59.108188 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:53:59.108186    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-c0daf4-fb8556c8c-k4ggk" event={"ID":"2e38f4c2-b630-4a05-abb0-49fce8594ec3","Type":"ContainerDied","Data":"dfa503c9274da2fee10e3d820d10522b6b69f5e3f8e97ca1c621aae42a8830bf"}
Oct 31 01:53:59.108757 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:53:59.108242    4964 scope.go:122] "RemoveContainer" containerID="6fed9f9efa0d238b8bfe654095d28e168804e352e9bb7846329132442452e608"
Oct 31 01:53:59.273286 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:53:59.273256    4964 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-c0daf4-fb8556c8c-k4ggk"]
Oct 31 01:53:59.322387 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:53:59.322347    4964 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-c0daf4-fb8556c8c-k4ggk"]
Oct 31 01:53:59.778074 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:53:59.778046    4964 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2e38f4c2-b630-4a05-abb0-49fce8594ec3" path="/var/lib/kubelet/pods/2e38f4c2-b630-4a05-abb0-49fce8594ec3/volumes"
Oct 31 01:54:00.018967 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:54:00.018913    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:54:04.888924 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:54:04.888879    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:54:08.470540 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:54:08.470506    4964 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-sbp55"]
Oct 31 01:54:08.471067 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:54:08.471053    4964 cpu_manager.go:414] "containerMap: removing container" podUID="2e38f4c2-b630-4a05-abb0-49fce8594ec3" containerName="payload-pre-processing"
Oct 31 01:54:08.471150 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:54:08.471068    4964 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2e38f4c2-b630-4a05-abb0-49fce8594ec3" containerName="payload-pre-processing"
Oct 31 01:54:08.471209 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:54:08.471181    4964 memory_manager.go:365] "RemoveStaleState removing state" podUID="2e38f4c2-b630-4a05-abb0-49fce8594ec3" containerName="payload-pre-processing"
Oct 31 01:54:08.475943 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:54:08.475920    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-sbp55"
Oct 31 01:54:08.516614 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:54:08.516586    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/31fe1f3a-23fe-4999-a8fc-d82b2185f0a7-workload-socket\") pod \"e2e-worker-w3-a8e157-openshift-default-7fb96689c7-sbp55\" (UID: \"31fe1f3a-23fe-4999-a8fc-d82b2185f0a7\") " pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-sbp55"
Oct 31 01:54:08.516794 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:54:08.516629    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/31fe1f3a-23fe-4999-a8fc-d82b2185f0a7-istio-envoy\") pod \"e2e-worker-w3-a8e157-openshift-default-7fb96689c7-sbp55\" (UID: \"31fe1f3a-23fe-4999-a8fc-d82b2185f0a7\") " pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-sbp55"
Oct 31 01:54:08.516794 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:54:08.516657    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/31fe1f3a-23fe-4999-a8fc-d82b2185f0a7-istiod-ca-cert\") pod \"e2e-worker-w3-a8e157-openshift-default-7fb96689c7-sbp55\" (UID: \"31fe1f3a-23fe-4999-a8fc-d82b2185f0a7\") " pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-sbp55"
Oct 31 01:54:08.516794 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:54:08.516738    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/31fe1f3a-23fe-4999-a8fc-d82b2185f0a7-istio-data\") pod \"e2e-worker-w3-a8e157-openshift-default-7fb96689c7-sbp55\" (UID: \"31fe1f3a-23fe-4999-a8fc-d82b2185f0a7\") " pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-sbp55"
Oct 31 01:54:08.516963 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:54:08.516928    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/31fe1f3a-23fe-4999-a8fc-d82b2185f0a7-credential-socket\") pod \"e2e-worker-w3-a8e157-openshift-default-7fb96689c7-sbp55\" (UID: \"31fe1f3a-23fe-4999-a8fc-d82b2185f0a7\") " pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-sbp55"
Oct 31 01:54:08.517026 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:54:08.516959    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/31fe1f3a-23fe-4999-a8fc-d82b2185f0a7-workload-certs\") pod \"e2e-worker-w3-a8e157-openshift-default-7fb96689c7-sbp55\" (UID: \"31fe1f3a-23fe-4999-a8fc-d82b2185f0a7\") " pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-sbp55"
Oct 31 01:54:08.517026 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:54:08.516995    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/31fe1f3a-23fe-4999-a8fc-d82b2185f0a7-istio-podinfo\") pod \"e2e-worker-w3-a8e157-openshift-default-7fb96689c7-sbp55\" (UID: \"31fe1f3a-23fe-4999-a8fc-d82b2185f0a7\") " pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-sbp55"
Oct 31 01:54:08.517026 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:54:08.517020    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4k7vl\" (UniqueName: \"kubernetes.io/projected/31fe1f3a-23fe-4999-a8fc-d82b2185f0a7-kube-api-access-4k7vl\") pod \"e2e-worker-w3-a8e157-openshift-default-7fb96689c7-sbp55\" (UID: \"31fe1f3a-23fe-4999-a8fc-d82b2185f0a7\") " pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-sbp55"
Oct 31 01:54:08.517166 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:54:08.517067    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/31fe1f3a-23fe-4999-a8fc-d82b2185f0a7-istio-token\") pod \"e2e-worker-w3-a8e157-openshift-default-7fb96689c7-sbp55\" (UID: \"31fe1f3a-23fe-4999-a8fc-d82b2185f0a7\") " pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-sbp55"
Oct 31 01:54:08.564086 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:54:08.564052    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-sbp55"]
Oct 31 01:54:08.618643 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:54:08.618606    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/31fe1f3a-23fe-4999-a8fc-d82b2185f0a7-credential-socket\") pod \"e2e-worker-w3-a8e157-openshift-default-7fb96689c7-sbp55\" (UID: \"31fe1f3a-23fe-4999-a8fc-d82b2185f0a7\") " pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-sbp55"
Oct 31 01:54:08.618643 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:54:08.618653    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/31fe1f3a-23fe-4999-a8fc-d82b2185f0a7-workload-certs\") pod \"e2e-worker-w3-a8e157-openshift-default-7fb96689c7-sbp55\" (UID: \"31fe1f3a-23fe-4999-a8fc-d82b2185f0a7\") " pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-sbp55"
Oct 31 01:54:08.618911 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:54:08.618690    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/31fe1f3a-23fe-4999-a8fc-d82b2185f0a7-istio-podinfo\") pod \"e2e-worker-w3-a8e157-openshift-default-7fb96689c7-sbp55\" (UID: \"31fe1f3a-23fe-4999-a8fc-d82b2185f0a7\") " pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-sbp55"
Oct 31 01:54:08.618911 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:54:08.618714    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4k7vl\" (UniqueName: \"kubernetes.io/projected/31fe1f3a-23fe-4999-a8fc-d82b2185f0a7-kube-api-access-4k7vl\") pod \"e2e-worker-w3-a8e157-openshift-default-7fb96689c7-sbp55\" (UID: \"31fe1f3a-23fe-4999-a8fc-d82b2185f0a7\") " pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-sbp55"
Oct 31 01:54:08.618911 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:54:08.618759    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/31fe1f3a-23fe-4999-a8fc-d82b2185f0a7-istio-token\") pod \"e2e-worker-w3-a8e157-openshift-default-7fb96689c7-sbp55\" (UID: \"31fe1f3a-23fe-4999-a8fc-d82b2185f0a7\") " pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-sbp55"
Oct 31 01:54:08.619093 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:54:08.618923    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/31fe1f3a-23fe-4999-a8fc-d82b2185f0a7-workload-socket\") pod \"e2e-worker-w3-a8e157-openshift-default-7fb96689c7-sbp55\" (UID: \"31fe1f3a-23fe-4999-a8fc-d82b2185f0a7\") " pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-sbp55"
Oct 31 01:54:08.619093 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:54:08.618952    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/31fe1f3a-23fe-4999-a8fc-d82b2185f0a7-istio-envoy\") pod \"e2e-worker-w3-a8e157-openshift-default-7fb96689c7-sbp55\" (UID: \"31fe1f3a-23fe-4999-a8fc-d82b2185f0a7\") " pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-sbp55"
Oct 31 01:54:08.619093 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:54:08.618978    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/31fe1f3a-23fe-4999-a8fc-d82b2185f0a7-istiod-ca-cert\") pod \"e2e-worker-w3-a8e157-openshift-default-7fb96689c7-sbp55\" (UID: \"31fe1f3a-23fe-4999-a8fc-d82b2185f0a7\") " pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-sbp55"
Oct 31 01:54:08.619093 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:54:08.619001    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/31fe1f3a-23fe-4999-a8fc-d82b2185f0a7-istio-data\") pod \"e2e-worker-w3-a8e157-openshift-default-7fb96689c7-sbp55\" (UID: \"31fe1f3a-23fe-4999-a8fc-d82b2185f0a7\") " pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-sbp55"
Oct 31 01:54:08.619537 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:54:08.619514    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/31fe1f3a-23fe-4999-a8fc-d82b2185f0a7-istio-data\") pod \"e2e-worker-w3-a8e157-openshift-default-7fb96689c7-sbp55\" (UID: \"31fe1f3a-23fe-4999-a8fc-d82b2185f0a7\") " pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-sbp55"
Oct 31 01:54:08.619786 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:54:08.619765    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/31fe1f3a-23fe-4999-a8fc-d82b2185f0a7-credential-socket\") pod \"e2e-worker-w3-a8e157-openshift-default-7fb96689c7-sbp55\" (UID: \"31fe1f3a-23fe-4999-a8fc-d82b2185f0a7\") " pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-sbp55"
Oct 31 01:54:08.620020 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:54:08.620000    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/31fe1f3a-23fe-4999-a8fc-d82b2185f0a7-workload-certs\") pod \"e2e-worker-w3-a8e157-openshift-default-7fb96689c7-sbp55\" (UID: \"31fe1f3a-23fe-4999-a8fc-d82b2185f0a7\") " pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-sbp55"
Oct 31 01:54:08.620696 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:54:08.620652    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/31fe1f3a-23fe-4999-a8fc-d82b2185f0a7-workload-socket\") pod \"e2e-worker-w3-a8e157-openshift-default-7fb96689c7-sbp55\" (UID: \"31fe1f3a-23fe-4999-a8fc-d82b2185f0a7\") " pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-sbp55"
Oct 31 01:54:08.623079 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:54:08.623044    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/31fe1f3a-23fe-4999-a8fc-d82b2185f0a7-istiod-ca-cert\") pod \"e2e-worker-w3-a8e157-openshift-default-7fb96689c7-sbp55\" (UID: \"31fe1f3a-23fe-4999-a8fc-d82b2185f0a7\") " pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-sbp55"
Oct 31 01:54:08.623756 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:54:08.623736    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/31fe1f3a-23fe-4999-a8fc-d82b2185f0a7-istio-podinfo\") pod \"e2e-worker-w3-a8e157-openshift-default-7fb96689c7-sbp55\" (UID: \"31fe1f3a-23fe-4999-a8fc-d82b2185f0a7\") " pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-sbp55"
Oct 31 01:54:08.633228 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:54:08.633203    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/31fe1f3a-23fe-4999-a8fc-d82b2185f0a7-istio-envoy\") pod \"e2e-worker-w3-a8e157-openshift-default-7fb96689c7-sbp55\" (UID: \"31fe1f3a-23fe-4999-a8fc-d82b2185f0a7\") " pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-sbp55"
Oct 31 01:54:08.677006 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:54:08.676984    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/31fe1f3a-23fe-4999-a8fc-d82b2185f0a7-istio-token\") pod \"e2e-worker-w3-a8e157-openshift-default-7fb96689c7-sbp55\" (UID: \"31fe1f3a-23fe-4999-a8fc-d82b2185f0a7\") " pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-sbp55"
Oct 31 01:54:08.677163 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:54:08.677146    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4k7vl\" (UniqueName: \"kubernetes.io/projected/31fe1f3a-23fe-4999-a8fc-d82b2185f0a7-kube-api-access-4k7vl\") pod \"e2e-worker-w3-a8e157-openshift-default-7fb96689c7-sbp55\" (UID: \"31fe1f3a-23fe-4999-a8fc-d82b2185f0a7\") " pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-sbp55"
Oct 31 01:54:08.789587 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:54:08.789512    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-sbp55"
Oct 31 01:54:09.229447 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:54:09.229419    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-sbp55"]
Oct 31 01:54:09.231694 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:54:09.231669    4964 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod31fe1f3a_23fe_4999_a8fc_d82b2185f0a7.slice/crio-dbed79999a734addb7df91c92771720443546bba4748514a8f2939fc4582ba15 WatchSource:0}: Error finding container dbed79999a734addb7df91c92771720443546bba4748514a8f2939fc4582ba15: Status 404 returned error can't find the container with id dbed79999a734addb7df91c92771720443546bba4748514a8f2939fc4582ba15
Oct 31 01:54:09.234077 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:54:09.234044    4964 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 01:54:09.234179 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:54:09.234112    4964 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 01:54:09.234248 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:54:09.234142    4964 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 01:54:09.696413 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:54:09.693226    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:54:10.178837 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:54:10.178805    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-sbp55" event={"ID":"31fe1f3a-23fe-4999-a8fc-d82b2185f0a7","Type":"ContainerStarted","Data":"e314062d6f06a9ccea3fa6eb68b01f515ffcfdc936fc7aae9d35bdd960fa44e6"}
Oct 31 01:54:10.179025 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:54:10.178851    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-sbp55" event={"ID":"31fe1f3a-23fe-4999-a8fc-d82b2185f0a7","Type":"ContainerStarted","Data":"dbed79999a734addb7df91c92771720443546bba4748514a8f2939fc4582ba15"}
Oct 31 01:54:10.413349 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:54:10.410833    4964 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-sbp55" podStartSLOduration=2.410813834 podStartE2EDuration="2.410813834s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:54:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:54:10.344700921 +0000 UTC m=+3018.821267300" watchObservedRunningTime="2026-10-31 01:54:10.410813834 +0000 UTC m=+3018.887380216"
Oct 31 01:54:10.800343 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:54:10.797211    4964 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-sbp55"
Oct 31 01:54:10.800343 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:54:10.797400    4964 patch_prober.go:29] interesting pod/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-sbp55 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.165:15021/healthz/ready\": dial tcp 10.132.0.165:15021: connect: connection refused" start-of-body=
Oct 31 01:54:10.800343 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:54:10.797445    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-sbp55" podUID="31fe1f3a-23fe-4999-a8fc-d82b2185f0a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.165:15021/healthz/ready\": dial tcp 10.132.0.165:15021: connect: connection refused"
Oct 31 01:54:11.256500 ip-10-0-133-197 kubenswrapper[4964]: E1031 01:54:11.256372    4964 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q56x5" containerName="extractor"
Oct 31 01:54:11.789914 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:54:11.789879    4964 patch_prober.go:29] interesting pod/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-sbp55 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.165:15021/healthz/ready\": dial tcp 10.132.0.165:15021: connect: connection refused" start-of-body=
Oct 31 01:54:11.790101 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:54:11.789943    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-sbp55" podUID="31fe1f3a-23fe-4999-a8fc-d82b2185f0a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.165:15021/healthz/ready\": dial tcp 10.132.0.165:15021: connect: connection refused"
Oct 31 01:54:12.789927 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:54:12.789885    4964 patch_prober.go:29] interesting pod/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-sbp55 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.165:15021/healthz/ready\": dial tcp 10.132.0.165:15021: connect: connection refused" start-of-body=
Oct 31 01:54:12.790576 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:54:12.789953    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-sbp55" podUID="31fe1f3a-23fe-4999-a8fc-d82b2185f0a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.165:15021/healthz/ready\": dial tcp 10.132.0.165:15021: connect: connection refused"
Oct 31 01:54:13.789896 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:54:13.789861    4964 patch_prober.go:29] interesting pod/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-sbp55 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.165:15021/healthz/ready\": dial tcp 10.132.0.165:15021: connect: connection refused" start-of-body=
Oct 31 01:54:13.790116 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:54:13.789922    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-sbp55" podUID="31fe1f3a-23fe-4999-a8fc-d82b2185f0a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.165:15021/healthz/ready\": dial tcp 10.132.0.165:15021: connect: connection refused"
Oct 31 01:54:14.658341 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:54:14.656416    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:54:14.800068 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:54:14.800038    4964 patch_prober.go:29] interesting pod/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-sbp55 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.165:15021/healthz/ready\": dial tcp 10.132.0.165:15021: connect: connection refused" start-of-body=
Oct 31 01:54:14.800679 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:54:14.800646    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-sbp55" podUID="31fe1f3a-23fe-4999-a8fc-d82b2185f0a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.165:15021/healthz/ready\": dial tcp 10.132.0.165:15021: connect: connection refused"
Oct 31 01:54:15.789898 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:54:15.789853    4964 patch_prober.go:29] interesting pod/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-sbp55 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.165:15021/healthz/ready\": dial tcp 10.132.0.165:15021: connect: connection refused" start-of-body=
Oct 31 01:54:15.790078 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:54:15.789905    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-sbp55" podUID="31fe1f3a-23fe-4999-a8fc-d82b2185f0a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.165:15021/healthz/ready\": dial tcp 10.132.0.165:15021: connect: connection refused"
Oct 31 01:54:17.812373 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:54:17.812155    4964 patch_prober.go:29] interesting pod/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-sbp55 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.165:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 01:54:17.812373 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:54:17.812237    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-sbp55" podUID="31fe1f3a-23fe-4999-a8fc-d82b2185f0a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.165:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:54:17.855053 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:54:17.855010    4964 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-sbp55"
Oct 31 01:54:17.855603 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:54:17.855579    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-sbp55"
Oct 31 01:54:17.877150 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:54:17.877132    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-sbp55"
Oct 31 01:54:19.602108 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:54:19.602071    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:54:19.927137 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:54:19.927062    4964 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-4xmfp"]
Oct 31 01:54:19.931332 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:54:19.931296    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-4xmfp"
Oct 31 01:54:19.986078 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:54:19.986047    4964 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-057e2d-openshift-default-dockercfg-2bvxk\""
Oct 31 01:54:20.040092 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:54:20.040064    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-4xmfp"]
Oct 31 01:54:20.097434 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:54:20.097392    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bb5fa688-49e8-4e29-9643-bab6160bec86-istio-podinfo\") pod \"e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-4xmfp\" (UID: \"bb5fa688-49e8-4e29-9643-bab6160bec86\") " pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-4xmfp"
Oct 31 01:54:20.097606 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:54:20.097471    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bb5fa688-49e8-4e29-9643-bab6160bec86-istiod-ca-cert\") pod \"e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-4xmfp\" (UID: \"bb5fa688-49e8-4e29-9643-bab6160bec86\") " pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-4xmfp"
Oct 31 01:54:20.097606 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:54:20.097502    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bb5fa688-49e8-4e29-9643-bab6160bec86-istio-token\") pod \"e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-4xmfp\" (UID: \"bb5fa688-49e8-4e29-9643-bab6160bec86\") " pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-4xmfp"
Oct 31 01:54:20.097606 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:54:20.097575    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bb5fa688-49e8-4e29-9643-bab6160bec86-istio-data\") pod \"e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-4xmfp\" (UID: \"bb5fa688-49e8-4e29-9643-bab6160bec86\") " pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-4xmfp"
Oct 31 01:54:20.097762 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:54:20.097633    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb5fa688-49e8-4e29-9643-bab6160bec86-workload-socket\") pod \"e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-4xmfp\" (UID: \"bb5fa688-49e8-4e29-9643-bab6160bec86\") " pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-4xmfp"
Oct 31 01:54:20.097762 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:54:20.097726    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r8bbj\" (UniqueName: \"kubernetes.io/projected/bb5fa688-49e8-4e29-9643-bab6160bec86-kube-api-access-r8bbj\") pod \"e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-4xmfp\" (UID: \"bb5fa688-49e8-4e29-9643-bab6160bec86\") " pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-4xmfp"
Oct 31 01:54:20.097831 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:54:20.097772    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bb5fa688-49e8-4e29-9643-bab6160bec86-istio-envoy\") pod \"e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-4xmfp\" (UID: \"bb5fa688-49e8-4e29-9643-bab6160bec86\") " pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-4xmfp"
Oct 31 01:54:20.097831 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:54:20.097803    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bb5fa688-49e8-4e29-9643-bab6160bec86-workload-certs\") pod \"e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-4xmfp\" (UID: \"bb5fa688-49e8-4e29-9643-bab6160bec86\") " pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-4xmfp"
Oct 31 01:54:20.097903 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:54:20.097830    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb5fa688-49e8-4e29-9643-bab6160bec86-credential-socket\") pod \"e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-4xmfp\" (UID: \"bb5fa688-49e8-4e29-9643-bab6160bec86\") " pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-4xmfp"
Oct 31 01:54:20.199559 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:54:20.199477    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bb5fa688-49e8-4e29-9643-bab6160bec86-istio-podinfo\") pod \"e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-4xmfp\" (UID: \"bb5fa688-49e8-4e29-9643-bab6160bec86\") " pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-4xmfp"
Oct 31 01:54:20.199559 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:54:20.199520    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bb5fa688-49e8-4e29-9643-bab6160bec86-istiod-ca-cert\") pod \"e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-4xmfp\" (UID: \"bb5fa688-49e8-4e29-9643-bab6160bec86\") " pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-4xmfp"
Oct 31 01:54:20.199559 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:54:20.199549    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bb5fa688-49e8-4e29-9643-bab6160bec86-istio-token\") pod \"e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-4xmfp\" (UID: \"bb5fa688-49e8-4e29-9643-bab6160bec86\") " pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-4xmfp"
Oct 31 01:54:20.199881 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:54:20.199586    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bb5fa688-49e8-4e29-9643-bab6160bec86-istio-data\") pod \"e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-4xmfp\" (UID: \"bb5fa688-49e8-4e29-9643-bab6160bec86\") " pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-4xmfp"
Oct 31 01:54:20.199881 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:54:20.199611    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb5fa688-49e8-4e29-9643-bab6160bec86-workload-socket\") pod \"e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-4xmfp\" (UID: \"bb5fa688-49e8-4e29-9643-bab6160bec86\") " pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-4xmfp"
Oct 31 01:54:20.199881 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:54:20.199672    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r8bbj\" (UniqueName: \"kubernetes.io/projected/bb5fa688-49e8-4e29-9643-bab6160bec86-kube-api-access-r8bbj\") pod \"e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-4xmfp\" (UID: \"bb5fa688-49e8-4e29-9643-bab6160bec86\") " pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-4xmfp"
Oct 31 01:54:20.199881 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:54:20.199714    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bb5fa688-49e8-4e29-9643-bab6160bec86-istio-envoy\") pod \"e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-4xmfp\" (UID: \"bb5fa688-49e8-4e29-9643-bab6160bec86\") " pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-4xmfp"
Oct 31 01:54:20.199881 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:54:20.199830    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bb5fa688-49e8-4e29-9643-bab6160bec86-workload-certs\") pod \"e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-4xmfp\" (UID: \"bb5fa688-49e8-4e29-9643-bab6160bec86\") " pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-4xmfp"
Oct 31 01:54:20.199881 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:54:20.199873    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb5fa688-49e8-4e29-9643-bab6160bec86-credential-socket\") pod \"e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-4xmfp\" (UID: \"bb5fa688-49e8-4e29-9643-bab6160bec86\") " pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-4xmfp"
Oct 31 01:54:20.200212 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:54:20.200066    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb5fa688-49e8-4e29-9643-bab6160bec86-workload-socket\") pod \"e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-4xmfp\" (UID: \"bb5fa688-49e8-4e29-9643-bab6160bec86\") " pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-4xmfp"
Oct 31 01:54:20.200212 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:54:20.200171    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bb5fa688-49e8-4e29-9643-bab6160bec86-workload-certs\") pod \"e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-4xmfp\" (UID: \"bb5fa688-49e8-4e29-9643-bab6160bec86\") " pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-4xmfp"
Oct 31 01:54:20.200402 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:54:20.200375    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb5fa688-49e8-4e29-9643-bab6160bec86-credential-socket\") pod \"e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-4xmfp\" (UID: \"bb5fa688-49e8-4e29-9643-bab6160bec86\") " pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-4xmfp"
Oct 31 01:54:20.200553 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:54:20.200533    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bb5fa688-49e8-4e29-9643-bab6160bec86-istio-data\") pod \"e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-4xmfp\" (UID: \"bb5fa688-49e8-4e29-9643-bab6160bec86\") " pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-4xmfp"
Oct 31 01:54:20.200860 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:54:20.200838    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bb5fa688-49e8-4e29-9643-bab6160bec86-istiod-ca-cert\") pod \"e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-4xmfp\" (UID: \"bb5fa688-49e8-4e29-9643-bab6160bec86\") " pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-4xmfp"
Oct 31 01:54:20.202512 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:54:20.202479    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bb5fa688-49e8-4e29-9643-bab6160bec86-istio-podinfo\") pod \"e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-4xmfp\" (UID: \"bb5fa688-49e8-4e29-9643-bab6160bec86\") " pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-4xmfp"
Oct 31 01:54:20.202619 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:54:20.202511    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bb5fa688-49e8-4e29-9643-bab6160bec86-istio-envoy\") pod \"e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-4xmfp\" (UID: \"bb5fa688-49e8-4e29-9643-bab6160bec86\") " pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-4xmfp"
Oct 31 01:54:20.255820 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:54:20.255788    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bb5fa688-49e8-4e29-9643-bab6160bec86-istio-token\") pod \"e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-4xmfp\" (UID: \"bb5fa688-49e8-4e29-9643-bab6160bec86\") " pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-4xmfp"
Oct 31 01:54:20.255971 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:54:20.255892    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r8bbj\" (UniqueName: \"kubernetes.io/projected/bb5fa688-49e8-4e29-9643-bab6160bec86-kube-api-access-r8bbj\") pod \"e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-4xmfp\" (UID: \"bb5fa688-49e8-4e29-9643-bab6160bec86\") " pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-4xmfp"
Oct 31 01:54:20.547331 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:54:20.547296    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-4xmfp"
Oct 31 01:54:20.677304 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:54:20.677270    4964 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-8k9j8"]
Oct 31 01:54:20.677771 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:54:20.677525    4964 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-8k9j8" podUID="c6252303-98d9-43e9-b6ca-f9087cf79430" containerName="istio-proxy" containerID="cri-o://a26ab811e35fc62fd8e9068f88b29a50a27e22a65c1974c198be545af1f4f515" gracePeriod=30
Oct 31 01:54:20.982514 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:54:20.982489    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-4xmfp"]
Oct 31 01:54:20.984241 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:54:20.984216    4964 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb5fa688_49e8_4e29_9643_bab6160bec86.slice/crio-736e21b8ae149c9b746cc99c5f38e0611e0c4ca05ff35fd291f97de311341ead WatchSource:0}: Error finding container 736e21b8ae149c9b746cc99c5f38e0611e0c4ca05ff35fd291f97de311341ead: Status 404 returned error can't find the container with id 736e21b8ae149c9b746cc99c5f38e0611e0c4ca05ff35fd291f97de311341ead
Oct 31 01:54:20.987046 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:54:20.987017    4964 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 01:54:20.987120 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:54:20.987091    4964 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 01:54:20.987158 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:54:20.987120    4964 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 01:54:21.339166 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:54:21.339079    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-4xmfp" event={"ID":"bb5fa688-49e8-4e29-9643-bab6160bec86","Type":"ContainerStarted","Data":"863e1be1786bcb96ef7cab29a270c0e0db116019f9970b424006c0b037cae7e3"}
Oct 31 01:54:21.339166 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:54:21.339137    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-4xmfp" event={"ID":"bb5fa688-49e8-4e29-9643-bab6160bec86","Type":"ContainerStarted","Data":"736e21b8ae149c9b746cc99c5f38e0611e0c4ca05ff35fd291f97de311341ead"}
Oct 31 01:54:21.457336 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:54:21.457257    4964 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-4xmfp" podStartSLOduration=2.457242505 podStartE2EDuration="2.457242505s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:54:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:54:21.455006664 +0000 UTC m=+3029.931573043" watchObservedRunningTime="2026-10-31 01:54:21.457242505 +0000 UTC m=+3029.933808882"
Oct 31 01:54:21.549488 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:54:21.549450    4964 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-4xmfp"
Oct 31 01:54:22.550137 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:54:22.550104    4964 patch_prober.go:29] interesting pod/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-4xmfp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.166:15021/healthz/ready\": dial tcp 10.132.0.166:15021: connect: connection refused" start-of-body=
Oct 31 01:54:22.550587 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:54:22.550154    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-4xmfp" podUID="bb5fa688-49e8-4e29-9643-bab6160bec86" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.166:15021/healthz/ready\": dial tcp 10.132.0.166:15021: connect: connection refused"
Oct 31 01:54:23.547651 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:54:23.547617    4964 patch_prober.go:29] interesting pod/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-4xmfp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.166:15021/healthz/ready\": dial tcp 10.132.0.166:15021: connect: connection refused" start-of-body=
Oct 31 01:54:23.547904 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:54:23.547675    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-4xmfp" podUID="bb5fa688-49e8-4e29-9643-bab6160bec86" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.166:15021/healthz/ready\": dial tcp 10.132.0.166:15021: connect: connection refused"
Oct 31 01:54:23.583291 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:54:23.583258    4964 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-5jgz8"]
Oct 31 01:54:23.587767 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:54:23.587744    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-5jgz8"
Oct 31 01:54:23.697954 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:54:23.697921    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-5jgz8"]
Oct 31 01:54:23.753696 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:54:23.753668    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/59e32fbb-3cc2-40cc-b2f5-03b3c1d71870-workload-socket\") pod \"e2e-worker-w3-a8e157-openshift-default-7fb96689c7-5jgz8\" (UID: \"59e32fbb-3cc2-40cc-b2f5-03b3c1d71870\") " pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-5jgz8"
Oct 31 01:54:23.753906 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:54:23.753890    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/59e32fbb-3cc2-40cc-b2f5-03b3c1d71870-istio-envoy\") pod \"e2e-worker-w3-a8e157-openshift-default-7fb96689c7-5jgz8\" (UID: \"59e32fbb-3cc2-40cc-b2f5-03b3c1d71870\") " pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-5jgz8"
Oct 31 01:54:23.754006 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:54:23.753996    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/59e32fbb-3cc2-40cc-b2f5-03b3c1d71870-istio-podinfo\") pod \"e2e-worker-w3-a8e157-openshift-default-7fb96689c7-5jgz8\" (UID: \"59e32fbb-3cc2-40cc-b2f5-03b3c1d71870\") " pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-5jgz8"
Oct 31 01:54:23.754102 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:54:23.754088    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/59e32fbb-3cc2-40cc-b2f5-03b3c1d71870-istio-data\") pod \"e2e-worker-w3-a8e157-openshift-default-7fb96689c7-5jgz8\" (UID: \"59e32fbb-3cc2-40cc-b2f5-03b3c1d71870\") " pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-5jgz8"
Oct 31 01:54:23.754272 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:54:23.754248    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/59e32fbb-3cc2-40cc-b2f5-03b3c1d71870-istio-token\") pod \"e2e-worker-w3-a8e157-openshift-default-7fb96689c7-5jgz8\" (UID: \"59e32fbb-3cc2-40cc-b2f5-03b3c1d71870\") " pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-5jgz8"
Oct 31 01:54:23.754342 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:54:23.754303    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/59e32fbb-3cc2-40cc-b2f5-03b3c1d71870-credential-socket\") pod \"e2e-worker-w3-a8e157-openshift-default-7fb96689c7-5jgz8\" (UID: \"59e32fbb-3cc2-40cc-b2f5-03b3c1d71870\") " pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-5jgz8"
Oct 31 01:54:23.754398 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:54:23.754355    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5z2q6\" (UniqueName: \"kubernetes.io/projected/59e32fbb-3cc2-40cc-b2f5-03b3c1d71870-kube-api-access-5z2q6\") pod \"e2e-worker-w3-a8e157-openshift-default-7fb96689c7-5jgz8\" (UID: \"59e32fbb-3cc2-40cc-b2f5-03b3c1d71870\") " pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-5jgz8"
Oct 31 01:54:23.754539 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:54:23.754522    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/59e32fbb-3cc2-40cc-b2f5-03b3c1d71870-istiod-ca-cert\") pod \"e2e-worker-w3-a8e157-openshift-default-7fb96689c7-5jgz8\" (UID: \"59e32fbb-3cc2-40cc-b2f5-03b3c1d71870\") " pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-5jgz8"
Oct 31 01:54:23.754635 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:54:23.754624    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/59e32fbb-3cc2-40cc-b2f5-03b3c1d71870-workload-certs\") pod \"e2e-worker-w3-a8e157-openshift-default-7fb96689c7-5jgz8\" (UID: \"59e32fbb-3cc2-40cc-b2f5-03b3c1d71870\") " pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-5jgz8"
Oct 31 01:54:23.855977 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:54:23.855884    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/59e32fbb-3cc2-40cc-b2f5-03b3c1d71870-credential-socket\") pod \"e2e-worker-w3-a8e157-openshift-default-7fb96689c7-5jgz8\" (UID: \"59e32fbb-3cc2-40cc-b2f5-03b3c1d71870\") " pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-5jgz8"
Oct 31 01:54:23.855977 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:54:23.855934    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5z2q6\" (UniqueName: \"kubernetes.io/projected/59e32fbb-3cc2-40cc-b2f5-03b3c1d71870-kube-api-access-5z2q6\") pod \"e2e-worker-w3-a8e157-openshift-default-7fb96689c7-5jgz8\" (UID: \"59e32fbb-3cc2-40cc-b2f5-03b3c1d71870\") " pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-5jgz8"
Oct 31 01:54:23.856217 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:54:23.856013    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/59e32fbb-3cc2-40cc-b2f5-03b3c1d71870-istiod-ca-cert\") pod \"e2e-worker-w3-a8e157-openshift-default-7fb96689c7-5jgz8\" (UID: \"59e32fbb-3cc2-40cc-b2f5-03b3c1d71870\") " pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-5jgz8"
Oct 31 01:54:23.856217 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:54:23.856101    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/59e32fbb-3cc2-40cc-b2f5-03b3c1d71870-workload-certs\") pod \"e2e-worker-w3-a8e157-openshift-default-7fb96689c7-5jgz8\" (UID: \"59e32fbb-3cc2-40cc-b2f5-03b3c1d71870\") " pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-5jgz8"
Oct 31 01:54:23.856217 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:54:23.856143    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/59e32fbb-3cc2-40cc-b2f5-03b3c1d71870-workload-socket\") pod \"e2e-worker-w3-a8e157-openshift-default-7fb96689c7-5jgz8\" (UID: \"59e32fbb-3cc2-40cc-b2f5-03b3c1d71870\") " pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-5jgz8"
Oct 31 01:54:23.856217 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:54:23.856172    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/59e32fbb-3cc2-40cc-b2f5-03b3c1d71870-istio-envoy\") pod \"e2e-worker-w3-a8e157-openshift-default-7fb96689c7-5jgz8\" (UID: \"59e32fbb-3cc2-40cc-b2f5-03b3c1d71870\") " pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-5jgz8"
Oct 31 01:54:23.856477 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:54:23.856230    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/59e32fbb-3cc2-40cc-b2f5-03b3c1d71870-istio-podinfo\") pod \"e2e-worker-w3-a8e157-openshift-default-7fb96689c7-5jgz8\" (UID: \"59e32fbb-3cc2-40cc-b2f5-03b3c1d71870\") " pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-5jgz8"
Oct 31 01:54:23.856477 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:54:23.856278    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/59e32fbb-3cc2-40cc-b2f5-03b3c1d71870-istio-data\") pod \"e2e-worker-w3-a8e157-openshift-default-7fb96689c7-5jgz8\" (UID: \"59e32fbb-3cc2-40cc-b2f5-03b3c1d71870\") " pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-5jgz8"
Oct 31 01:54:23.856477 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:54:23.856341    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/59e32fbb-3cc2-40cc-b2f5-03b3c1d71870-istio-token\") pod \"e2e-worker-w3-a8e157-openshift-default-7fb96689c7-5jgz8\" (UID: \"59e32fbb-3cc2-40cc-b2f5-03b3c1d71870\") " pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-5jgz8"
Oct 31 01:54:23.856477 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:54:23.856372    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/59e32fbb-3cc2-40cc-b2f5-03b3c1d71870-credential-socket\") pod \"e2e-worker-w3-a8e157-openshift-default-7fb96689c7-5jgz8\" (UID: \"59e32fbb-3cc2-40cc-b2f5-03b3c1d71870\") " pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-5jgz8"
Oct 31 01:54:23.856702 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:54:23.856508    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/59e32fbb-3cc2-40cc-b2f5-03b3c1d71870-workload-certs\") pod \"e2e-worker-w3-a8e157-openshift-default-7fb96689c7-5jgz8\" (UID: \"59e32fbb-3cc2-40cc-b2f5-03b3c1d71870\") " pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-5jgz8"
Oct 31 01:54:23.856702 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:54:23.856629    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/59e32fbb-3cc2-40cc-b2f5-03b3c1d71870-workload-socket\") pod \"e2e-worker-w3-a8e157-openshift-default-7fb96689c7-5jgz8\" (UID: \"59e32fbb-3cc2-40cc-b2f5-03b3c1d71870\") " pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-5jgz8"
Oct 31 01:54:23.856702 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:54:23.856674    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/59e32fbb-3cc2-40cc-b2f5-03b3c1d71870-istio-data\") pod \"e2e-worker-w3-a8e157-openshift-default-7fb96689c7-5jgz8\" (UID: \"59e32fbb-3cc2-40cc-b2f5-03b3c1d71870\") " pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-5jgz8"
Oct 31 01:54:23.856836 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:54:23.856798    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/59e32fbb-3cc2-40cc-b2f5-03b3c1d71870-istiod-ca-cert\") pod \"e2e-worker-w3-a8e157-openshift-default-7fb96689c7-5jgz8\" (UID: \"59e32fbb-3cc2-40cc-b2f5-03b3c1d71870\") " pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-5jgz8"
Oct 31 01:54:23.858797 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:54:23.858769    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/59e32fbb-3cc2-40cc-b2f5-03b3c1d71870-istio-envoy\") pod \"e2e-worker-w3-a8e157-openshift-default-7fb96689c7-5jgz8\" (UID: \"59e32fbb-3cc2-40cc-b2f5-03b3c1d71870\") " pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-5jgz8"
Oct 31 01:54:23.858913 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:54:23.858830    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/59e32fbb-3cc2-40cc-b2f5-03b3c1d71870-istio-podinfo\") pod \"e2e-worker-w3-a8e157-openshift-default-7fb96689c7-5jgz8\" (UID: \"59e32fbb-3cc2-40cc-b2f5-03b3c1d71870\") " pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-5jgz8"
Oct 31 01:54:23.917707 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:54:23.917677    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/59e32fbb-3cc2-40cc-b2f5-03b3c1d71870-istio-token\") pod \"e2e-worker-w3-a8e157-openshift-default-7fb96689c7-5jgz8\" (UID: \"59e32fbb-3cc2-40cc-b2f5-03b3c1d71870\") " pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-5jgz8"
Oct 31 01:54:23.918560 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:54:23.918540    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5z2q6\" (UniqueName: \"kubernetes.io/projected/59e32fbb-3cc2-40cc-b2f5-03b3c1d71870-kube-api-access-5z2q6\") pod \"e2e-worker-w3-a8e157-openshift-default-7fb96689c7-5jgz8\" (UID: \"59e32fbb-3cc2-40cc-b2f5-03b3c1d71870\") " pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-5jgz8"
Oct 31 01:54:23.930704 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:54:23.930680    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-5jgz8"
Oct 31 01:54:24.363092 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:54:24.363067    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-5jgz8"]
Oct 31 01:54:24.364493 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:54:24.364470    4964 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod59e32fbb_3cc2_40cc_b2f5_03b3c1d71870.slice/crio-c3fa5861612e2b6a0f6a90100104409f96c115c9ead0a3233d3299a173f570ba WatchSource:0}: Error finding container c3fa5861612e2b6a0f6a90100104409f96c115c9ead0a3233d3299a173f570ba: Status 404 returned error can't find the container with id c3fa5861612e2b6a0f6a90100104409f96c115c9ead0a3233d3299a173f570ba
Oct 31 01:54:24.367153 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:54:24.367118    4964 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 01:54:24.367243 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:54:24.367195    4964 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 01:54:24.367243 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:54:24.367238    4964 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 01:54:24.419784 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:54:24.419731    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:54:24.547948 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:54:24.547911    4964 patch_prober.go:29] interesting pod/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-4xmfp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.166:15021/healthz/ready\": dial tcp 10.132.0.166:15021: connect: connection refused" start-of-body=
Oct 31 01:54:24.548148 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:54:24.547978    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-4xmfp" podUID="bb5fa688-49e8-4e29-9643-bab6160bec86" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.166:15021/healthz/ready\": dial tcp 10.132.0.166:15021: connect: connection refused"
Oct 31 01:54:25.370225 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:54:25.370183    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-5jgz8" event={"ID":"59e32fbb-3cc2-40cc-b2f5-03b3c1d71870","Type":"ContainerStarted","Data":"910e66802e5f0d58fde77ba312626cc5b6b776d4112bae6c68076da98709d1fc"}
Oct 31 01:54:25.370975 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:54:25.370952    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-5jgz8" event={"ID":"59e32fbb-3cc2-40cc-b2f5-03b3c1d71870","Type":"ContainerStarted","Data":"c3fa5861612e2b6a0f6a90100104409f96c115c9ead0a3233d3299a173f570ba"}
Oct 31 01:54:25.548254 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:54:25.547799    4964 patch_prober.go:29] interesting pod/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-4xmfp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.166:15021/healthz/ready\": dial tcp 10.132.0.166:15021: connect: connection refused" start-of-body=
Oct 31 01:54:25.548254 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:54:25.547859    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-4xmfp" podUID="bb5fa688-49e8-4e29-9643-bab6160bec86" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.166:15021/healthz/ready\": dial tcp 10.132.0.166:15021: connect: connection refused"
Oct 31 01:54:25.953952 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:54:25.953811    4964 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-5jgz8"
Oct 31 01:54:25.958200 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:54:25.957843    4964 patch_prober.go:29] interesting pod/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-5jgz8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused" start-of-body=
Oct 31 01:54:25.958200 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:54:25.957897    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-5jgz8" podUID="59e32fbb-3cc2-40cc-b2f5-03b3c1d71870" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused"
Oct 31 01:54:26.265305 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:54:26.265269    4964 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-8k9j8"
Oct 31 01:54:26.282769 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:54:26.282741    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c6252303-98d9-43e9-b6ca-f9087cf79430-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c6252303-98d9-43e9-b6ca-f9087cf79430-istio-podinfo\") pod \"c6252303-98d9-43e9-b6ca-f9087cf79430\" (UID: \"c6252303-98d9-43e9-b6ca-f9087cf79430\") "
Oct 31 01:54:26.282878 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:54:26.282805    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c6252303-98d9-43e9-b6ca-f9087cf79430-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6252303-98d9-43e9-b6ca-f9087cf79430-workload-socket\") pod \"c6252303-98d9-43e9-b6ca-f9087cf79430\" (UID: \"c6252303-98d9-43e9-b6ca-f9087cf79430\") "
Oct 31 01:54:26.282878 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:54:26.282842    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c6252303-98d9-43e9-b6ca-f9087cf79430-istio-token\" (UniqueName: \"kubernetes.io/projected/c6252303-98d9-43e9-b6ca-f9087cf79430-istio-token\") pod \"c6252303-98d9-43e9-b6ca-f9087cf79430\" (UID: \"c6252303-98d9-43e9-b6ca-f9087cf79430\") "
Oct 31 01:54:26.282949 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:54:26.282900    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c6252303-98d9-43e9-b6ca-f9087cf79430-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c6252303-98d9-43e9-b6ca-f9087cf79430-istio-envoy\") pod \"c6252303-98d9-43e9-b6ca-f9087cf79430\" (UID: \"c6252303-98d9-43e9-b6ca-f9087cf79430\") "
Oct 31 01:54:26.282949 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:54:26.282943    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c6252303-98d9-43e9-b6ca-f9087cf79430-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c6252303-98d9-43e9-b6ca-f9087cf79430-istio-data\") pod \"c6252303-98d9-43e9-b6ca-f9087cf79430\" (UID: \"c6252303-98d9-43e9-b6ca-f9087cf79430\") "
Oct 31 01:54:26.283012 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:54:26.282997    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c6252303-98d9-43e9-b6ca-f9087cf79430-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c6252303-98d9-43e9-b6ca-f9087cf79430-istiod-ca-cert\") pod \"c6252303-98d9-43e9-b6ca-f9087cf79430\" (UID: \"c6252303-98d9-43e9-b6ca-f9087cf79430\") "
Oct 31 01:54:26.283045 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:54:26.283029    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c6252303-98d9-43e9-b6ca-f9087cf79430-kube-api-access-s6558\" (UniqueName: \"kubernetes.io/projected/c6252303-98d9-43e9-b6ca-f9087cf79430-kube-api-access-s6558\") pod \"c6252303-98d9-43e9-b6ca-f9087cf79430\" (UID: \"c6252303-98d9-43e9-b6ca-f9087cf79430\") "
Oct 31 01:54:26.283110 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:54:26.283089    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6252303-98d9-43e9-b6ca-f9087cf79430-workload-socket" pod "c6252303-98d9-43e9-b6ca-f9087cf79430" (UID: "c6252303-98d9-43e9-b6ca-f9087cf79430"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:54:26.283201 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:54:26.283182    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c6252303-98d9-43e9-b6ca-f9087cf79430-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6252303-98d9-43e9-b6ca-f9087cf79430-credential-socket\") pod \"c6252303-98d9-43e9-b6ca-f9087cf79430\" (UID: \"c6252303-98d9-43e9-b6ca-f9087cf79430\") "
Oct 31 01:54:26.283262 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:54:26.283228    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6252303-98d9-43e9-b6ca-f9087cf79430-istio-data" pod "c6252303-98d9-43e9-b6ca-f9087cf79430" (UID: "c6252303-98d9-43e9-b6ca-f9087cf79430"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:54:26.283262 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:54:26.283229    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c6252303-98d9-43e9-b6ca-f9087cf79430-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c6252303-98d9-43e9-b6ca-f9087cf79430-workload-certs\") pod \"c6252303-98d9-43e9-b6ca-f9087cf79430\" (UID: \"c6252303-98d9-43e9-b6ca-f9087cf79430\") "
Oct 31 01:54:26.283776 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:54:26.283424    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6252303-98d9-43e9-b6ca-f9087cf79430-workload-certs" pod "c6252303-98d9-43e9-b6ca-f9087cf79430" (UID: "c6252303-98d9-43e9-b6ca-f9087cf79430"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:54:26.283776 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:54:26.283613    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6252303-98d9-43e9-b6ca-f9087cf79430-credential-socket" pod "c6252303-98d9-43e9-b6ca-f9087cf79430" (UID: "c6252303-98d9-43e9-b6ca-f9087cf79430"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:54:26.283776 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:54:26.283771    4964 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6252303-98d9-43e9-b6ca-f9087cf79430-credential-socket\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:54:26.283941 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:54:26.283779    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c6252303-98d9-43e9-b6ca-f9087cf79430-istiod-ca-cert" pod "c6252303-98d9-43e9-b6ca-f9087cf79430" (UID: "c6252303-98d9-43e9-b6ca-f9087cf79430"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:54:26.283941 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:54:26.283793    4964 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c6252303-98d9-43e9-b6ca-f9087cf79430-workload-certs\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:54:26.283941 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:54:26.283808    4964 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6252303-98d9-43e9-b6ca-f9087cf79430-workload-socket\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:54:26.283941 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:54:26.283821    4964 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c6252303-98d9-43e9-b6ca-f9087cf79430-istio-data\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:54:26.285865 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:54:26.285650    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6252303-98d9-43e9-b6ca-f9087cf79430-istio-envoy" pod "c6252303-98d9-43e9-b6ca-f9087cf79430" (UID: "c6252303-98d9-43e9-b6ca-f9087cf79430"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:54:26.285865 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:54:26.285657    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c6252303-98d9-43e9-b6ca-f9087cf79430-kube-api-access-s6558" pod "c6252303-98d9-43e9-b6ca-f9087cf79430" (UID: "c6252303-98d9-43e9-b6ca-f9087cf79430"). InnerVolumeSpecName "kube-api-access-s6558". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:54:26.286403 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:54:26.286376    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c6252303-98d9-43e9-b6ca-f9087cf79430-istio-token" pod "c6252303-98d9-43e9-b6ca-f9087cf79430" (UID: "c6252303-98d9-43e9-b6ca-f9087cf79430"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:54:26.286403 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:54:26.286387    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c6252303-98d9-43e9-b6ca-f9087cf79430-istio-podinfo" pod "c6252303-98d9-43e9-b6ca-f9087cf79430" (UID: "c6252303-98d9-43e9-b6ca-f9087cf79430"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 01:54:26.377440 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:54:26.377403    4964 generic.go:356] "Generic (PLEG): container finished" podID="c6252303-98d9-43e9-b6ca-f9087cf79430" containerID="a26ab811e35fc62fd8e9068f88b29a50a27e22a65c1974c198be545af1f4f515" exitCode=0
Oct 31 01:54:26.377911 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:54:26.377449    4964 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-8k9j8"
Oct 31 01:54:26.377911 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:54:26.377858    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-8k9j8" event={"ID":"c6252303-98d9-43e9-b6ca-f9087cf79430","Type":"ContainerDied","Data":"a26ab811e35fc62fd8e9068f88b29a50a27e22a65c1974c198be545af1f4f515"}
Oct 31 01:54:26.378027 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:54:26.377920    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-8k9j8" event={"ID":"c6252303-98d9-43e9-b6ca-f9087cf79430","Type":"ContainerDied","Data":"5c52d9fd41dba24fc6e0a949177ee1f075fbf4032994aa97c41380bb29b38738"}
Oct 31 01:54:26.378027 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:54:26.377943    4964 scope.go:122] "RemoveContainer" containerID="a26ab811e35fc62fd8e9068f88b29a50a27e22a65c1974c198be545af1f4f515"
Oct 31 01:54:26.384975 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:54:26.384947    4964 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c6252303-98d9-43e9-b6ca-f9087cf79430-istio-podinfo\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:54:26.384975 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:54:26.384975    4964 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c6252303-98d9-43e9-b6ca-f9087cf79430-istio-token\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:54:26.384975 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:54:26.384991    4964 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c6252303-98d9-43e9-b6ca-f9087cf79430-istio-envoy\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:54:26.384975 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:54:26.385003    4964 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c6252303-98d9-43e9-b6ca-f9087cf79430-istiod-ca-cert\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:54:26.384975 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:54:26.385016    4964 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s6558\" (UniqueName: \"kubernetes.io/projected/c6252303-98d9-43e9-b6ca-f9087cf79430-kube-api-access-s6558\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:54:26.389809 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:54:26.389791    4964 scope.go:122] "RemoveContainer" containerID="a26ab811e35fc62fd8e9068f88b29a50a27e22a65c1974c198be545af1f4f515"
Oct 31 01:54:26.390090 ip-10-0-133-197 kubenswrapper[4964]: E1031 01:54:26.390067    4964 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a26ab811e35fc62fd8e9068f88b29a50a27e22a65c1974c198be545af1f4f515\": container with ID starting with a26ab811e35fc62fd8e9068f88b29a50a27e22a65c1974c198be545af1f4f515 not found: ID does not exist" containerID="a26ab811e35fc62fd8e9068f88b29a50a27e22a65c1974c198be545af1f4f515"
Oct 31 01:54:26.390145 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:54:26.390102    4964 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a26ab811e35fc62fd8e9068f88b29a50a27e22a65c1974c198be545af1f4f515"} err="failed to get container status \"a26ab811e35fc62fd8e9068f88b29a50a27e22a65c1974c198be545af1f4f515\": rpc error: code = NotFound desc = could not find container \"a26ab811e35fc62fd8e9068f88b29a50a27e22a65c1974c198be545af1f4f515\": container with ID starting with a26ab811e35fc62fd8e9068f88b29a50a27e22a65c1974c198be545af1f4f515 not found: ID does not exist"
Oct 31 01:54:26.468240 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:54:26.468167    4964 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-5jgz8" podStartSLOduration=3.468147548 podStartE2EDuration="3.468147548s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:54:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:54:25.539639105 +0000 UTC m=+3034.016205488" watchObservedRunningTime="2026-10-31 01:54:26.468147548 +0000 UTC m=+3034.944713929"
Oct 31 01:54:26.479720 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:54:26.479691    4964 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-psgmw"]
Oct 31 01:54:26.480263 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:54:26.480249    4964 cpu_manager.go:414] "containerMap: removing container" podUID="c6252303-98d9-43e9-b6ca-f9087cf79430" containerName="istio-proxy"
Oct 31 01:54:26.480263 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:54:26.480265    4964 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c6252303-98d9-43e9-b6ca-f9087cf79430" containerName="istio-proxy"
Oct 31 01:54:26.480456 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:54:26.480390    4964 memory_manager.go:365] "RemoveStaleState removing state" podUID="c6252303-98d9-43e9-b6ca-f9087cf79430" containerName="istio-proxy"
Oct 31 01:54:26.485354 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:54:26.485274    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-psgmw"
Oct 31 01:54:26.562750 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:54:26.555517    4964 patch_prober.go:29] interesting pod/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-4xmfp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.166:15021/healthz/ready\": dial tcp 10.132.0.166:15021: connect: connection refused" start-of-body=
Oct 31 01:54:26.562750 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:54:26.555573    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-4xmfp" podUID="bb5fa688-49e8-4e29-9643-bab6160bec86" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.166:15021/healthz/ready\": dial tcp 10.132.0.166:15021: connect: connection refused"
Oct 31 01:54:26.575638 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:54:26.575611    4964 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-8k9j8"]
Oct 31 01:54:26.580014 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:54:26.579987    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-psgmw"]
Oct 31 01:54:26.588225 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:54:26.587200    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a331bf16-e6db-4827-a796-b6a3cca34ead-istio-data\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-psgmw\" (UID: \"a331bf16-e6db-4827-a796-b6a3cca34ead\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-psgmw"
Oct 31 01:54:26.588225 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:54:26.587279    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a331bf16-e6db-4827-a796-b6a3cca34ead-credential-socket\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-psgmw\" (UID: \"a331bf16-e6db-4827-a796-b6a3cca34ead\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-psgmw"
Oct 31 01:54:26.588225 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:54:26.587346    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a331bf16-e6db-4827-a796-b6a3cca34ead-istiod-ca-cert\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-psgmw\" (UID: \"a331bf16-e6db-4827-a796-b6a3cca34ead\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-psgmw"
Oct 31 01:54:26.588225 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:54:26.587381    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a331bf16-e6db-4827-a796-b6a3cca34ead-workload-socket\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-psgmw\" (UID: \"a331bf16-e6db-4827-a796-b6a3cca34ead\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-psgmw"
Oct 31 01:54:26.588225 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:54:26.587411    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a331bf16-e6db-4827-a796-b6a3cca34ead-workload-certs\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-psgmw\" (UID: \"a331bf16-e6db-4827-a796-b6a3cca34ead\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-psgmw"
Oct 31 01:54:26.588225 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:54:26.587435    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wq5qz\" (UniqueName: \"kubernetes.io/projected/a331bf16-e6db-4827-a796-b6a3cca34ead-kube-api-access-wq5qz\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-psgmw\" (UID: \"a331bf16-e6db-4827-a796-b6a3cca34ead\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-psgmw"
Oct 31 01:54:26.588225 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:54:26.587506    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a331bf16-e6db-4827-a796-b6a3cca34ead-istio-podinfo\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-psgmw\" (UID: \"a331bf16-e6db-4827-a796-b6a3cca34ead\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-psgmw"
Oct 31 01:54:26.588225 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:54:26.587540    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a331bf16-e6db-4827-a796-b6a3cca34ead-istio-envoy\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-psgmw\" (UID: \"a331bf16-e6db-4827-a796-b6a3cca34ead\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-psgmw"
Oct 31 01:54:26.588225 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:54:26.587582    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a331bf16-e6db-4827-a796-b6a3cca34ead-istio-token\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-psgmw\" (UID: \"a331bf16-e6db-4827-a796-b6a3cca34ead\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-psgmw"
Oct 31 01:54:26.589550 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:54:26.589527    4964 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-8k9j8"]
Oct 31 01:54:26.688617 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:54:26.688582    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a331bf16-e6db-4827-a796-b6a3cca34ead-istio-token\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-psgmw\" (UID: \"a331bf16-e6db-4827-a796-b6a3cca34ead\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-psgmw"
Oct 31 01:54:26.688800 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:54:26.688666    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a331bf16-e6db-4827-a796-b6a3cca34ead-istio-data\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-psgmw\" (UID: \"a331bf16-e6db-4827-a796-b6a3cca34ead\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-psgmw"
Oct 31 01:54:26.688800 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:54:26.688727    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a331bf16-e6db-4827-a796-b6a3cca34ead-credential-socket\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-psgmw\" (UID: \"a331bf16-e6db-4827-a796-b6a3cca34ead\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-psgmw"
Oct 31 01:54:26.688800 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:54:26.688762    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a331bf16-e6db-4827-a796-b6a3cca34ead-istiod-ca-cert\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-psgmw\" (UID: \"a331bf16-e6db-4827-a796-b6a3cca34ead\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-psgmw"
Oct 31 01:54:26.688800 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:54:26.688793    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a331bf16-e6db-4827-a796-b6a3cca34ead-workload-socket\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-psgmw\" (UID: \"a331bf16-e6db-4827-a796-b6a3cca34ead\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-psgmw"
Oct 31 01:54:26.689024 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:54:26.688812    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a331bf16-e6db-4827-a796-b6a3cca34ead-workload-certs\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-psgmw\" (UID: \"a331bf16-e6db-4827-a796-b6a3cca34ead\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-psgmw"
Oct 31 01:54:26.689024 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:54:26.688828    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wq5qz\" (UniqueName: \"kubernetes.io/projected/a331bf16-e6db-4827-a796-b6a3cca34ead-kube-api-access-wq5qz\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-psgmw\" (UID: \"a331bf16-e6db-4827-a796-b6a3cca34ead\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-psgmw"
Oct 31 01:54:26.689024 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:54:26.688884    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a331bf16-e6db-4827-a796-b6a3cca34ead-istio-podinfo\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-psgmw\" (UID: \"a331bf16-e6db-4827-a796-b6a3cca34ead\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-psgmw"
Oct 31 01:54:26.689024 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:54:26.688921    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a331bf16-e6db-4827-a796-b6a3cca34ead-istio-envoy\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-psgmw\" (UID: \"a331bf16-e6db-4827-a796-b6a3cca34ead\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-psgmw"
Oct 31 01:54:26.689251 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:54:26.689128    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a331bf16-e6db-4827-a796-b6a3cca34ead-istio-data\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-psgmw\" (UID: \"a331bf16-e6db-4827-a796-b6a3cca34ead\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-psgmw"
Oct 31 01:54:26.689570 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:54:26.689544    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a331bf16-e6db-4827-a796-b6a3cca34ead-istiod-ca-cert\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-psgmw\" (UID: \"a331bf16-e6db-4827-a796-b6a3cca34ead\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-psgmw"
Oct 31 01:54:26.689875 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:54:26.689831    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a331bf16-e6db-4827-a796-b6a3cca34ead-workload-socket\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-psgmw\" (UID: \"a331bf16-e6db-4827-a796-b6a3cca34ead\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-psgmw"
Oct 31 01:54:26.689875 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:54:26.689848    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a331bf16-e6db-4827-a796-b6a3cca34ead-workload-certs\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-psgmw\" (UID: \"a331bf16-e6db-4827-a796-b6a3cca34ead\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-psgmw"
Oct 31 01:54:26.690044 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:54:26.689908    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a331bf16-e6db-4827-a796-b6a3cca34ead-credential-socket\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-psgmw\" (UID: \"a331bf16-e6db-4827-a796-b6a3cca34ead\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-psgmw"
Oct 31 01:54:26.758147 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:54:26.758110    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a331bf16-e6db-4827-a796-b6a3cca34ead-istio-token\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-psgmw\" (UID: \"a331bf16-e6db-4827-a796-b6a3cca34ead\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-psgmw"
Oct 31 01:54:26.760694 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:54:26.760622    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wq5qz\" (UniqueName: \"kubernetes.io/projected/a331bf16-e6db-4827-a796-b6a3cca34ead-kube-api-access-wq5qz\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-psgmw\" (UID: \"a331bf16-e6db-4827-a796-b6a3cca34ead\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-psgmw"
Oct 31 01:54:26.761134 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:54:26.761112    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a331bf16-e6db-4827-a796-b6a3cca34ead-istio-envoy\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-psgmw\" (UID: \"a331bf16-e6db-4827-a796-b6a3cca34ead\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-psgmw"
Oct 31 01:54:26.764340 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:54:26.762927    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:54:26.764340 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:54:26.763625    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a331bf16-e6db-4827-a796-b6a3cca34ead-istio-podinfo\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-psgmw\" (UID: \"a331bf16-e6db-4827-a796-b6a3cca34ead\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-psgmw"
Oct 31 01:54:26.866607 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:54:26.866517    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-psgmw"
Oct 31 01:54:26.937509 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:54:26.934914    4964 patch_prober.go:29] interesting pod/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-5jgz8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused" start-of-body=
Oct 31 01:54:26.937509 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:54:26.934971    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-5jgz8" podUID="59e32fbb-3cc2-40cc-b2f5-03b3c1d71870" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused"
Oct 31 01:54:27.073477 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:54:27.073432    4964 patch_prober.go:29] interesting pod/e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-8k9j8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.159:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 01:54:27.073666 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:54:27.073513    4964 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-8k9j8" podUID="c6252303-98d9-43e9-b6ca-f9087cf79430" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.159:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:54:27.350821 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:54:27.350796    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-psgmw"]
Oct 31 01:54:27.353628 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:54:27.353600    4964 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda331bf16_e6db_4827_a796_b6a3cca34ead.slice/crio-e5d7320266914d9f00f9ca55430d4b693551a1c47d87d2ab81af1426b1ef9d23 WatchSource:0}: Error finding container e5d7320266914d9f00f9ca55430d4b693551a1c47d87d2ab81af1426b1ef9d23: Status 404 returned error can't find the container with id e5d7320266914d9f00f9ca55430d4b693551a1c47d87d2ab81af1426b1ef9d23
Oct 31 01:54:27.356634 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:54:27.356602    4964 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 01:54:27.356746 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:54:27.356663    4964 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 01:54:27.356746 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:54:27.356692    4964 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 01:54:27.386996 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:54:27.386973    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-psgmw" event={"ID":"a331bf16-e6db-4827-a796-b6a3cca34ead","Type":"ContainerStarted","Data":"e5d7320266914d9f00f9ca55430d4b693551a1c47d87d2ab81af1426b1ef9d23"}
Oct 31 01:54:27.548552 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:54:27.548515    4964 patch_prober.go:29] interesting pod/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-4xmfp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.166:15021/healthz/ready\": dial tcp 10.132.0.166:15021: connect: connection refused" start-of-body=
Oct 31 01:54:27.548736 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:54:27.548580    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-4xmfp" podUID="bb5fa688-49e8-4e29-9643-bab6160bec86" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.166:15021/healthz/ready\": dial tcp 10.132.0.166:15021: connect: connection refused"
Oct 31 01:54:27.776904 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:54:27.776862    4964 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c6252303-98d9-43e9-b6ca-f9087cf79430" path="/var/lib/kubelet/pods/c6252303-98d9-43e9-b6ca-f9087cf79430/volumes"
Oct 31 01:54:27.932021 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:54:27.931983    4964 patch_prober.go:29] interesting pod/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-5jgz8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused" start-of-body=
Oct 31 01:54:27.932211 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:54:27.932041    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-5jgz8" podUID="59e32fbb-3cc2-40cc-b2f5-03b3c1d71870" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused"
Oct 31 01:54:27.950376 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:54:27.950309    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:54:28.405282 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:54:28.405157    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-psgmw" event={"ID":"a331bf16-e6db-4827-a796-b6a3cca34ead","Type":"ContainerStarted","Data":"178462a4a89e2e8f0d7dd44478445cf52c24cbce514be5af9c369e8eecb4f763"}
Oct 31 01:54:28.528038 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:54:28.527057    4964 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-psgmw" podStartSLOduration=2.527036398 podStartE2EDuration="2.527036398s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:54:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:54:28.523798711 +0000 UTC m=+3037.000365093" watchObservedRunningTime="2026-10-31 01:54:28.527036398 +0000 UTC m=+3037.003602776"
Oct 31 01:54:28.553755 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:54:28.553713    4964 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-b-476ce2-openshift-default-6df8c7ff5c-wqzzs"]
Oct 31 01:54:28.563502 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:54:28.563472    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-476ce2-openshift-default-6df8c7ff5c-wqzzs"
Oct 31 01:54:28.759960 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:54:28.759741    4964 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-b-476ce2-openshift-default-dockercfg-x8z7q\""
Oct 31 01:54:28.784726 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:54:28.783255    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x95wq\" (UniqueName: \"kubernetes.io/projected/5a313b20-f6bb-446c-b23b-925c75a5df66-kube-api-access-x95wq\") pod \"e2e-praxis-b-476ce2-openshift-default-6df8c7ff5c-wqzzs\" (UID: \"5a313b20-f6bb-446c-b23b-925c75a5df66\") " pod="openshift-ingress/e2e-praxis-b-476ce2-openshift-default-6df8c7ff5c-wqzzs"
Oct 31 01:54:28.784726 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:54:28.783306    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a313b20-f6bb-446c-b23b-925c75a5df66-istio-data\") pod \"e2e-praxis-b-476ce2-openshift-default-6df8c7ff5c-wqzzs\" (UID: \"5a313b20-f6bb-446c-b23b-925c75a5df66\") " pod="openshift-ingress/e2e-praxis-b-476ce2-openshift-default-6df8c7ff5c-wqzzs"
Oct 31 01:54:28.784726 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:54:28.783357    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a313b20-f6bb-446c-b23b-925c75a5df66-istiod-ca-cert\") pod \"e2e-praxis-b-476ce2-openshift-default-6df8c7ff5c-wqzzs\" (UID: \"5a313b20-f6bb-446c-b23b-925c75a5df66\") " pod="openshift-ingress/e2e-praxis-b-476ce2-openshift-default-6df8c7ff5c-wqzzs"
Oct 31 01:54:28.784726 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:54:28.783420    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a313b20-f6bb-446c-b23b-925c75a5df66-workload-socket\") pod \"e2e-praxis-b-476ce2-openshift-default-6df8c7ff5c-wqzzs\" (UID: \"5a313b20-f6bb-446c-b23b-925c75a5df66\") " pod="openshift-ingress/e2e-praxis-b-476ce2-openshift-default-6df8c7ff5c-wqzzs"
Oct 31 01:54:28.784726 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:54:28.783468    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5a313b20-f6bb-446c-b23b-925c75a5df66-istio-token\") pod \"e2e-praxis-b-476ce2-openshift-default-6df8c7ff5c-wqzzs\" (UID: \"5a313b20-f6bb-446c-b23b-925c75a5df66\") " pod="openshift-ingress/e2e-praxis-b-476ce2-openshift-default-6df8c7ff5c-wqzzs"
Oct 31 01:54:28.784726 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:54:28.783494    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a313b20-f6bb-446c-b23b-925c75a5df66-credential-socket\") pod \"e2e-praxis-b-476ce2-openshift-default-6df8c7ff5c-wqzzs\" (UID: \"5a313b20-f6bb-446c-b23b-925c75a5df66\") " pod="openshift-ingress/e2e-praxis-b-476ce2-openshift-default-6df8c7ff5c-wqzzs"
Oct 31 01:54:28.784726 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:54:28.783590    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a313b20-f6bb-446c-b23b-925c75a5df66-workload-certs\") pod \"e2e-praxis-b-476ce2-openshift-default-6df8c7ff5c-wqzzs\" (UID: \"5a313b20-f6bb-446c-b23b-925c75a5df66\") " pod="openshift-ingress/e2e-praxis-b-476ce2-openshift-default-6df8c7ff5c-wqzzs"
Oct 31 01:54:28.784726 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:54:28.783698    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a313b20-f6bb-446c-b23b-925c75a5df66-istio-envoy\") pod \"e2e-praxis-b-476ce2-openshift-default-6df8c7ff5c-wqzzs\" (UID: \"5a313b20-f6bb-446c-b23b-925c75a5df66\") " pod="openshift-ingress/e2e-praxis-b-476ce2-openshift-default-6df8c7ff5c-wqzzs"
Oct 31 01:54:28.784726 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:54:28.783731    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a313b20-f6bb-446c-b23b-925c75a5df66-istio-podinfo\") pod \"e2e-praxis-b-476ce2-openshift-default-6df8c7ff5c-wqzzs\" (UID: \"5a313b20-f6bb-446c-b23b-925c75a5df66\") " pod="openshift-ingress/e2e-praxis-b-476ce2-openshift-default-6df8c7ff5c-wqzzs"
Oct 31 01:54:28.784726 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:54:28.783921    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-476ce2-openshift-default-6df8c7ff5c-wqzzs"]
Oct 31 01:54:28.817384 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:54:28.788679    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:54:28.874343 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:54:28.873730    4964 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-psgmw"
Oct 31 01:54:28.874343 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:54:28.874204    4964 patch_prober.go:29] interesting pod/e2e-worker-w2-00e840-openshift-default-59b96fb879-psgmw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.168:15021/healthz/ready\": dial tcp 10.132.0.168:15021: connect: connection refused" start-of-body=
Oct 31 01:54:28.874343 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:54:28.874250    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-psgmw" podUID="a331bf16-e6db-4827-a796-b6a3cca34ead" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.168:15021/healthz/ready\": dial tcp 10.132.0.168:15021: connect: connection refused"
Oct 31 01:54:28.885546 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:54:28.885517    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a313b20-f6bb-446c-b23b-925c75a5df66-istio-podinfo\") pod \"e2e-praxis-b-476ce2-openshift-default-6df8c7ff5c-wqzzs\" (UID: \"5a313b20-f6bb-446c-b23b-925c75a5df66\") " pod="openshift-ingress/e2e-praxis-b-476ce2-openshift-default-6df8c7ff5c-wqzzs"
Oct 31 01:54:28.885695 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:54:28.885618    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x95wq\" (UniqueName: \"kubernetes.io/projected/5a313b20-f6bb-446c-b23b-925c75a5df66-kube-api-access-x95wq\") pod \"e2e-praxis-b-476ce2-openshift-default-6df8c7ff5c-wqzzs\" (UID: \"5a313b20-f6bb-446c-b23b-925c75a5df66\") " pod="openshift-ingress/e2e-praxis-b-476ce2-openshift-default-6df8c7ff5c-wqzzs"
Oct 31 01:54:28.885695 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:54:28.885659    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a313b20-f6bb-446c-b23b-925c75a5df66-istio-data\") pod \"e2e-praxis-b-476ce2-openshift-default-6df8c7ff5c-wqzzs\" (UID: \"5a313b20-f6bb-446c-b23b-925c75a5df66\") " pod="openshift-ingress/e2e-praxis-b-476ce2-openshift-default-6df8c7ff5c-wqzzs"
Oct 31 01:54:28.885776 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:54:28.885683    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a313b20-f6bb-446c-b23b-925c75a5df66-istiod-ca-cert\") pod \"e2e-praxis-b-476ce2-openshift-default-6df8c7ff5c-wqzzs\" (UID: \"5a313b20-f6bb-446c-b23b-925c75a5df66\") " pod="openshift-ingress/e2e-praxis-b-476ce2-openshift-default-6df8c7ff5c-wqzzs"
Oct 31 01:54:28.885835 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:54:28.885819    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a313b20-f6bb-446c-b23b-925c75a5df66-workload-socket\") pod \"e2e-praxis-b-476ce2-openshift-default-6df8c7ff5c-wqzzs\" (UID: \"5a313b20-f6bb-446c-b23b-925c75a5df66\") " pod="openshift-ingress/e2e-praxis-b-476ce2-openshift-default-6df8c7ff5c-wqzzs"
Oct 31 01:54:28.885899 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:54:28.885884    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5a313b20-f6bb-446c-b23b-925c75a5df66-istio-token\") pod \"e2e-praxis-b-476ce2-openshift-default-6df8c7ff5c-wqzzs\" (UID: \"5a313b20-f6bb-446c-b23b-925c75a5df66\") " pod="openshift-ingress/e2e-praxis-b-476ce2-openshift-default-6df8c7ff5c-wqzzs"
Oct 31 01:54:28.885944 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:54:28.885915    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a313b20-f6bb-446c-b23b-925c75a5df66-credential-socket\") pod \"e2e-praxis-b-476ce2-openshift-default-6df8c7ff5c-wqzzs\" (UID: \"5a313b20-f6bb-446c-b23b-925c75a5df66\") " pod="openshift-ingress/e2e-praxis-b-476ce2-openshift-default-6df8c7ff5c-wqzzs"
Oct 31 01:54:28.886048 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:54:28.886032    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a313b20-f6bb-446c-b23b-925c75a5df66-workload-certs\") pod \"e2e-praxis-b-476ce2-openshift-default-6df8c7ff5c-wqzzs\" (UID: \"5a313b20-f6bb-446c-b23b-925c75a5df66\") " pod="openshift-ingress/e2e-praxis-b-476ce2-openshift-default-6df8c7ff5c-wqzzs"
Oct 31 01:54:28.886210 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:54:28.886193    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a313b20-f6bb-446c-b23b-925c75a5df66-istio-envoy\") pod \"e2e-praxis-b-476ce2-openshift-default-6df8c7ff5c-wqzzs\" (UID: \"5a313b20-f6bb-446c-b23b-925c75a5df66\") " pod="openshift-ingress/e2e-praxis-b-476ce2-openshift-default-6df8c7ff5c-wqzzs"
Oct 31 01:54:28.887630 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:54:28.887009    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a313b20-f6bb-446c-b23b-925c75a5df66-workload-socket\") pod \"e2e-praxis-b-476ce2-openshift-default-6df8c7ff5c-wqzzs\" (UID: \"5a313b20-f6bb-446c-b23b-925c75a5df66\") " pod="openshift-ingress/e2e-praxis-b-476ce2-openshift-default-6df8c7ff5c-wqzzs"
Oct 31 01:54:28.887630 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:54:28.887158    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a313b20-f6bb-446c-b23b-925c75a5df66-credential-socket\") pod \"e2e-praxis-b-476ce2-openshift-default-6df8c7ff5c-wqzzs\" (UID: \"5a313b20-f6bb-446c-b23b-925c75a5df66\") " pod="openshift-ingress/e2e-praxis-b-476ce2-openshift-default-6df8c7ff5c-wqzzs"
Oct 31 01:54:28.887630 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:54:28.887431    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a313b20-f6bb-446c-b23b-925c75a5df66-workload-certs\") pod \"e2e-praxis-b-476ce2-openshift-default-6df8c7ff5c-wqzzs\" (UID: \"5a313b20-f6bb-446c-b23b-925c75a5df66\") " pod="openshift-ingress/e2e-praxis-b-476ce2-openshift-default-6df8c7ff5c-wqzzs"
Oct 31 01:54:28.887916 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:54:28.887672    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a313b20-f6bb-446c-b23b-925c75a5df66-istio-data\") pod \"e2e-praxis-b-476ce2-openshift-default-6df8c7ff5c-wqzzs\" (UID: \"5a313b20-f6bb-446c-b23b-925c75a5df66\") " pod="openshift-ingress/e2e-praxis-b-476ce2-openshift-default-6df8c7ff5c-wqzzs"
Oct 31 01:54:28.888666 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:54:28.888636    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a313b20-f6bb-446c-b23b-925c75a5df66-istiod-ca-cert\") pod \"e2e-praxis-b-476ce2-openshift-default-6df8c7ff5c-wqzzs\" (UID: \"5a313b20-f6bb-446c-b23b-925c75a5df66\") " pod="openshift-ingress/e2e-praxis-b-476ce2-openshift-default-6df8c7ff5c-wqzzs"
Oct 31 01:54:28.894336 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:54:28.890803    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a313b20-f6bb-446c-b23b-925c75a5df66-istio-podinfo\") pod \"e2e-praxis-b-476ce2-openshift-default-6df8c7ff5c-wqzzs\" (UID: \"5a313b20-f6bb-446c-b23b-925c75a5df66\") " pod="openshift-ingress/e2e-praxis-b-476ce2-openshift-default-6df8c7ff5c-wqzzs"
Oct 31 01:54:28.959645 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:54:28.957752    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a313b20-f6bb-446c-b23b-925c75a5df66-istio-envoy\") pod \"e2e-praxis-b-476ce2-openshift-default-6df8c7ff5c-wqzzs\" (UID: \"5a313b20-f6bb-446c-b23b-925c75a5df66\") " pod="openshift-ingress/e2e-praxis-b-476ce2-openshift-default-6df8c7ff5c-wqzzs"
Oct 31 01:54:28.959645 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:54:28.957876    4964 patch_prober.go:29] interesting pod/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-5jgz8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused" start-of-body=
Oct 31 01:54:28.959645 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:54:28.957929    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-5jgz8" podUID="59e32fbb-3cc2-40cc-b2f5-03b3c1d71870" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused"
Oct 31 01:54:28.969523 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:54:28.969478    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5a313b20-f6bb-446c-b23b-925c75a5df66-istio-token\") pod \"e2e-praxis-b-476ce2-openshift-default-6df8c7ff5c-wqzzs\" (UID: \"5a313b20-f6bb-446c-b23b-925c75a5df66\") " pod="openshift-ingress/e2e-praxis-b-476ce2-openshift-default-6df8c7ff5c-wqzzs"
Oct 31 01:54:28.969851 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:54:28.969819    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x95wq\" (UniqueName: \"kubernetes.io/projected/5a313b20-f6bb-446c-b23b-925c75a5df66-kube-api-access-x95wq\") pod \"e2e-praxis-b-476ce2-openshift-default-6df8c7ff5c-wqzzs\" (UID: \"5a313b20-f6bb-446c-b23b-925c75a5df66\") " pod="openshift-ingress/e2e-praxis-b-476ce2-openshift-default-6df8c7ff5c-wqzzs"
Oct 31 01:54:29.049417 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:54:29.048506    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:54:29.085225 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:54:29.085198    4964 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-4xmfp"
Oct 31 01:54:29.085667 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:54:29.085647    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-4xmfp"
Oct 31 01:54:29.088710 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:54:29.088683    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-4xmfp"
Oct 31 01:54:29.195346 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:54:29.194524    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-476ce2-openshift-default-6df8c7ff5c-wqzzs"
Oct 31 01:54:29.548430 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:54:29.547114    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:54:29.755981 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:54:29.749271    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-476ce2-openshift-default-6df8c7ff5c-wqzzs"]
Oct 31 01:54:29.758103 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:54:29.758000    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:54:29.759351 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:54:29.758669    4964 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a313b20_f6bb_446c_b23b_925c75a5df66.slice/crio-06ab6d1c1655cbdd3e17454fb4bde664f16e000348e935f8984d4e961271a5d2 WatchSource:0}: Error finding container 06ab6d1c1655cbdd3e17454fb4bde664f16e000348e935f8984d4e961271a5d2: Status 404 returned error can't find the container with id 06ab6d1c1655cbdd3e17454fb4bde664f16e000348e935f8984d4e961271a5d2
Oct 31 01:54:29.763241 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:54:29.763210    4964 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 01:54:29.763346 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:54:29.763283    4964 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 01:54:29.763346 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:54:29.763336    4964 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 01:54:29.867209 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:54:29.867178    4964 patch_prober.go:29] interesting pod/e2e-worker-w2-00e840-openshift-default-59b96fb879-psgmw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.168:15021/healthz/ready\": dial tcp 10.132.0.168:15021: connect: connection refused" start-of-body=
Oct 31 01:54:29.867378 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:54:29.867235    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-psgmw" podUID="a331bf16-e6db-4827-a796-b6a3cca34ead" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.168:15021/healthz/ready\": dial tcp 10.132.0.168:15021: connect: connection refused"
Oct 31 01:54:29.939830 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:54:29.939725    4964 patch_prober.go:29] interesting pod/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-5jgz8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused" start-of-body=
Oct 31 01:54:29.939830 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:54:29.939776    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-5jgz8" podUID="59e32fbb-3cc2-40cc-b2f5-03b3c1d71870" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused"
Oct 31 01:54:30.485896 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:54:30.485848    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-476ce2-openshift-default-6df8c7ff5c-wqzzs" event={"ID":"5a313b20-f6bb-446c-b23b-925c75a5df66","Type":"ContainerStarted","Data":"a3cbedca00622e1ebaa8aae6974a651e953cb06626288052f24eed3e79af01a0"}
Oct 31 01:54:30.486112 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:54:30.485931    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-476ce2-openshift-default-6df8c7ff5c-wqzzs" event={"ID":"5a313b20-f6bb-446c-b23b-925c75a5df66","Type":"ContainerStarted","Data":"06ab6d1c1655cbdd3e17454fb4bde664f16e000348e935f8984d4e961271a5d2"}
Oct 31 01:54:30.622364 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:54:30.621821    4964 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-b-476ce2-openshift-default-6df8c7ff5c-wqzzs" podStartSLOduration=2.6218026610000003 podStartE2EDuration="2.621802661s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:54:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:54:30.604716709 +0000 UTC m=+3039.081283089" watchObservedRunningTime="2026-10-31 01:54:30.621802661 +0000 UTC m=+3039.098369042"
Oct 31 01:54:30.867663 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:54:30.867582    4964 patch_prober.go:29] interesting pod/e2e-worker-w2-00e840-openshift-default-59b96fb879-psgmw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.168:15021/healthz/ready\": dial tcp 10.132.0.168:15021: connect: connection refused" start-of-body=
Oct 31 01:54:30.867663 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:54:30.867635    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-psgmw" podUID="a331bf16-e6db-4827-a796-b6a3cca34ead" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.168:15021/healthz/ready\": dial tcp 10.132.0.168:15021: connect: connection refused"
Oct 31 01:54:30.931254 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:54:30.931225    4964 patch_prober.go:29] interesting pod/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-5jgz8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused" start-of-body=
Oct 31 01:54:30.931465 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:54:30.931274    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-5jgz8" podUID="59e32fbb-3cc2-40cc-b2f5-03b3c1d71870" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused"
Oct 31 01:54:31.215514 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:54:31.215425    4964 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-476ce2-openshift-default-6df8c7ff5c-wqzzs"
Oct 31 01:54:31.224557 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:54:31.224529    4964 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-476ce2-openshift-default-6df8c7ff5c-wqzzs"
Oct 31 01:54:31.496348 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:54:31.496299    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-476ce2-openshift-default-6df8c7ff5c-wqzzs"
Oct 31 01:54:31.506589 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:54:31.506566    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-476ce2-openshift-default-6df8c7ff5c-wqzzs"
Oct 31 01:54:31.867076 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:54:31.866992    4964 patch_prober.go:29] interesting pod/e2e-worker-w2-00e840-openshift-default-59b96fb879-psgmw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.168:15021/healthz/ready\": dial tcp 10.132.0.168:15021: connect: connection refused" start-of-body=
Oct 31 01:54:31.867076 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:54:31.867057    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-psgmw" podUID="a331bf16-e6db-4827-a796-b6a3cca34ead" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.168:15021/healthz/ready\": dial tcp 10.132.0.168:15021: connect: connection refused"
Oct 31 01:54:32.035798 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:54:32.035764    4964 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-5jgz8"
Oct 31 01:54:32.036277 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:54:32.036181    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-5jgz8"
Oct 31 01:54:32.037569 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:54:32.037545    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-5jgz8"
Oct 31 01:54:32.868089 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:54:32.868051    4964 patch_prober.go:29] interesting pod/e2e-worker-w2-00e840-openshift-default-59b96fb879-psgmw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.168:15021/healthz/ready\": dial tcp 10.132.0.168:15021: connect: connection refused" start-of-body=
Oct 31 01:54:32.868521 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:54:32.868119    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-psgmw" podUID="a331bf16-e6db-4827-a796-b6a3cca34ead" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.168:15021/healthz/ready\": dial tcp 10.132.0.168:15021: connect: connection refused"
Oct 31 01:54:33.601616 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:54:33.601586    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:54:33.837078 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:54:33.837036    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:54:33.867000 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:54:33.866929    4964 patch_prober.go:29] interesting pod/e2e-worker-w2-00e840-openshift-default-59b96fb879-psgmw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.168:15021/healthz/ready\": dial tcp 10.132.0.168:15021: connect: connection refused" start-of-body=
Oct 31 01:54:33.867000 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:54:33.866976    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-psgmw" podUID="a331bf16-e6db-4827-a796-b6a3cca34ead" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.168:15021/healthz/ready\": dial tcp 10.132.0.168:15021: connect: connection refused"
Oct 31 01:54:34.264521 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:54:34.264486    4964 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w0-98b70c-openshift-default-7b4cfc957-9ntnk"]
Oct 31 01:54:34.269212 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:54:34.269192    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-98b70c-openshift-default-7b4cfc957-9ntnk"
Oct 31 01:54:34.354990 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:54:34.354953    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-98b70c-openshift-default-7b4cfc957-9ntnk"]
Oct 31 01:54:34.403447 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:54:34.403413    4964 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w0-98b70c-openshift-default-dockercfg-p82mf\""
Oct 31 01:54:34.429615 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:54:34.429230    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:54:34.436214 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:54:34.436180    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/55ee39ac-8129-4228-b0d7-a9b7841d3969-istio-podinfo\") pod \"e2e-shared-a-w0-98b70c-openshift-default-7b4cfc957-9ntnk\" (UID: \"55ee39ac-8129-4228-b0d7-a9b7841d3969\") " pod="openshift-ingress/e2e-shared-a-w0-98b70c-openshift-default-7b4cfc957-9ntnk"
Oct 31 01:54:34.436516 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:54:34.436494    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/55ee39ac-8129-4228-b0d7-a9b7841d3969-credential-socket\") pod \"e2e-shared-a-w0-98b70c-openshift-default-7b4cfc957-9ntnk\" (UID: \"55ee39ac-8129-4228-b0d7-a9b7841d3969\") " pod="openshift-ingress/e2e-shared-a-w0-98b70c-openshift-default-7b4cfc957-9ntnk"
Oct 31 01:54:34.436663 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:54:34.436646    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/55ee39ac-8129-4228-b0d7-a9b7841d3969-workload-certs\") pod \"e2e-shared-a-w0-98b70c-openshift-default-7b4cfc957-9ntnk\" (UID: \"55ee39ac-8129-4228-b0d7-a9b7841d3969\") " pod="openshift-ingress/e2e-shared-a-w0-98b70c-openshift-default-7b4cfc957-9ntnk"
Oct 31 01:54:34.436795 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:54:34.436779    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/55ee39ac-8129-4228-b0d7-a9b7841d3969-istio-envoy\") pod \"e2e-shared-a-w0-98b70c-openshift-default-7b4cfc957-9ntnk\" (UID: \"55ee39ac-8129-4228-b0d7-a9b7841d3969\") " pod="openshift-ingress/e2e-shared-a-w0-98b70c-openshift-default-7b4cfc957-9ntnk"
Oct 31 01:54:34.437033 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:54:34.437015    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/55ee39ac-8129-4228-b0d7-a9b7841d3969-istio-data\") pod \"e2e-shared-a-w0-98b70c-openshift-default-7b4cfc957-9ntnk\" (UID: \"55ee39ac-8129-4228-b0d7-a9b7841d3969\") " pod="openshift-ingress/e2e-shared-a-w0-98b70c-openshift-default-7b4cfc957-9ntnk"
Oct 31 01:54:34.437195 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:54:34.437179    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/55ee39ac-8129-4228-b0d7-a9b7841d3969-istio-token\") pod \"e2e-shared-a-w0-98b70c-openshift-default-7b4cfc957-9ntnk\" (UID: \"55ee39ac-8129-4228-b0d7-a9b7841d3969\") " pod="openshift-ingress/e2e-shared-a-w0-98b70c-openshift-default-7b4cfc957-9ntnk"
Oct 31 01:54:34.437408 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:54:34.437354    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w4rpm\" (UniqueName: \"kubernetes.io/projected/55ee39ac-8129-4228-b0d7-a9b7841d3969-kube-api-access-w4rpm\") pod \"e2e-shared-a-w0-98b70c-openshift-default-7b4cfc957-9ntnk\" (UID: \"55ee39ac-8129-4228-b0d7-a9b7841d3969\") " pod="openshift-ingress/e2e-shared-a-w0-98b70c-openshift-default-7b4cfc957-9ntnk"
Oct 31 01:54:34.437646 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:54:34.437626    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/55ee39ac-8129-4228-b0d7-a9b7841d3969-workload-socket\") pod \"e2e-shared-a-w0-98b70c-openshift-default-7b4cfc957-9ntnk\" (UID: \"55ee39ac-8129-4228-b0d7-a9b7841d3969\") " pod="openshift-ingress/e2e-shared-a-w0-98b70c-openshift-default-7b4cfc957-9ntnk"
Oct 31 01:54:34.437804 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:54:34.437787    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/55ee39ac-8129-4228-b0d7-a9b7841d3969-istiod-ca-cert\") pod \"e2e-shared-a-w0-98b70c-openshift-default-7b4cfc957-9ntnk\" (UID: \"55ee39ac-8129-4228-b0d7-a9b7841d3969\") " pod="openshift-ingress/e2e-shared-a-w0-98b70c-openshift-default-7b4cfc957-9ntnk"
Oct 31 01:54:34.538785 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:54:34.538682    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/55ee39ac-8129-4228-b0d7-a9b7841d3969-istio-data\") pod \"e2e-shared-a-w0-98b70c-openshift-default-7b4cfc957-9ntnk\" (UID: \"55ee39ac-8129-4228-b0d7-a9b7841d3969\") " pod="openshift-ingress/e2e-shared-a-w0-98b70c-openshift-default-7b4cfc957-9ntnk"
Oct 31 01:54:34.538785 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:54:34.538769    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/55ee39ac-8129-4228-b0d7-a9b7841d3969-istio-token\") pod \"e2e-shared-a-w0-98b70c-openshift-default-7b4cfc957-9ntnk\" (UID: \"55ee39ac-8129-4228-b0d7-a9b7841d3969\") " pod="openshift-ingress/e2e-shared-a-w0-98b70c-openshift-default-7b4cfc957-9ntnk"
Oct 31 01:54:34.539025 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:54:34.538799    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w4rpm\" (UniqueName: \"kubernetes.io/projected/55ee39ac-8129-4228-b0d7-a9b7841d3969-kube-api-access-w4rpm\") pod \"e2e-shared-a-w0-98b70c-openshift-default-7b4cfc957-9ntnk\" (UID: \"55ee39ac-8129-4228-b0d7-a9b7841d3969\") " pod="openshift-ingress/e2e-shared-a-w0-98b70c-openshift-default-7b4cfc957-9ntnk"
Oct 31 01:54:34.539025 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:54:34.538902    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/55ee39ac-8129-4228-b0d7-a9b7841d3969-workload-socket\") pod \"e2e-shared-a-w0-98b70c-openshift-default-7b4cfc957-9ntnk\" (UID: \"55ee39ac-8129-4228-b0d7-a9b7841d3969\") " pod="openshift-ingress/e2e-shared-a-w0-98b70c-openshift-default-7b4cfc957-9ntnk"
Oct 31 01:54:34.539025 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:54:34.538925    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/55ee39ac-8129-4228-b0d7-a9b7841d3969-istiod-ca-cert\") pod \"e2e-shared-a-w0-98b70c-openshift-default-7b4cfc957-9ntnk\" (UID: \"55ee39ac-8129-4228-b0d7-a9b7841d3969\") " pod="openshift-ingress/e2e-shared-a-w0-98b70c-openshift-default-7b4cfc957-9ntnk"
Oct 31 01:54:34.539025 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:54:34.538969    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/55ee39ac-8129-4228-b0d7-a9b7841d3969-istio-podinfo\") pod \"e2e-shared-a-w0-98b70c-openshift-default-7b4cfc957-9ntnk\" (UID: \"55ee39ac-8129-4228-b0d7-a9b7841d3969\") " pod="openshift-ingress/e2e-shared-a-w0-98b70c-openshift-default-7b4cfc957-9ntnk"
Oct 31 01:54:34.539025 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:54:34.539009    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/55ee39ac-8129-4228-b0d7-a9b7841d3969-credential-socket\") pod \"e2e-shared-a-w0-98b70c-openshift-default-7b4cfc957-9ntnk\" (UID: \"55ee39ac-8129-4228-b0d7-a9b7841d3969\") " pod="openshift-ingress/e2e-shared-a-w0-98b70c-openshift-default-7b4cfc957-9ntnk"
Oct 31 01:54:34.539280 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:54:34.539035    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/55ee39ac-8129-4228-b0d7-a9b7841d3969-workload-certs\") pod \"e2e-shared-a-w0-98b70c-openshift-default-7b4cfc957-9ntnk\" (UID: \"55ee39ac-8129-4228-b0d7-a9b7841d3969\") " pod="openshift-ingress/e2e-shared-a-w0-98b70c-openshift-default-7b4cfc957-9ntnk"
Oct 31 01:54:34.539280 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:54:34.539066    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/55ee39ac-8129-4228-b0d7-a9b7841d3969-istio-envoy\") pod \"e2e-shared-a-w0-98b70c-openshift-default-7b4cfc957-9ntnk\" (UID: \"55ee39ac-8129-4228-b0d7-a9b7841d3969\") " pod="openshift-ingress/e2e-shared-a-w0-98b70c-openshift-default-7b4cfc957-9ntnk"
Oct 31 01:54:34.539280 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:54:34.539109    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/55ee39ac-8129-4228-b0d7-a9b7841d3969-istio-data\") pod \"e2e-shared-a-w0-98b70c-openshift-default-7b4cfc957-9ntnk\" (UID: \"55ee39ac-8129-4228-b0d7-a9b7841d3969\") " pod="openshift-ingress/e2e-shared-a-w0-98b70c-openshift-default-7b4cfc957-9ntnk"
Oct 31 01:54:34.539515 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:54:34.539495    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/55ee39ac-8129-4228-b0d7-a9b7841d3969-credential-socket\") pod \"e2e-shared-a-w0-98b70c-openshift-default-7b4cfc957-9ntnk\" (UID: \"55ee39ac-8129-4228-b0d7-a9b7841d3969\") " pod="openshift-ingress/e2e-shared-a-w0-98b70c-openshift-default-7b4cfc957-9ntnk"
Oct 31 01:54:34.539691 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:54:34.539671    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/55ee39ac-8129-4228-b0d7-a9b7841d3969-workload-socket\") pod \"e2e-shared-a-w0-98b70c-openshift-default-7b4cfc957-9ntnk\" (UID: \"55ee39ac-8129-4228-b0d7-a9b7841d3969\") " pod="openshift-ingress/e2e-shared-a-w0-98b70c-openshift-default-7b4cfc957-9ntnk"
Oct 31 01:54:34.539765 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:54:34.539728    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/55ee39ac-8129-4228-b0d7-a9b7841d3969-workload-certs\") pod \"e2e-shared-a-w0-98b70c-openshift-default-7b4cfc957-9ntnk\" (UID: \"55ee39ac-8129-4228-b0d7-a9b7841d3969\") " pod="openshift-ingress/e2e-shared-a-w0-98b70c-openshift-default-7b4cfc957-9ntnk"
Oct 31 01:54:34.540276 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:54:34.540255    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/55ee39ac-8129-4228-b0d7-a9b7841d3969-istiod-ca-cert\") pod \"e2e-shared-a-w0-98b70c-openshift-default-7b4cfc957-9ntnk\" (UID: \"55ee39ac-8129-4228-b0d7-a9b7841d3969\") " pod="openshift-ingress/e2e-shared-a-w0-98b70c-openshift-default-7b4cfc957-9ntnk"
Oct 31 01:54:34.543470 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:54:34.543449    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/55ee39ac-8129-4228-b0d7-a9b7841d3969-istio-podinfo\") pod \"e2e-shared-a-w0-98b70c-openshift-default-7b4cfc957-9ntnk\" (UID: \"55ee39ac-8129-4228-b0d7-a9b7841d3969\") " pod="openshift-ingress/e2e-shared-a-w0-98b70c-openshift-default-7b4cfc957-9ntnk"
Oct 31 01:54:34.559700 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:54:34.559677    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/55ee39ac-8129-4228-b0d7-a9b7841d3969-istio-envoy\") pod \"e2e-shared-a-w0-98b70c-openshift-default-7b4cfc957-9ntnk\" (UID: \"55ee39ac-8129-4228-b0d7-a9b7841d3969\") " pod="openshift-ingress/e2e-shared-a-w0-98b70c-openshift-default-7b4cfc957-9ntnk"
Oct 31 01:54:34.604082 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:54:34.604019    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/55ee39ac-8129-4228-b0d7-a9b7841d3969-istio-token\") pod \"e2e-shared-a-w0-98b70c-openshift-default-7b4cfc957-9ntnk\" (UID: \"55ee39ac-8129-4228-b0d7-a9b7841d3969\") " pod="openshift-ingress/e2e-shared-a-w0-98b70c-openshift-default-7b4cfc957-9ntnk"
Oct 31 01:54:34.605937 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:54:34.605906    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w4rpm\" (UniqueName: \"kubernetes.io/projected/55ee39ac-8129-4228-b0d7-a9b7841d3969-kube-api-access-w4rpm\") pod \"e2e-shared-a-w0-98b70c-openshift-default-7b4cfc957-9ntnk\" (UID: \"55ee39ac-8129-4228-b0d7-a9b7841d3969\") " pod="openshift-ingress/e2e-shared-a-w0-98b70c-openshift-default-7b4cfc957-9ntnk"
Oct 31 01:54:34.635378 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:54:34.634569    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:54:34.805913 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:54:34.805801    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:54:34.883572 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:54:34.883539    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-98b70c-openshift-default-7b4cfc957-9ntnk"
Oct 31 01:54:34.922774 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:54:34.921607    4964 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-psgmw"
Oct 31 01:54:34.922774 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:54:34.922586    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-psgmw"
Oct 31 01:54:34.923858 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:54:34.923812    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-psgmw"
Oct 31 01:54:35.340430 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:54:35.340405    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-98b70c-openshift-default-7b4cfc957-9ntnk"]
Oct 31 01:54:35.341834 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:54:35.341803    4964 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod55ee39ac_8129_4228_b0d7_a9b7841d3969.slice/crio-485c294d6340ce038f813985ddde58bc9fa215199bfb32efc9eabcbf8f4489d8 WatchSource:0}: Error finding container 485c294d6340ce038f813985ddde58bc9fa215199bfb32efc9eabcbf8f4489d8: Status 404 returned error can't find the container with id 485c294d6340ce038f813985ddde58bc9fa215199bfb32efc9eabcbf8f4489d8
Oct 31 01:54:35.344480 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:54:35.344449    4964 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 01:54:35.344559 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:54:35.344514    4964 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 01:54:35.344559 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:54:35.344542    4964 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 01:54:35.543727 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:54:35.543698    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-98b70c-openshift-default-7b4cfc957-9ntnk" event={"ID":"55ee39ac-8129-4228-b0d7-a9b7841d3969","Type":"ContainerStarted","Data":"1395b33b38671b1bce5439e4857d67389b27b91047f8ad7debd534f9773edd3f"}
Oct 31 01:54:35.543889 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:54:35.543748    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-98b70c-openshift-default-7b4cfc957-9ntnk" event={"ID":"55ee39ac-8129-4228-b0d7-a9b7841d3969","Type":"ContainerStarted","Data":"485c294d6340ce038f813985ddde58bc9fa215199bfb32efc9eabcbf8f4489d8"}
Oct 31 01:54:35.884296 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:54:35.884266    4964 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w0-98b70c-openshift-default-7b4cfc957-9ntnk"
Oct 31 01:54:36.876720 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:54:36.876687    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:54:36.890011 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:54:36.889991    4964 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w0-98b70c-openshift-default-7b4cfc957-9ntnk"
Oct 31 01:54:37.051822 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:54:37.051778    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:54:37.058659 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:54:37.058618    4964 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w0-98b70c-openshift-default-7b4cfc957-9ntnk" podStartSLOduration=3.058606817 podStartE2EDuration="3.058606817s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:54:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:54:35.665490009 +0000 UTC m=+3044.142056393" watchObservedRunningTime="2026-10-31 01:54:37.058606817 +0000 UTC m=+3045.535173195"
Oct 31 01:54:37.557522 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:54:37.557489    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w0-98b70c-openshift-default-7b4cfc957-9ntnk"
Oct 31 01:54:37.558520 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:54:37.558501    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w0-98b70c-openshift-default-7b4cfc957-9ntnk"
Oct 31 01:54:38.588895 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:54:38.588852    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:54:38.832384 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:54:38.832352    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:54:40.055645 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:54:40.055611    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:54:41.237897 ip-10-0-133-197 kubenswrapper[4964]: E1031 01:54:41.237857    4964 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q56x5" containerName="extractor"
Oct 31 01:54:41.783269 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:54:41.783242    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:54:41.955788 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:54:41.955758    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:54:42.316965 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:54:42.316032    4964 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-98b70c-779f8fcd7f-6lb9k"]
Oct 31 01:54:42.323069 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:54:42.323040    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-98b70c-779f8fcd7f-6lb9k"
Oct 31 01:54:42.366938 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:54:42.366912    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-98b70c-779f8fcd7f-6lb9k"]
Oct 31 01:54:42.412383 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:54:42.412357    4964 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w0-98b70c\""
Oct 31 01:54:42.412544 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:54:42.412497    4964 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w0-98b70c-dockercfg-hq2dv\""
Oct 31 01:54:42.451090 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:54:42.451053    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/380f14cc-28b6-4557-8dd5-7a36f0bbb3be-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w0-98b70c-779f8fcd7f-6lb9k\" (UID: \"380f14cc-28b6-4557-8dd5-7a36f0bbb3be\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-98b70c-779f8fcd7f-6lb9k"
Oct 31 01:54:42.451231 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:54:42.451184    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nb2zr\" (UniqueName: \"kubernetes.io/projected/380f14cc-28b6-4557-8dd5-7a36f0bbb3be-kube-api-access-nb2zr\") pod \"payload-processing-e2e-shared-a-w0-98b70c-779f8fcd7f-6lb9k\" (UID: \"380f14cc-28b6-4557-8dd5-7a36f0bbb3be\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-98b70c-779f8fcd7f-6lb9k"
Oct 31 01:54:42.552921 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:54:42.552890    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/380f14cc-28b6-4557-8dd5-7a36f0bbb3be-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w0-98b70c-779f8fcd7f-6lb9k\" (UID: \"380f14cc-28b6-4557-8dd5-7a36f0bbb3be\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-98b70c-779f8fcd7f-6lb9k"
Oct 31 01:54:42.553043 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:54:42.552965    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nb2zr\" (UniqueName: \"kubernetes.io/projected/380f14cc-28b6-4557-8dd5-7a36f0bbb3be-kube-api-access-nb2zr\") pod \"payload-processing-e2e-shared-a-w0-98b70c-779f8fcd7f-6lb9k\" (UID: \"380f14cc-28b6-4557-8dd5-7a36f0bbb3be\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-98b70c-779f8fcd7f-6lb9k"
Oct 31 01:54:42.553886 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:54:42.553860    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/380f14cc-28b6-4557-8dd5-7a36f0bbb3be-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w0-98b70c-779f8fcd7f-6lb9k\" (UID: \"380f14cc-28b6-4557-8dd5-7a36f0bbb3be\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-98b70c-779f8fcd7f-6lb9k"
Oct 31 01:54:42.609929 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:54:42.609866    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nb2zr\" (UniqueName: \"kubernetes.io/projected/380f14cc-28b6-4557-8dd5-7a36f0bbb3be-kube-api-access-nb2zr\") pod \"payload-processing-e2e-shared-a-w0-98b70c-779f8fcd7f-6lb9k\" (UID: \"380f14cc-28b6-4557-8dd5-7a36f0bbb3be\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-98b70c-779f8fcd7f-6lb9k"
Oct 31 01:54:42.641518 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:54:42.641493    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-98b70c-779f8fcd7f-6lb9k"
Oct 31 01:54:43.019867 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:54:43.019844    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-98b70c-779f8fcd7f-6lb9k"]
Oct 31 01:54:43.021944 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:54:43.021915    4964 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod380f14cc_28b6_4557_8dd5_7a36f0bbb3be.slice/crio-6d7e314d1f3fe59bd0eb36216938b5b7e480caa82d32d733c33bad6b74c08e94 WatchSource:0}: Error finding container 6d7e314d1f3fe59bd0eb36216938b5b7e480caa82d32d733c33bad6b74c08e94: Status 404 returned error can't find the container with id 6d7e314d1f3fe59bd0eb36216938b5b7e480caa82d32d733c33bad6b74c08e94
Oct 31 01:54:43.604276 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:54:43.604219    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-98b70c-779f8fcd7f-6lb9k" event={"ID":"380f14cc-28b6-4557-8dd5-7a36f0bbb3be","Type":"ContainerStarted","Data":"8bd9b2f828f29767d01f336da42721f416e078887cf4e62980fe1d5522e7e7fe"}
Oct 31 01:54:43.604823 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:54:43.604295    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-98b70c-779f8fcd7f-6lb9k" event={"ID":"380f14cc-28b6-4557-8dd5-7a36f0bbb3be","Type":"ContainerStarted","Data":"6d7e314d1f3fe59bd0eb36216938b5b7e480caa82d32d733c33bad6b74c08e94"}
Oct 31 01:54:43.605709 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:54:43.605685    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-98b70c-779f8fcd7f-6lb9k"
Oct 31 01:54:43.651851 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:54:43.646895    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:54:43.729443 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:54:43.729176    4964 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-98b70c-779f8fcd7f-6lb9k" podStartSLOduration=1.7291570699999999 podStartE2EDuration="1.72915707s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:54:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:54:43.723051301 +0000 UTC m=+3052.199617682" watchObservedRunningTime="2026-10-31 01:54:43.72915707 +0000 UTC m=+3052.205723451"
Oct 31 01:54:44.526342 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:54:44.473403    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:54:44.998344 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:54:44.998195    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:54:45.735060 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:54:45.735027    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:54:50.742289 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:54:50.742249    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:54:52.444521 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:54:52.444487    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:54:54.753187 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:54:54.753148    4964 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-c0daf4-7f578c867c-84zhj"]
Oct 31 01:54:54.753827 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:54:54.753399    4964 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-c0daf4-7f578c867c-84zhj" podUID="15340841-26cc-4a0e-adbe-3bbba9e6bb4f" containerName="maas-api" containerID="cri-o://27a8ce86be642c14cf81ad86dce60ebeb0d7a5a5b8c97c57d1792759ed9e5ac4" gracePeriod=30
Oct 31 01:54:54.888142 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:54:54.888101    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:54:55.064770 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:54:55.064738    4964 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-c0daf4-7f578c867c-84zhj"
Oct 31 01:54:55.253735 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:54:55.253686    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/15340841-26cc-4a0e-adbe-3bbba9e6bb4f-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/15340841-26cc-4a0e-adbe-3bbba9e6bb4f-maas-api-tls\") pod \"15340841-26cc-4a0e-adbe-3bbba9e6bb4f\" (UID: \"15340841-26cc-4a0e-adbe-3bbba9e6bb4f\") "
Oct 31 01:54:55.253998 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:54:55.253856    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/15340841-26cc-4a0e-adbe-3bbba9e6bb4f-metrics-certs\" (UniqueName: \"kubernetes.io/secret/15340841-26cc-4a0e-adbe-3bbba9e6bb4f-metrics-certs\") pod \"15340841-26cc-4a0e-adbe-3bbba9e6bb4f\" (UID: \"15340841-26cc-4a0e-adbe-3bbba9e6bb4f\") "
Oct 31 01:54:55.253998 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:54:55.253937    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/15340841-26cc-4a0e-adbe-3bbba9e6bb4f-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/15340841-26cc-4a0e-adbe-3bbba9e6bb4f-openshift-service-ca\") pod \"15340841-26cc-4a0e-adbe-3bbba9e6bb4f\" (UID: \"15340841-26cc-4a0e-adbe-3bbba9e6bb4f\") "
Oct 31 01:54:55.253998 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:54:55.253977    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/15340841-26cc-4a0e-adbe-3bbba9e6bb4f-kube-api-access-qdlrk\" (UniqueName: \"kubernetes.io/projected/15340841-26cc-4a0e-adbe-3bbba9e6bb4f-kube-api-access-qdlrk\") pod \"15340841-26cc-4a0e-adbe-3bbba9e6bb4f\" (UID: \"15340841-26cc-4a0e-adbe-3bbba9e6bb4f\") "
Oct 31 01:54:55.254651 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:54:55.254377    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/15340841-26cc-4a0e-adbe-3bbba9e6bb4f-openshift-service-ca" pod "15340841-26cc-4a0e-adbe-3bbba9e6bb4f" (UID: "15340841-26cc-4a0e-adbe-3bbba9e6bb4f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:54:55.254651 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:54:55.254602    4964 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/15340841-26cc-4a0e-adbe-3bbba9e6bb4f-openshift-service-ca\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:54:55.256060 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:54:55.256038    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/15340841-26cc-4a0e-adbe-3bbba9e6bb4f-maas-api-tls" pod "15340841-26cc-4a0e-adbe-3bbba9e6bb4f" (UID: "15340841-26cc-4a0e-adbe-3bbba9e6bb4f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:54:55.256243 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:54:55.256223    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/15340841-26cc-4a0e-adbe-3bbba9e6bb4f-kube-api-access-qdlrk" pod "15340841-26cc-4a0e-adbe-3bbba9e6bb4f" (UID: "15340841-26cc-4a0e-adbe-3bbba9e6bb4f"). InnerVolumeSpecName "kube-api-access-qdlrk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:54:55.256302 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:54:55.256273    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/15340841-26cc-4a0e-adbe-3bbba9e6bb4f-metrics-certs" pod "15340841-26cc-4a0e-adbe-3bbba9e6bb4f" (UID: "15340841-26cc-4a0e-adbe-3bbba9e6bb4f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:54:55.356881 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:54:55.356793    4964 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qdlrk\" (UniqueName: \"kubernetes.io/projected/15340841-26cc-4a0e-adbe-3bbba9e6bb4f-kube-api-access-qdlrk\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:54:55.356881 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:54:55.356835    4964 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/15340841-26cc-4a0e-adbe-3bbba9e6bb4f-maas-api-tls\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:54:55.356881 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:54:55.356849    4964 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/15340841-26cc-4a0e-adbe-3bbba9e6bb4f-metrics-certs\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:54:55.814057 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:54:55.814022    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-98b70c-779f8fcd7f-6lb9k"
Oct 31 01:54:55.856576 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:54:55.856542    4964 generic.go:356] "Generic (PLEG): container finished" podID="15340841-26cc-4a0e-adbe-3bbba9e6bb4f" containerID="27a8ce86be642c14cf81ad86dce60ebeb0d7a5a5b8c97c57d1792759ed9e5ac4" exitCode=0
Oct 31 01:54:55.856786 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:54:55.856607    4964 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-c0daf4-7f578c867c-84zhj"
Oct 31 01:54:55.856786 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:54:55.856623    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-c0daf4-7f578c867c-84zhj" event={"ID":"15340841-26cc-4a0e-adbe-3bbba9e6bb4f","Type":"ContainerDied","Data":"27a8ce86be642c14cf81ad86dce60ebeb0d7a5a5b8c97c57d1792759ed9e5ac4"}
Oct 31 01:54:55.856786 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:54:55.856677    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-c0daf4-7f578c867c-84zhj" event={"ID":"15340841-26cc-4a0e-adbe-3bbba9e6bb4f","Type":"ContainerDied","Data":"c6c39d711b723bcd62d2f8180a38b3ebc2537a19f372b87881791b9d58bec451"}
Oct 31 01:54:55.856786 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:54:55.856717    4964 scope.go:122] "RemoveContainer" containerID="27a8ce86be642c14cf81ad86dce60ebeb0d7a5a5b8c97c57d1792759ed9e5ac4"
Oct 31 01:54:55.868169 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:54:55.868146    4964 scope.go:122] "RemoveContainer" containerID="27a8ce86be642c14cf81ad86dce60ebeb0d7a5a5b8c97c57d1792759ed9e5ac4"
Oct 31 01:54:55.868535 ip-10-0-133-197 kubenswrapper[4964]: E1031 01:54:55.868517    4964 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"27a8ce86be642c14cf81ad86dce60ebeb0d7a5a5b8c97c57d1792759ed9e5ac4\": container with ID starting with 27a8ce86be642c14cf81ad86dce60ebeb0d7a5a5b8c97c57d1792759ed9e5ac4 not found: ID does not exist" containerID="27a8ce86be642c14cf81ad86dce60ebeb0d7a5a5b8c97c57d1792759ed9e5ac4"
Oct 31 01:54:55.868593 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:54:55.868544    4964 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"27a8ce86be642c14cf81ad86dce60ebeb0d7a5a5b8c97c57d1792759ed9e5ac4"} err="failed to get container status \"27a8ce86be642c14cf81ad86dce60ebeb0d7a5a5b8c97c57d1792759ed9e5ac4\": rpc error: code = NotFound desc = could not find container \"27a8ce86be642c14cf81ad86dce60ebeb0d7a5a5b8c97c57d1792759ed9e5ac4\": container with ID starting with 27a8ce86be642c14cf81ad86dce60ebeb0d7a5a5b8c97c57d1792759ed9e5ac4 not found: ID does not exist"
Oct 31 01:54:56.043396 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:54:56.043356    4964 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-c0daf4-7f578c867c-84zhj"]
Oct 31 01:54:56.046884 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:54:56.046847    4964 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-c0daf4-7f578c867c-84zhj"]
Oct 31 01:54:56.491517 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:54:56.491479    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:54:56.770836 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:54:56.770749    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:54:56.948588 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:54:56.948559    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:54:57.471731 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:54:57.471700    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:54:57.713142 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:54:57.713114    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:54:57.778648 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:54:57.778554    4964 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="15340841-26cc-4a0e-adbe-3bbba9e6bb4f" path="/var/lib/kubelet/pods/15340841-26cc-4a0e-adbe-3bbba9e6bb4f/volumes"
Oct 31 01:55:00.719490 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:55:00.719449    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:55:00.939918 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:55:00.939867    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:55:01.265345 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:55:01.262239    4964 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-051514-546778dd4c-vbm49"]
Oct 31 01:55:01.277302 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:55:01.274287    4964 cpu_manager.go:414] "containerMap: removing container" podUID="15340841-26cc-4a0e-adbe-3bbba9e6bb4f" containerName="maas-api"
Oct 31 01:55:01.277302 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:55:01.274334    4964 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="15340841-26cc-4a0e-adbe-3bbba9e6bb4f" containerName="maas-api"
Oct 31 01:55:01.277302 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:55:01.274540    4964 memory_manager.go:365] "RemoveStaleState removing state" podUID="15340841-26cc-4a0e-adbe-3bbba9e6bb4f" containerName="maas-api"
Oct 31 01:55:01.285659 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:55:01.285627    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-051514-546778dd4c-vbm49"
Oct 31 01:55:01.337404 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:55:01.337297    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-051514-546778dd4c-vbm49"]
Oct 31 01:55:01.338564 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:55:01.338534    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2vzgm\" (UniqueName: \"kubernetes.io/projected/d2790587-1361-49b0-a964-2e7fa86effad-kube-api-access-2vzgm\") pod \"payload-processing-e2e-shared-a-w3-051514-546778dd4c-vbm49\" (UID: \"d2790587-1361-49b0-a964-2e7fa86effad\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-051514-546778dd4c-vbm49"
Oct 31 01:55:01.338717 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:55:01.338656    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d2790587-1361-49b0-a964-2e7fa86effad-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-051514-546778dd4c-vbm49\" (UID: \"d2790587-1361-49b0-a964-2e7fa86effad\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-051514-546778dd4c-vbm49"
Oct 31 01:55:01.358970 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:55:01.358930    4964 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w3-051514\""
Oct 31 01:55:01.358970 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:55:01.358930    4964 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w3-051514-dockercfg-g6lhw\""
Oct 31 01:55:01.414810 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:55:01.414775    4964 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-051514-bd7d495b7-pkp25"]
Oct 31 01:55:01.419067 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:55:01.419046    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-051514-bd7d495b7-pkp25"
Oct 31 01:55:01.440461 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:55:01.440425    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d2790587-1361-49b0-a964-2e7fa86effad-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-051514-546778dd4c-vbm49\" (UID: \"d2790587-1361-49b0-a964-2e7fa86effad\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-051514-546778dd4c-vbm49"
Oct 31 01:55:01.440606 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:55:01.440575    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2vzgm\" (UniqueName: \"kubernetes.io/projected/d2790587-1361-49b0-a964-2e7fa86effad-kube-api-access-2vzgm\") pod \"payload-processing-e2e-shared-a-w3-051514-546778dd4c-vbm49\" (UID: \"d2790587-1361-49b0-a964-2e7fa86effad\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-051514-546778dd4c-vbm49"
Oct 31 01:55:01.441038 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:55:01.441019    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d2790587-1361-49b0-a964-2e7fa86effad-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-051514-546778dd4c-vbm49\" (UID: \"d2790587-1361-49b0-a964-2e7fa86effad\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-051514-546778dd4c-vbm49"
Oct 31 01:55:01.477844 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:55:01.477808    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-051514-bd7d495b7-pkp25"]
Oct 31 01:55:01.497001 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:55:01.496963    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2vzgm\" (UniqueName: \"kubernetes.io/projected/d2790587-1361-49b0-a964-2e7fa86effad-kube-api-access-2vzgm\") pod \"payload-processing-e2e-shared-a-w3-051514-546778dd4c-vbm49\" (UID: \"d2790587-1361-49b0-a964-2e7fa86effad\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-051514-546778dd4c-vbm49"
Oct 31 01:55:01.541674 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:55:01.541592    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cz9gh\" (UniqueName: \"kubernetes.io/projected/64532a2f-bc54-4250-a617-95865b1d7e26-kube-api-access-cz9gh\") pod \"payload-pre-processing-e2e-shared-a-w3-051514-bd7d495b7-pkp25\" (UID: \"64532a2f-bc54-4250-a617-95865b1d7e26\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-051514-bd7d495b7-pkp25"
Oct 31 01:55:01.541813 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:55:01.541755    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/64532a2f-bc54-4250-a617-95865b1d7e26-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-051514-bd7d495b7-pkp25\" (UID: \"64532a2f-bc54-4250-a617-95865b1d7e26\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-051514-bd7d495b7-pkp25"
Oct 31 01:55:01.613928 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:55:01.613894    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-051514-546778dd4c-vbm49"
Oct 31 01:55:01.643585 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:55:01.643540    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/64532a2f-bc54-4250-a617-95865b1d7e26-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-051514-bd7d495b7-pkp25\" (UID: \"64532a2f-bc54-4250-a617-95865b1d7e26\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-051514-bd7d495b7-pkp25"
Oct 31 01:55:01.643747 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:55:01.643679    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cz9gh\" (UniqueName: \"kubernetes.io/projected/64532a2f-bc54-4250-a617-95865b1d7e26-kube-api-access-cz9gh\") pod \"payload-pre-processing-e2e-shared-a-w3-051514-bd7d495b7-pkp25\" (UID: \"64532a2f-bc54-4250-a617-95865b1d7e26\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-051514-bd7d495b7-pkp25"
Oct 31 01:55:01.644457 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:55:01.644429    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/64532a2f-bc54-4250-a617-95865b1d7e26-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-051514-bd7d495b7-pkp25\" (UID: \"64532a2f-bc54-4250-a617-95865b1d7e26\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-051514-bd7d495b7-pkp25"
Oct 31 01:55:01.701374 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:55:01.701258    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cz9gh\" (UniqueName: \"kubernetes.io/projected/64532a2f-bc54-4250-a617-95865b1d7e26-kube-api-access-cz9gh\") pod \"payload-pre-processing-e2e-shared-a-w3-051514-bd7d495b7-pkp25\" (UID: \"64532a2f-bc54-4250-a617-95865b1d7e26\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-051514-bd7d495b7-pkp25"
Oct 31 01:55:01.732756 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:55:01.732726    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-051514-bd7d495b7-pkp25"
Oct 31 01:55:01.995408 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:55:01.995382    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-051514-546778dd4c-vbm49"]
Oct 31 01:55:01.996020 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:55:01.995997    4964 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd2790587_1361_49b0_a964_2e7fa86effad.slice/crio-2078284688fc53cc4f6f42ab006f411e317e0b9b9a232f520fc711d180578957 WatchSource:0}: Error finding container 2078284688fc53cc4f6f42ab006f411e317e0b9b9a232f520fc711d180578957: Status 404 returned error can't find the container with id 2078284688fc53cc4f6f42ab006f411e317e0b9b9a232f520fc711d180578957
Oct 31 01:55:02.127888 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:55:02.127860    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-051514-bd7d495b7-pkp25"]
Oct 31 01:55:02.129839 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:55:02.129815    4964 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod64532a2f_bc54_4250_a617_95865b1d7e26.slice/crio-408c6920fed8d5a343b7b119c192ad7bf114f2e0db0f651cc0ff734dcdfc4b03 WatchSource:0}: Error finding container 408c6920fed8d5a343b7b119c192ad7bf114f2e0db0f651cc0ff734dcdfc4b03: Status 404 returned error can't find the container with id 408c6920fed8d5a343b7b119c192ad7bf114f2e0db0f651cc0ff734dcdfc4b03
Oct 31 01:55:02.153979 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:55:02.153942    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:55:02.910516 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:55:02.910476    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-051514-bd7d495b7-pkp25" event={"ID":"64532a2f-bc54-4250-a617-95865b1d7e26","Type":"ContainerStarted","Data":"b3d139e9aa9743d9393155d2f71bc9cce78e1912dfb6c132af9fc7eb37f995e8"}
Oct 31 01:55:02.910979 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:55:02.910533    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-051514-bd7d495b7-pkp25" event={"ID":"64532a2f-bc54-4250-a617-95865b1d7e26","Type":"ContainerStarted","Data":"408c6920fed8d5a343b7b119c192ad7bf114f2e0db0f651cc0ff734dcdfc4b03"}
Oct 31 01:55:02.910979 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:55:02.910571    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-051514-bd7d495b7-pkp25"
Oct 31 01:55:02.912144 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:55:02.912119    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-051514-546778dd4c-vbm49" event={"ID":"d2790587-1361-49b0-a964-2e7fa86effad","Type":"ContainerStarted","Data":"a611d7f94cf8ff2f2c79886fa950e3434e575d851c56efde595a05cd71a74577"}
Oct 31 01:55:02.912308 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:55:02.912153    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-051514-546778dd4c-vbm49" event={"ID":"d2790587-1361-49b0-a964-2e7fa86effad","Type":"ContainerStarted","Data":"2078284688fc53cc4f6f42ab006f411e317e0b9b9a232f520fc711d180578957"}
Oct 31 01:55:02.912308 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:55:02.912182    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-051514-546778dd4c-vbm49"
Oct 31 01:55:03.025926 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:55:03.025861    4964 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-051514-bd7d495b7-pkp25" podStartSLOduration=2.025838916 podStartE2EDuration="2.025838916s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:55:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:55:03.023593664 +0000 UTC m=+3071.500160054" watchObservedRunningTime="2026-10-31 01:55:03.025838916 +0000 UTC m=+3071.502405297"
Oct 31 01:55:03.142295 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:55:03.142231    4964 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-051514-546778dd4c-vbm49" podStartSLOduration=2.142210327 podStartE2EDuration="2.142210327s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:55:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:55:03.137232814 +0000 UTC m=+3071.613799194" watchObservedRunningTime="2026-10-31 01:55:03.142210327 +0000 UTC m=+3071.618776708"
Oct 31 01:55:03.314426 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:55:03.314385    4964 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-a-c0daf4-openshift-default-84d68dc6f-g2858"]
Oct 31 01:55:03.319687 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:55:03.319664    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-c0daf4-openshift-default-84d68dc6f-g2858"
Oct 31 01:55:03.365761 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:55:03.365733    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3d84fdf4-9d55-49fc-9784-0b008d97a20e-istiod-ca-cert\") pod \"e2e-praxis-a-c0daf4-openshift-default-84d68dc6f-g2858\" (UID: \"3d84fdf4-9d55-49fc-9784-0b008d97a20e\") " pod="openshift-ingress/e2e-praxis-a-c0daf4-openshift-default-84d68dc6f-g2858"
Oct 31 01:55:03.365922 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:55:03.365774    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d84fdf4-9d55-49fc-9784-0b008d97a20e-credential-socket\") pod \"e2e-praxis-a-c0daf4-openshift-default-84d68dc6f-g2858\" (UID: \"3d84fdf4-9d55-49fc-9784-0b008d97a20e\") " pod="openshift-ingress/e2e-praxis-a-c0daf4-openshift-default-84d68dc6f-g2858"
Oct 31 01:55:03.365922 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:55:03.365891    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3d84fdf4-9d55-49fc-9784-0b008d97a20e-istio-podinfo\") pod \"e2e-praxis-a-c0daf4-openshift-default-84d68dc6f-g2858\" (UID: \"3d84fdf4-9d55-49fc-9784-0b008d97a20e\") " pod="openshift-ingress/e2e-praxis-a-c0daf4-openshift-default-84d68dc6f-g2858"
Oct 31 01:55:03.366003 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:55:03.365964    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d84fdf4-9d55-49fc-9784-0b008d97a20e-workload-socket\") pod \"e2e-praxis-a-c0daf4-openshift-default-84d68dc6f-g2858\" (UID: \"3d84fdf4-9d55-49fc-9784-0b008d97a20e\") " pod="openshift-ingress/e2e-praxis-a-c0daf4-openshift-default-84d68dc6f-g2858"
Oct 31 01:55:03.366003 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:55:03.365984    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3d84fdf4-9d55-49fc-9784-0b008d97a20e-istio-token\") pod \"e2e-praxis-a-c0daf4-openshift-default-84d68dc6f-g2858\" (UID: \"3d84fdf4-9d55-49fc-9784-0b008d97a20e\") " pod="openshift-ingress/e2e-praxis-a-c0daf4-openshift-default-84d68dc6f-g2858"
Oct 31 01:55:03.366070 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:55:03.366002    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d84fdf4-9d55-49fc-9784-0b008d97a20e-workload-certs\") pod \"e2e-praxis-a-c0daf4-openshift-default-84d68dc6f-g2858\" (UID: \"3d84fdf4-9d55-49fc-9784-0b008d97a20e\") " pod="openshift-ingress/e2e-praxis-a-c0daf4-openshift-default-84d68dc6f-g2858"
Oct 31 01:55:03.366172 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:55:03.366152    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3d84fdf4-9d55-49fc-9784-0b008d97a20e-istio-data\") pod \"e2e-praxis-a-c0daf4-openshift-default-84d68dc6f-g2858\" (UID: \"3d84fdf4-9d55-49fc-9784-0b008d97a20e\") " pod="openshift-ingress/e2e-praxis-a-c0daf4-openshift-default-84d68dc6f-g2858"
Oct 31 01:55:03.366211 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:55:03.366182    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m8zkn\" (UniqueName: \"kubernetes.io/projected/3d84fdf4-9d55-49fc-9784-0b008d97a20e-kube-api-access-m8zkn\") pod \"e2e-praxis-a-c0daf4-openshift-default-84d68dc6f-g2858\" (UID: \"3d84fdf4-9d55-49fc-9784-0b008d97a20e\") " pod="openshift-ingress/e2e-praxis-a-c0daf4-openshift-default-84d68dc6f-g2858"
Oct 31 01:55:03.366243 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:55:03.366232    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3d84fdf4-9d55-49fc-9784-0b008d97a20e-istio-envoy\") pod \"e2e-praxis-a-c0daf4-openshift-default-84d68dc6f-g2858\" (UID: \"3d84fdf4-9d55-49fc-9784-0b008d97a20e\") " pod="openshift-ingress/e2e-praxis-a-c0daf4-openshift-default-84d68dc6f-g2858"
Oct 31 01:55:03.387338 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:55:03.371810    4964 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-a-c0daf4-openshift-default-dockercfg-r975r\""
Oct 31 01:55:03.405055 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:55:03.405019    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-c0daf4-openshift-default-84d68dc6f-g2858"]
Oct 31 01:55:03.467409 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:55:03.467364    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3d84fdf4-9d55-49fc-9784-0b008d97a20e-istio-envoy\") pod \"e2e-praxis-a-c0daf4-openshift-default-84d68dc6f-g2858\" (UID: \"3d84fdf4-9d55-49fc-9784-0b008d97a20e\") " pod="openshift-ingress/e2e-praxis-a-c0daf4-openshift-default-84d68dc6f-g2858"
Oct 31 01:55:03.467620 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:55:03.467483    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3d84fdf4-9d55-49fc-9784-0b008d97a20e-istiod-ca-cert\") pod \"e2e-praxis-a-c0daf4-openshift-default-84d68dc6f-g2858\" (UID: \"3d84fdf4-9d55-49fc-9784-0b008d97a20e\") " pod="openshift-ingress/e2e-praxis-a-c0daf4-openshift-default-84d68dc6f-g2858"
Oct 31 01:55:03.467620 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:55:03.467524    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d84fdf4-9d55-49fc-9784-0b008d97a20e-credential-socket\") pod \"e2e-praxis-a-c0daf4-openshift-default-84d68dc6f-g2858\" (UID: \"3d84fdf4-9d55-49fc-9784-0b008d97a20e\") " pod="openshift-ingress/e2e-praxis-a-c0daf4-openshift-default-84d68dc6f-g2858"
Oct 31 01:55:03.467620 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:55:03.467588    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3d84fdf4-9d55-49fc-9784-0b008d97a20e-istio-podinfo\") pod \"e2e-praxis-a-c0daf4-openshift-default-84d68dc6f-g2858\" (UID: \"3d84fdf4-9d55-49fc-9784-0b008d97a20e\") " pod="openshift-ingress/e2e-praxis-a-c0daf4-openshift-default-84d68dc6f-g2858"
Oct 31 01:55:03.467783 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:55:03.467635    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d84fdf4-9d55-49fc-9784-0b008d97a20e-workload-socket\") pod \"e2e-praxis-a-c0daf4-openshift-default-84d68dc6f-g2858\" (UID: \"3d84fdf4-9d55-49fc-9784-0b008d97a20e\") " pod="openshift-ingress/e2e-praxis-a-c0daf4-openshift-default-84d68dc6f-g2858"
Oct 31 01:55:03.467783 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:55:03.467677    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3d84fdf4-9d55-49fc-9784-0b008d97a20e-istio-token\") pod \"e2e-praxis-a-c0daf4-openshift-default-84d68dc6f-g2858\" (UID: \"3d84fdf4-9d55-49fc-9784-0b008d97a20e\") " pod="openshift-ingress/e2e-praxis-a-c0daf4-openshift-default-84d68dc6f-g2858"
Oct 31 01:55:03.467783 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:55:03.467710    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d84fdf4-9d55-49fc-9784-0b008d97a20e-workload-certs\") pod \"e2e-praxis-a-c0daf4-openshift-default-84d68dc6f-g2858\" (UID: \"3d84fdf4-9d55-49fc-9784-0b008d97a20e\") " pod="openshift-ingress/e2e-praxis-a-c0daf4-openshift-default-84d68dc6f-g2858"
Oct 31 01:55:03.467945 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:55:03.467817    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3d84fdf4-9d55-49fc-9784-0b008d97a20e-istio-data\") pod \"e2e-praxis-a-c0daf4-openshift-default-84d68dc6f-g2858\" (UID: \"3d84fdf4-9d55-49fc-9784-0b008d97a20e\") " pod="openshift-ingress/e2e-praxis-a-c0daf4-openshift-default-84d68dc6f-g2858"
Oct 31 01:55:03.467945 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:55:03.467847    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m8zkn\" (UniqueName: \"kubernetes.io/projected/3d84fdf4-9d55-49fc-9784-0b008d97a20e-kube-api-access-m8zkn\") pod \"e2e-praxis-a-c0daf4-openshift-default-84d68dc6f-g2858\" (UID: \"3d84fdf4-9d55-49fc-9784-0b008d97a20e\") " pod="openshift-ingress/e2e-praxis-a-c0daf4-openshift-default-84d68dc6f-g2858"
Oct 31 01:55:03.468043 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:55:03.467969    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d84fdf4-9d55-49fc-9784-0b008d97a20e-credential-socket\") pod \"e2e-praxis-a-c0daf4-openshift-default-84d68dc6f-g2858\" (UID: \"3d84fdf4-9d55-49fc-9784-0b008d97a20e\") " pod="openshift-ingress/e2e-praxis-a-c0daf4-openshift-default-84d68dc6f-g2858"
Oct 31 01:55:03.468230 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:55:03.468171    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d84fdf4-9d55-49fc-9784-0b008d97a20e-workload-socket\") pod \"e2e-praxis-a-c0daf4-openshift-default-84d68dc6f-g2858\" (UID: \"3d84fdf4-9d55-49fc-9784-0b008d97a20e\") " pod="openshift-ingress/e2e-praxis-a-c0daf4-openshift-default-84d68dc6f-g2858"
Oct 31 01:55:03.468368 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:55:03.468237    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3d84fdf4-9d55-49fc-9784-0b008d97a20e-istio-data\") pod \"e2e-praxis-a-c0daf4-openshift-default-84d68dc6f-g2858\" (UID: \"3d84fdf4-9d55-49fc-9784-0b008d97a20e\") " pod="openshift-ingress/e2e-praxis-a-c0daf4-openshift-default-84d68dc6f-g2858"
Oct 31 01:55:03.468368 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:55:03.468270    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d84fdf4-9d55-49fc-9784-0b008d97a20e-workload-certs\") pod \"e2e-praxis-a-c0daf4-openshift-default-84d68dc6f-g2858\" (UID: \"3d84fdf4-9d55-49fc-9784-0b008d97a20e\") " pod="openshift-ingress/e2e-praxis-a-c0daf4-openshift-default-84d68dc6f-g2858"
Oct 31 01:55:03.468368 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:55:03.468309    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3d84fdf4-9d55-49fc-9784-0b008d97a20e-istiod-ca-cert\") pod \"e2e-praxis-a-c0daf4-openshift-default-84d68dc6f-g2858\" (UID: \"3d84fdf4-9d55-49fc-9784-0b008d97a20e\") " pod="openshift-ingress/e2e-praxis-a-c0daf4-openshift-default-84d68dc6f-g2858"
Oct 31 01:55:03.469937 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:55:03.469914    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3d84fdf4-9d55-49fc-9784-0b008d97a20e-istio-envoy\") pod \"e2e-praxis-a-c0daf4-openshift-default-84d68dc6f-g2858\" (UID: \"3d84fdf4-9d55-49fc-9784-0b008d97a20e\") " pod="openshift-ingress/e2e-praxis-a-c0daf4-openshift-default-84d68dc6f-g2858"
Oct 31 01:55:03.470658 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:55:03.470635    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3d84fdf4-9d55-49fc-9784-0b008d97a20e-istio-podinfo\") pod \"e2e-praxis-a-c0daf4-openshift-default-84d68dc6f-g2858\" (UID: \"3d84fdf4-9d55-49fc-9784-0b008d97a20e\") " pod="openshift-ingress/e2e-praxis-a-c0daf4-openshift-default-84d68dc6f-g2858"
Oct 31 01:55:03.524460 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:55:03.524429    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3d84fdf4-9d55-49fc-9784-0b008d97a20e-istio-token\") pod \"e2e-praxis-a-c0daf4-openshift-default-84d68dc6f-g2858\" (UID: \"3d84fdf4-9d55-49fc-9784-0b008d97a20e\") " pod="openshift-ingress/e2e-praxis-a-c0daf4-openshift-default-84d68dc6f-g2858"
Oct 31 01:55:03.524648 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:55:03.524577    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m8zkn\" (UniqueName: \"kubernetes.io/projected/3d84fdf4-9d55-49fc-9784-0b008d97a20e-kube-api-access-m8zkn\") pod \"e2e-praxis-a-c0daf4-openshift-default-84d68dc6f-g2858\" (UID: \"3d84fdf4-9d55-49fc-9784-0b008d97a20e\") " pod="openshift-ingress/e2e-praxis-a-c0daf4-openshift-default-84d68dc6f-g2858"
Oct 31 01:55:03.636853 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:55:03.636754    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-c0daf4-openshift-default-84d68dc6f-g2858"
Oct 31 01:55:04.076032 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:55:04.076005    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-c0daf4-openshift-default-84d68dc6f-g2858"]
Oct 31 01:55:04.077427 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:55:04.077304    4964 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d84fdf4_9d55_49fc_9784_0b008d97a20e.slice/crio-6f4a6c50bc9ba0bfc73199b14c263b3da1f50df141359d1c82111a279f5f9b0d WatchSource:0}: Error finding container 6f4a6c50bc9ba0bfc73199b14c263b3da1f50df141359d1c82111a279f5f9b0d: Status 404 returned error can't find the container with id 6f4a6c50bc9ba0bfc73199b14c263b3da1f50df141359d1c82111a279f5f9b0d
Oct 31 01:55:04.080253 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:55:04.080223    4964 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 01:55:04.080364 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:55:04.080283    4964 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 01:55:04.080406 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:55:04.080368    4964 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 01:55:04.195545 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:55:04.195509    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:55:04.374577 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:55:04.374438    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:55:04.935478 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:55:04.935433    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-c0daf4-openshift-default-84d68dc6f-g2858" event={"ID":"3d84fdf4-9d55-49fc-9784-0b008d97a20e","Type":"ContainerStarted","Data":"c8494344c3e8f9dea2ad9a895300a3f549b570e62facf3b6bb2cd85d58080748"}
Oct 31 01:55:04.935654 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:55:04.935512    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-c0daf4-openshift-default-84d68dc6f-g2858" event={"ID":"3d84fdf4-9d55-49fc-9784-0b008d97a20e","Type":"ContainerStarted","Data":"6f4a6c50bc9ba0bfc73199b14c263b3da1f50df141359d1c82111a279f5f9b0d"}
Oct 31 01:55:05.274193 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:55:05.274141    4964 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-a-c0daf4-openshift-default-84d68dc6f-g2858" podStartSLOduration=2.274127438 podStartE2EDuration="2.274127438s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:55:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:55:05.108586131 +0000 UTC m=+3073.585152537" watchObservedRunningTime="2026-10-31 01:55:05.274127438 +0000 UTC m=+3073.750693822"
Oct 31 01:55:05.274872 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:55:05.274854    4964 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-pxz92"]
Oct 31 01:55:05.279433 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:55:05.279414    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-pxz92"
Oct 31 01:55:05.339940 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:55:05.339832    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-pxz92"]
Oct 31 01:55:05.427352 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:55:05.412736    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4b2d61c3-25da-4fdb-a53b-5989ff15122d-workload-certs\") pod \"e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-pxz92\" (UID: \"4b2d61c3-25da-4fdb-a53b-5989ff15122d\") " pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-pxz92"
Oct 31 01:55:05.427352 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:55:05.412797    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4b2d61c3-25da-4fdb-a53b-5989ff15122d-istiod-ca-cert\") pod \"e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-pxz92\" (UID: \"4b2d61c3-25da-4fdb-a53b-5989ff15122d\") " pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-pxz92"
Oct 31 01:55:05.427352 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:55:05.412838    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b2d61c3-25da-4fdb-a53b-5989ff15122d-credential-socket\") pod \"e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-pxz92\" (UID: \"4b2d61c3-25da-4fdb-a53b-5989ff15122d\") " pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-pxz92"
Oct 31 01:55:05.427352 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:55:05.412873    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4b2d61c3-25da-4fdb-a53b-5989ff15122d-istio-token\") pod \"e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-pxz92\" (UID: \"4b2d61c3-25da-4fdb-a53b-5989ff15122d\") " pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-pxz92"
Oct 31 01:55:05.427352 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:55:05.412929    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4b2d61c3-25da-4fdb-a53b-5989ff15122d-istio-data\") pod \"e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-pxz92\" (UID: \"4b2d61c3-25da-4fdb-a53b-5989ff15122d\") " pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-pxz92"
Oct 31 01:55:05.427352 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:55:05.412955    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2hrp8\" (UniqueName: \"kubernetes.io/projected/4b2d61c3-25da-4fdb-a53b-5989ff15122d-kube-api-access-2hrp8\") pod \"e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-pxz92\" (UID: \"4b2d61c3-25da-4fdb-a53b-5989ff15122d\") " pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-pxz92"
Oct 31 01:55:05.427352 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:55:05.413015    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4b2d61c3-25da-4fdb-a53b-5989ff15122d-istio-envoy\") pod \"e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-pxz92\" (UID: \"4b2d61c3-25da-4fdb-a53b-5989ff15122d\") " pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-pxz92"
Oct 31 01:55:05.427352 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:55:05.413058    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4b2d61c3-25da-4fdb-a53b-5989ff15122d-istio-podinfo\") pod \"e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-pxz92\" (UID: \"4b2d61c3-25da-4fdb-a53b-5989ff15122d\") " pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-pxz92"
Oct 31 01:55:05.427352 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:55:05.413103    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b2d61c3-25da-4fdb-a53b-5989ff15122d-workload-socket\") pod \"e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-pxz92\" (UID: \"4b2d61c3-25da-4fdb-a53b-5989ff15122d\") " pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-pxz92"
Oct 31 01:55:05.515352 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:55:05.514220    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4b2d61c3-25da-4fdb-a53b-5989ff15122d-workload-certs\") pod \"e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-pxz92\" (UID: \"4b2d61c3-25da-4fdb-a53b-5989ff15122d\") " pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-pxz92"
Oct 31 01:55:05.515352 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:55:05.514295    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4b2d61c3-25da-4fdb-a53b-5989ff15122d-istiod-ca-cert\") pod \"e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-pxz92\" (UID: \"4b2d61c3-25da-4fdb-a53b-5989ff15122d\") " pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-pxz92"
Oct 31 01:55:05.515352 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:55:05.514355    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b2d61c3-25da-4fdb-a53b-5989ff15122d-credential-socket\") pod \"e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-pxz92\" (UID: \"4b2d61c3-25da-4fdb-a53b-5989ff15122d\") " pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-pxz92"
Oct 31 01:55:05.515352 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:55:05.514394    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4b2d61c3-25da-4fdb-a53b-5989ff15122d-istio-token\") pod \"e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-pxz92\" (UID: \"4b2d61c3-25da-4fdb-a53b-5989ff15122d\") " pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-pxz92"
Oct 31 01:55:05.515352 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:55:05.514455    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4b2d61c3-25da-4fdb-a53b-5989ff15122d-istio-data\") pod \"e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-pxz92\" (UID: \"4b2d61c3-25da-4fdb-a53b-5989ff15122d\") " pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-pxz92"
Oct 31 01:55:05.515352 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:55:05.514500    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2hrp8\" (UniqueName: \"kubernetes.io/projected/4b2d61c3-25da-4fdb-a53b-5989ff15122d-kube-api-access-2hrp8\") pod \"e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-pxz92\" (UID: \"4b2d61c3-25da-4fdb-a53b-5989ff15122d\") " pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-pxz92"
Oct 31 01:55:05.515352 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:55:05.514561    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4b2d61c3-25da-4fdb-a53b-5989ff15122d-istio-envoy\") pod \"e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-pxz92\" (UID: \"4b2d61c3-25da-4fdb-a53b-5989ff15122d\") " pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-pxz92"
Oct 31 01:55:05.515352 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:55:05.514610    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4b2d61c3-25da-4fdb-a53b-5989ff15122d-istio-podinfo\") pod \"e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-pxz92\" (UID: \"4b2d61c3-25da-4fdb-a53b-5989ff15122d\") " pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-pxz92"
Oct 31 01:55:05.515352 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:55:05.514655    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b2d61c3-25da-4fdb-a53b-5989ff15122d-workload-socket\") pod \"e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-pxz92\" (UID: \"4b2d61c3-25da-4fdb-a53b-5989ff15122d\") " pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-pxz92"
Oct 31 01:55:05.515352 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:55:05.514794    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4b2d61c3-25da-4fdb-a53b-5989ff15122d-workload-certs\") pod \"e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-pxz92\" (UID: \"4b2d61c3-25da-4fdb-a53b-5989ff15122d\") " pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-pxz92"
Oct 31 01:55:05.515352 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:55:05.515050    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4b2d61c3-25da-4fdb-a53b-5989ff15122d-istio-data\") pod \"e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-pxz92\" (UID: \"4b2d61c3-25da-4fdb-a53b-5989ff15122d\") " pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-pxz92"
Oct 31 01:55:05.515352 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:55:05.515266    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b2d61c3-25da-4fdb-a53b-5989ff15122d-workload-socket\") pod \"e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-pxz92\" (UID: \"4b2d61c3-25da-4fdb-a53b-5989ff15122d\") " pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-pxz92"
Oct 31 01:55:05.523384 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:55:05.516586    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4b2d61c3-25da-4fdb-a53b-5989ff15122d-istiod-ca-cert\") pod \"e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-pxz92\" (UID: \"4b2d61c3-25da-4fdb-a53b-5989ff15122d\") " pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-pxz92"
Oct 31 01:55:05.524575 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:55:05.524508    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b2d61c3-25da-4fdb-a53b-5989ff15122d-credential-socket\") pod \"e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-pxz92\" (UID: \"4b2d61c3-25da-4fdb-a53b-5989ff15122d\") " pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-pxz92"
Oct 31 01:55:05.526776 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:55:05.526747    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4b2d61c3-25da-4fdb-a53b-5989ff15122d-istio-envoy\") pod \"e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-pxz92\" (UID: \"4b2d61c3-25da-4fdb-a53b-5989ff15122d\") " pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-pxz92"
Oct 31 01:55:05.528536 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:55:05.528502    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4b2d61c3-25da-4fdb-a53b-5989ff15122d-istio-podinfo\") pod \"e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-pxz92\" (UID: \"4b2d61c3-25da-4fdb-a53b-5989ff15122d\") " pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-pxz92"
Oct 31 01:55:05.571290 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:55:05.571253    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2hrp8\" (UniqueName: \"kubernetes.io/projected/4b2d61c3-25da-4fdb-a53b-5989ff15122d-kube-api-access-2hrp8\") pod \"e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-pxz92\" (UID: \"4b2d61c3-25da-4fdb-a53b-5989ff15122d\") " pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-pxz92"
Oct 31 01:55:05.579301 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:55:05.579277    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4b2d61c3-25da-4fdb-a53b-5989ff15122d-istio-token\") pod \"e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-pxz92\" (UID: \"4b2d61c3-25da-4fdb-a53b-5989ff15122d\") " pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-pxz92"
Oct 31 01:55:05.603883 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:55:05.603578    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-pxz92"
Oct 31 01:55:05.638658 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:55:05.638629    4964 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-c0daf4-openshift-default-84d68dc6f-g2858"
Oct 31 01:55:05.640127 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:55:05.640100    4964 patch_prober.go:29] interesting pod/e2e-praxis-a-c0daf4-openshift-default-84d68dc6f-g2858 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.174:15021/healthz/ready\": dial tcp 10.132.0.174:15021: connect: connection refused" start-of-body=
Oct 31 01:55:05.640242 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:55:05.640146    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-c0daf4-openshift-default-84d68dc6f-g2858" podUID="3d84fdf4-9d55-49fc-9784-0b008d97a20e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.174:15021/healthz/ready\": dial tcp 10.132.0.174:15021: connect: connection refused"
Oct 31 01:55:06.075114 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:55:06.075086    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-pxz92"]
Oct 31 01:55:06.076337 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:55:06.076291    4964 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b2d61c3_25da_4fdb_a53b_5989ff15122d.slice/crio-ba32c7005c5deca98b3ff9f291c60eaf3655f647c84818a40ef6f90ae0ca3674 WatchSource:0}: Error finding container ba32c7005c5deca98b3ff9f291c60eaf3655f647c84818a40ef6f90ae0ca3674: Status 404 returned error can't find the container with id ba32c7005c5deca98b3ff9f291c60eaf3655f647c84818a40ef6f90ae0ca3674
Oct 31 01:55:06.079021 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:55:06.078991    4964 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 01:55:06.079097 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:55:06.079061    4964 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 01:55:06.079097 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:55:06.079088    4964 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 01:55:06.637589 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:55:06.637555    4964 patch_prober.go:29] interesting pod/e2e-praxis-a-c0daf4-openshift-default-84d68dc6f-g2858 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.174:15021/healthz/ready\": dial tcp 10.132.0.174:15021: connect: connection refused" start-of-body=
Oct 31 01:55:06.638120 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:55:06.637612    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-c0daf4-openshift-default-84d68dc6f-g2858" podUID="3d84fdf4-9d55-49fc-9784-0b008d97a20e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.174:15021/healthz/ready\": dial tcp 10.132.0.174:15021: connect: connection refused"
Oct 31 01:55:06.959656 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:55:06.959550    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-pxz92" event={"ID":"4b2d61c3-25da-4fdb-a53b-5989ff15122d","Type":"ContainerStarted","Data":"c8b224ecbd33a5d60ae48b4414a7eda333e5652dd9455a8912cdc60c08baf7d7"}
Oct 31 01:55:06.959656 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:55:06.959648    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-pxz92" event={"ID":"4b2d61c3-25da-4fdb-a53b-5989ff15122d","Type":"ContainerStarted","Data":"ba32c7005c5deca98b3ff9f291c60eaf3655f647c84818a40ef6f90ae0ca3674"}
Oct 31 01:55:07.080423 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:55:07.080363    4964 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-pxz92" podStartSLOduration=2.080346602 podStartE2EDuration="2.080346602s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:55:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:55:07.077052745 +0000 UTC m=+3075.553619123" watchObservedRunningTime="2026-10-31 01:55:07.080346602 +0000 UTC m=+3075.556912982"
Oct 31 01:55:07.215397 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:55:07.214289    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:55:07.462448 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:55:07.462408    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:55:07.607344 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:55:07.606705    4964 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-pxz92"
Oct 31 01:55:07.609074 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:55:07.609041    4964 patch_prober.go:29] interesting pod/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-pxz92 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.175:15021/healthz/ready\": dial tcp 10.132.0.175:15021: connect: connection refused" start-of-body=
Oct 31 01:55:07.609356 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:55:07.609294    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-pxz92" podUID="4b2d61c3-25da-4fdb-a53b-5989ff15122d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.175:15021/healthz/ready\": dial tcp 10.132.0.175:15021: connect: connection refused"
Oct 31 01:55:07.637832 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:55:07.637795    4964 patch_prober.go:29] interesting pod/e2e-praxis-a-c0daf4-openshift-default-84d68dc6f-g2858 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.174:15021/healthz/ready\": dial tcp 10.132.0.174:15021: connect: connection refused" start-of-body=
Oct 31 01:55:07.638460 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:55:07.638430    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-c0daf4-openshift-default-84d68dc6f-g2858" podUID="3d84fdf4-9d55-49fc-9784-0b008d97a20e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.174:15021/healthz/ready\": dial tcp 10.132.0.174:15021: connect: connection refused"
Oct 31 01:55:07.687238 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:55:07.687206    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:55:08.607345 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:55:08.604761    4964 patch_prober.go:29] interesting pod/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-pxz92 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.175:15021/healthz/ready\": dial tcp 10.132.0.175:15021: connect: connection refused" start-of-body=
Oct 31 01:55:08.607345 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:55:08.604839    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-pxz92" podUID="4b2d61c3-25da-4fdb-a53b-5989ff15122d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.175:15021/healthz/ready\": dial tcp 10.132.0.175:15021: connect: connection refused"
Oct 31 01:55:08.645348 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:55:08.644477    4964 patch_prober.go:29] interesting pod/e2e-praxis-a-c0daf4-openshift-default-84d68dc6f-g2858 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.174:15021/healthz/ready\": dial tcp 10.132.0.174:15021: connect: connection refused" start-of-body=
Oct 31 01:55:08.645348 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:55:08.644543    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-c0daf4-openshift-default-84d68dc6f-g2858" podUID="3d84fdf4-9d55-49fc-9784-0b008d97a20e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.174:15021/healthz/ready\": dial tcp 10.132.0.174:15021: connect: connection refused"
Oct 31 01:55:09.612152 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:55:09.612117    4964 patch_prober.go:29] interesting pod/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-pxz92 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.175:15021/healthz/ready\": dial tcp 10.132.0.175:15021: connect: connection refused" start-of-body=
Oct 31 01:55:09.612478 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:55:09.612447    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-pxz92" podUID="4b2d61c3-25da-4fdb-a53b-5989ff15122d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.175:15021/healthz/ready\": dial tcp 10.132.0.175:15021: connect: connection refused"
Oct 31 01:55:09.640743 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:55:09.640706    4964 patch_prober.go:29] interesting pod/e2e-praxis-a-c0daf4-openshift-default-84d68dc6f-g2858 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.174:15021/healthz/ready\": dial tcp 10.132.0.174:15021: connect: connection refused" start-of-body=
Oct 31 01:55:09.641067 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:55:09.641030    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-c0daf4-openshift-default-84d68dc6f-g2858" podUID="3d84fdf4-9d55-49fc-9784-0b008d97a20e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.174:15021/healthz/ready\": dial tcp 10.132.0.174:15021: connect: connection refused"
Oct 31 01:55:10.605345 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:55:10.604984    4964 patch_prober.go:29] interesting pod/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-pxz92 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.175:15021/healthz/ready\": dial tcp 10.132.0.175:15021: connect: connection refused" start-of-body=
Oct 31 01:55:10.605345 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:55:10.605053    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-pxz92" podUID="4b2d61c3-25da-4fdb-a53b-5989ff15122d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.175:15021/healthz/ready\": dial tcp 10.132.0.175:15021: connect: connection refused"
Oct 31 01:55:10.932657 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:55:10.932561    4964 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-c0daf4-openshift-default-84d68dc6f-g2858"
Oct 31 01:55:10.933276 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:55:10.933135    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-c0daf4-openshift-default-84d68dc6f-g2858"
Oct 31 01:55:10.934556 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:55:10.934525    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-c0daf4-openshift-default-84d68dc6f-g2858"
Oct 31 01:55:11.272434 ip-10-0-133-197 kubenswrapper[4964]: E1031 01:55:11.270100    4964 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q56x5" containerName="extractor"
Oct 31 01:55:11.606767 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:55:11.606673    4964 patch_prober.go:29] interesting pod/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-pxz92 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.175:15021/healthz/ready\": dial tcp 10.132.0.175:15021: connect: connection refused" start-of-body=
Oct 31 01:55:11.607365 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:55:11.607310    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-pxz92" podUID="4b2d61c3-25da-4fdb-a53b-5989ff15122d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.175:15021/healthz/ready\": dial tcp 10.132.0.175:15021: connect: connection refused"
Oct 31 01:55:12.117253 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:55:12.117213    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:55:12.491583 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:55:12.491074    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:55:12.605235 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:55:12.605205    4964 patch_prober.go:29] interesting pod/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-pxz92 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.175:15021/healthz/ready\": dial tcp 10.132.0.175:15021: connect: connection refused" start-of-body=
Oct 31 01:55:12.605445 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:55:12.605272    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-pxz92" podUID="4b2d61c3-25da-4fdb-a53b-5989ff15122d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.175:15021/healthz/ready\": dial tcp 10.132.0.175:15021: connect: connection refused"
Oct 31 01:55:13.718974 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:55:13.718926    4964 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-pxz92"
Oct 31 01:55:13.719654 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:55:13.719631    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-pxz92"
Oct 31 01:55:13.720555 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:55:13.720536    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-pxz92"
Oct 31 01:55:13.923011 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:55:13.922945    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-051514-bd7d495b7-pkp25"
Oct 31 01:55:13.923186 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:55:13.923072    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-051514-546778dd4c-vbm49"
Oct 31 01:55:16.903525 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:55:16.903465    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:55:18.135215 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:55:18.135177    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:55:18.779585 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:55:18.779554    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:55:19.219545 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:55:19.219493    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:55:21.700570 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:55:21.700535    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:55:26.078503 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:55:26.078456    4964 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-476ce2-6f4f4c79d5-zv7hs"]
Oct 31 01:55:26.083053 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:55:26.083031    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-476ce2-6f4f4c79d5-zv7hs"
Oct 31 01:55:26.115033 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:55:26.114998    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0c20b1e6-06d4-4fcc-b595-588ca57b04e9-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-476ce2-6f4f4c79d5-zv7hs\" (UID: \"0c20b1e6-06d4-4fcc-b595-588ca57b04e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-476ce2-6f4f4c79d5-zv7hs"
Oct 31 01:55:26.115180 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:55:26.115065    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0c20b1e6-06d4-4fcc-b595-588ca57b04e9-maas-api-tls\") pod \"maas-api-e2e-praxis-b-476ce2-6f4f4c79d5-zv7hs\" (UID: \"0c20b1e6-06d4-4fcc-b595-588ca57b04e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-476ce2-6f4f4c79d5-zv7hs"
Oct 31 01:55:26.115228 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:55:26.115197    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ndnkj\" (UniqueName: \"kubernetes.io/projected/0c20b1e6-06d4-4fcc-b595-588ca57b04e9-kube-api-access-ndnkj\") pod \"maas-api-e2e-praxis-b-476ce2-6f4f4c79d5-zv7hs\" (UID: \"0c20b1e6-06d4-4fcc-b595-588ca57b04e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-476ce2-6f4f4c79d5-zv7hs"
Oct 31 01:55:26.115338 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:55:26.115303    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0c20b1e6-06d4-4fcc-b595-588ca57b04e9-metrics-certs\") pod \"maas-api-e2e-praxis-b-476ce2-6f4f4c79d5-zv7hs\" (UID: \"0c20b1e6-06d4-4fcc-b595-588ca57b04e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-476ce2-6f4f4c79d5-zv7hs"
Oct 31 01:55:26.141746 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:55:26.141706    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-476ce2-6f4f4c79d5-zv7hs"]
Oct 31 01:55:26.141934 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:55:26.141807    4964 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-b-476ce2-serving-cert\""
Oct 31 01:55:26.216905 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:55:26.216870    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0c20b1e6-06d4-4fcc-b595-588ca57b04e9-maas-api-tls\") pod \"maas-api-e2e-praxis-b-476ce2-6f4f4c79d5-zv7hs\" (UID: \"0c20b1e6-06d4-4fcc-b595-588ca57b04e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-476ce2-6f4f4c79d5-zv7hs"
Oct 31 01:55:26.217104 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:55:26.216953    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ndnkj\" (UniqueName: \"kubernetes.io/projected/0c20b1e6-06d4-4fcc-b595-588ca57b04e9-kube-api-access-ndnkj\") pod \"maas-api-e2e-praxis-b-476ce2-6f4f4c79d5-zv7hs\" (UID: \"0c20b1e6-06d4-4fcc-b595-588ca57b04e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-476ce2-6f4f4c79d5-zv7hs"
Oct 31 01:55:26.217104 ip-10-0-133-197 kubenswrapper[4964]: E1031 01:55:26.216980    4964 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-praxis-b-476ce2-serving-cert: secret "maas-api-e2e-praxis-b-476ce2-serving-cert" not found
Oct 31 01:55:26.217104 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:55:26.217033    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0c20b1e6-06d4-4fcc-b595-588ca57b04e9-metrics-certs\") pod \"maas-api-e2e-praxis-b-476ce2-6f4f4c79d5-zv7hs\" (UID: \"0c20b1e6-06d4-4fcc-b595-588ca57b04e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-476ce2-6f4f4c79d5-zv7hs"
Oct 31 01:55:26.217104 ip-10-0-133-197 kubenswrapper[4964]: E1031 01:55:26.217058    4964 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0c20b1e6-06d4-4fcc-b595-588ca57b04e9-maas-api-tls podName:0c20b1e6-06d4-4fcc-b595-588ca57b04e9 nodeName:}" failed. No retries permitted until 2026-10-31 01:55:26.71703851 +0000 UTC m=+3095.193604869 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/0c20b1e6-06d4-4fcc-b595-588ca57b04e9-maas-api-tls") pod "maas-api-e2e-praxis-b-476ce2-6f4f4c79d5-zv7hs" (UID: "0c20b1e6-06d4-4fcc-b595-588ca57b04e9") : secret "maas-api-e2e-praxis-b-476ce2-serving-cert" not found
Oct 31 01:55:26.217455 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:55:26.217288    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0c20b1e6-06d4-4fcc-b595-588ca57b04e9-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-476ce2-6f4f4c79d5-zv7hs\" (UID: \"0c20b1e6-06d4-4fcc-b595-588ca57b04e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-476ce2-6f4f4c79d5-zv7hs"
Oct 31 01:55:26.218116 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:55:26.218074    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0c20b1e6-06d4-4fcc-b595-588ca57b04e9-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-476ce2-6f4f4c79d5-zv7hs\" (UID: \"0c20b1e6-06d4-4fcc-b595-588ca57b04e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-476ce2-6f4f4c79d5-zv7hs"
Oct 31 01:55:26.221241 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:55:26.221218    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0c20b1e6-06d4-4fcc-b595-588ca57b04e9-metrics-certs\") pod \"maas-api-e2e-praxis-b-476ce2-6f4f4c79d5-zv7hs\" (UID: \"0c20b1e6-06d4-4fcc-b595-588ca57b04e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-476ce2-6f4f4c79d5-zv7hs"
Oct 31 01:55:26.274259 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:55:26.274231    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ndnkj\" (UniqueName: \"kubernetes.io/projected/0c20b1e6-06d4-4fcc-b595-588ca57b04e9-kube-api-access-ndnkj\") pod \"maas-api-e2e-praxis-b-476ce2-6f4f4c79d5-zv7hs\" (UID: \"0c20b1e6-06d4-4fcc-b595-588ca57b04e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-476ce2-6f4f4c79d5-zv7hs"
Oct 31 01:55:26.318081 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:55:26.318046    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:55:26.497887 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:55:26.497819    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:55:26.676871 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:55:26.676820    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:55:26.724813 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:55:26.724763    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0c20b1e6-06d4-4fcc-b595-588ca57b04e9-maas-api-tls\") pod \"maas-api-e2e-praxis-b-476ce2-6f4f4c79d5-zv7hs\" (UID: \"0c20b1e6-06d4-4fcc-b595-588ca57b04e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-476ce2-6f4f4c79d5-zv7hs"
Oct 31 01:55:26.727624 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:55:26.727598    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0c20b1e6-06d4-4fcc-b595-588ca57b04e9-maas-api-tls\") pod \"maas-api-e2e-praxis-b-476ce2-6f4f4c79d5-zv7hs\" (UID: \"0c20b1e6-06d4-4fcc-b595-588ca57b04e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-476ce2-6f4f4c79d5-zv7hs"
Oct 31 01:55:26.997285 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:55:26.997253    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-476ce2-6f4f4c79d5-zv7hs"
Oct 31 01:55:27.380254 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:55:27.380226    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-476ce2-6f4f4c79d5-zv7hs"]
Oct 31 01:55:27.381522 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:55:27.381487    4964 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c20b1e6_06d4_4fcc_b595_588ca57b04e9.slice/crio-cb3b7f14b85532f2e9c48cf0ff0a8140a29b9208148fc415bcf77a2916a57412 WatchSource:0}: Error finding container cb3b7f14b85532f2e9c48cf0ff0a8140a29b9208148fc415bcf77a2916a57412: Status 404 returned error can't find the container with id cb3b7f14b85532f2e9c48cf0ff0a8140a29b9208148fc415bcf77a2916a57412
Oct 31 01:55:28.139216 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:55:28.139182    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-476ce2-6f4f4c79d5-zv7hs" event={"ID":"0c20b1e6-06d4-4fcc-b595-588ca57b04e9","Type":"ContainerStarted","Data":"cb3b7f14b85532f2e9c48cf0ff0a8140a29b9208148fc415bcf77a2916a57412"}
Oct 31 01:55:29.154176 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:55:29.154133    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-476ce2-6f4f4c79d5-zv7hs" event={"ID":"0c20b1e6-06d4-4fcc-b595-588ca57b04e9","Type":"ContainerStarted","Data":"598e89b73c06c0bff1aa140da31b5e498339284dfd9bdc67d0c0c8e234aabb57"}
Oct 31 01:55:29.155300 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:55:29.155274    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-476ce2-6f4f4c79d5-zv7hs"
Oct 31 01:55:29.272069 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:55:29.272018    4964 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-476ce2-6f4f4c79d5-zv7hs" podStartSLOduration=2.448520791 podStartE2EDuration="3.27200499s" totalImagesPullingTime="823.484199ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:55:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:55:29.268389979 +0000 UTC m=+3097.744956357" watchObservedRunningTime="2026-10-31 01:55:29.27200499 +0000 UTC m=+3097.748571365"
Oct 31 01:55:29.879777 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:55:29.879746    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:55:31.541560 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:55:31.541449    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:55:32.538271 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:55:32.538230    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:55:32.728698 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:55:32.728653    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:55:32.912185 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:55:32.912076    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:55:35.726332 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:55:35.726285    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:55:35.942574 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:55:35.942529    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:55:36.173843 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:55:36.173756    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-476ce2-6f4f4c79d5-zv7hs"
Oct 31 01:55:36.289128 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:55:36.289070    4964 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-qt5vs"]
Oct 31 01:55:36.289798 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:55:36.289755    4964 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-qt5vs" podUID="fb838d78-ff4a-4199-a6e7-d69bd176a022" containerName="istio-proxy" containerID="cri-o://ea890ad2297c2b1c42671bdcbf1b3c333d54df0506082ee88e234310a7662fcb" gracePeriod=30
Oct 31 01:55:36.536702 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:55:36.536661    4964 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-98b70c-55c5cd675b-ltbhs"]
Oct 31 01:55:36.543373 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:55:36.543350    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-98b70c-55c5cd675b-ltbhs"
Oct 31 01:55:36.595595 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:55:36.595163    4964 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w0-98b70c-serving-cert\""
Oct 31 01:55:36.599982 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:55:36.599952    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-98b70c-55c5cd675b-ltbhs"]
Oct 31 01:55:36.667994 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:55:36.667959    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5ce8150f-4c48-4d65-8416-442f611c53fc-maas-api-tls\") pod \"maas-api-e2e-shared-a-w0-98b70c-55c5cd675b-ltbhs\" (UID: \"5ce8150f-4c48-4d65-8416-442f611c53fc\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-98b70c-55c5cd675b-ltbhs"
Oct 31 01:55:36.668197 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:55:36.668176    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5ce8150f-4c48-4d65-8416-442f611c53fc-metrics-certs\") pod \"maas-api-e2e-shared-a-w0-98b70c-55c5cd675b-ltbhs\" (UID: \"5ce8150f-4c48-4d65-8416-442f611c53fc\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-98b70c-55c5cd675b-ltbhs"
Oct 31 01:55:36.668327 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:55:36.668295    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tdmkx\" (UniqueName: \"kubernetes.io/projected/5ce8150f-4c48-4d65-8416-442f611c53fc-kube-api-access-tdmkx\") pod \"maas-api-e2e-shared-a-w0-98b70c-55c5cd675b-ltbhs\" (UID: \"5ce8150f-4c48-4d65-8416-442f611c53fc\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-98b70c-55c5cd675b-ltbhs"
Oct 31 01:55:36.668375 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:55:36.668344    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5ce8150f-4c48-4d65-8416-442f611c53fc-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w0-98b70c-55c5cd675b-ltbhs\" (UID: \"5ce8150f-4c48-4d65-8416-442f611c53fc\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-98b70c-55c5cd675b-ltbhs"
Oct 31 01:55:36.765144 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:55:36.765107    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:55:36.769161 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:55:36.769128    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tdmkx\" (UniqueName: \"kubernetes.io/projected/5ce8150f-4c48-4d65-8416-442f611c53fc-kube-api-access-tdmkx\") pod \"maas-api-e2e-shared-a-w0-98b70c-55c5cd675b-ltbhs\" (UID: \"5ce8150f-4c48-4d65-8416-442f611c53fc\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-98b70c-55c5cd675b-ltbhs"
Oct 31 01:55:36.769431 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:55:36.769175    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5ce8150f-4c48-4d65-8416-442f611c53fc-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w0-98b70c-55c5cd675b-ltbhs\" (UID: \"5ce8150f-4c48-4d65-8416-442f611c53fc\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-98b70c-55c5cd675b-ltbhs"
Oct 31 01:55:36.769431 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:55:36.769263    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5ce8150f-4c48-4d65-8416-442f611c53fc-maas-api-tls\") pod \"maas-api-e2e-shared-a-w0-98b70c-55c5cd675b-ltbhs\" (UID: \"5ce8150f-4c48-4d65-8416-442f611c53fc\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-98b70c-55c5cd675b-ltbhs"
Oct 31 01:55:36.769756 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:55:36.769441    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5ce8150f-4c48-4d65-8416-442f611c53fc-metrics-certs\") pod \"maas-api-e2e-shared-a-w0-98b70c-55c5cd675b-ltbhs\" (UID: \"5ce8150f-4c48-4d65-8416-442f611c53fc\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-98b70c-55c5cd675b-ltbhs"
Oct 31 01:55:36.770909 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:55:36.770885    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5ce8150f-4c48-4d65-8416-442f611c53fc-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w0-98b70c-55c5cd675b-ltbhs\" (UID: \"5ce8150f-4c48-4d65-8416-442f611c53fc\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-98b70c-55c5cd675b-ltbhs"
Oct 31 01:55:36.771732 ip-10-0-133-197 kubenswrapper[4964]: E1031 01:55:36.771707    4964 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-98b70c-serving-cert: secret "maas-api-e2e-shared-a-w0-98b70c-serving-cert" not found
Oct 31 01:55:36.771933 ip-10-0-133-197 kubenswrapper[4964]: E1031 01:55:36.771778    4964 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5ce8150f-4c48-4d65-8416-442f611c53fc-maas-api-tls podName:5ce8150f-4c48-4d65-8416-442f611c53fc nodeName:}" failed. No retries permitted until 2026-10-31 01:55:37.271760568 +0000 UTC m=+3105.748326932 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/5ce8150f-4c48-4d65-8416-442f611c53fc-maas-api-tls") pod "maas-api-e2e-shared-a-w0-98b70c-55c5cd675b-ltbhs" (UID: "5ce8150f-4c48-4d65-8416-442f611c53fc") : secret "maas-api-e2e-shared-a-w0-98b70c-serving-cert" not found
Oct 31 01:55:36.776011 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:55:36.775986    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5ce8150f-4c48-4d65-8416-442f611c53fc-metrics-certs\") pod \"maas-api-e2e-shared-a-w0-98b70c-55c5cd675b-ltbhs\" (UID: \"5ce8150f-4c48-4d65-8416-442f611c53fc\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-98b70c-55c5cd675b-ltbhs"
Oct 31 01:55:36.831588 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:55:36.831504    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tdmkx\" (UniqueName: \"kubernetes.io/projected/5ce8150f-4c48-4d65-8416-442f611c53fc-kube-api-access-tdmkx\") pod \"maas-api-e2e-shared-a-w0-98b70c-55c5cd675b-ltbhs\" (UID: \"5ce8150f-4c48-4d65-8416-442f611c53fc\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-98b70c-55c5cd675b-ltbhs"
Oct 31 01:55:36.985762 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:55:36.985724    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:55:37.159531 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:55:37.159444    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:55:37.277463 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:55:37.277431    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5ce8150f-4c48-4d65-8416-442f611c53fc-maas-api-tls\") pod \"maas-api-e2e-shared-a-w0-98b70c-55c5cd675b-ltbhs\" (UID: \"5ce8150f-4c48-4d65-8416-442f611c53fc\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-98b70c-55c5cd675b-ltbhs"
Oct 31 01:55:37.280078 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:55:37.280046    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5ce8150f-4c48-4d65-8416-442f611c53fc-maas-api-tls\") pod \"maas-api-e2e-shared-a-w0-98b70c-55c5cd675b-ltbhs\" (UID: \"5ce8150f-4c48-4d65-8416-442f611c53fc\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-98b70c-55c5cd675b-ltbhs"
Oct 31 01:55:37.456815 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:55:37.456734    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-98b70c-55c5cd675b-ltbhs"
Oct 31 01:55:37.840646 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:55:37.840613    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-98b70c-55c5cd675b-ltbhs"]
Oct 31 01:55:37.841879 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:55:37.841848    4964 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ce8150f_4c48_4d65_8416_442f611c53fc.slice/crio-23ea9be5e35113119900969b02547ec097eaadc4ec9380df9a4641bfd9a938d2 WatchSource:0}: Error finding container 23ea9be5e35113119900969b02547ec097eaadc4ec9380df9a4641bfd9a938d2: Status 404 returned error can't find the container with id 23ea9be5e35113119900969b02547ec097eaadc4ec9380df9a4641bfd9a938d2
Oct 31 01:55:38.239148 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:55:38.239106    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-98b70c-55c5cd675b-ltbhs" event={"ID":"5ce8150f-4c48-4d65-8416-442f611c53fc","Type":"ContainerStarted","Data":"23ea9be5e35113119900969b02547ec097eaadc4ec9380df9a4641bfd9a938d2"}
Oct 31 01:55:39.247919 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:55:39.247888    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-98b70c-55c5cd675b-ltbhs" event={"ID":"5ce8150f-4c48-4d65-8416-442f611c53fc","Type":"ContainerStarted","Data":"cad610a9b748cbf4df64bcd9825a3333c00ab79bfdc3e210e5fc40ab22ac17a3"}
Oct 31 01:55:39.248406 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:55:39.248042    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-98b70c-55c5cd675b-ltbhs"
Oct 31 01:55:39.363921 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:55:39.363863    4964 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-98b70c-55c5cd675b-ltbhs" podStartSLOduration=2.496024064 podStartE2EDuration="3.363843577s" totalImagesPullingTime="867.819513ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:55:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:55:39.363279682 +0000 UTC m=+3107.839846063" watchObservedRunningTime="2026-10-31 01:55:39.363843577 +0000 UTC m=+3107.840409958"
Oct 31 01:55:40.720837 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:55:40.720803    4964 patch_prober.go:29] interesting pod/e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-qt5vs container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 01:55:40.721369 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:55:40.720856    4964 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-qt5vs" podUID="fb838d78-ff4a-4199-a6e7-d69bd176a022" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 01:55:41.240361 ip-10-0-133-197 kubenswrapper[4964]: E1031 01:55:41.240308    4964 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q56x5" containerName="extractor"
Oct 31 01:55:41.594094 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:55:41.594067    4964 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-qt5vs"
Oct 31 01:55:41.680685 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:55:41.680648    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:55:41.738279 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:55:41.738234    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fb838d78-ff4a-4199-a6e7-d69bd176a022-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fb838d78-ff4a-4199-a6e7-d69bd176a022-workload-certs\") pod \"fb838d78-ff4a-4199-a6e7-d69bd176a022\" (UID: \"fb838d78-ff4a-4199-a6e7-d69bd176a022\") "
Oct 31 01:55:41.738795 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:55:41.738305    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fb838d78-ff4a-4199-a6e7-d69bd176a022-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fb838d78-ff4a-4199-a6e7-d69bd176a022-istiod-ca-cert\") pod \"fb838d78-ff4a-4199-a6e7-d69bd176a022\" (UID: \"fb838d78-ff4a-4199-a6e7-d69bd176a022\") "
Oct 31 01:55:41.738795 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:55:41.738366    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fb838d78-ff4a-4199-a6e7-d69bd176a022-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fb838d78-ff4a-4199-a6e7-d69bd176a022-istio-data\") pod \"fb838d78-ff4a-4199-a6e7-d69bd176a022\" (UID: \"fb838d78-ff4a-4199-a6e7-d69bd176a022\") "
Oct 31 01:55:41.738795 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:55:41.738397    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fb838d78-ff4a-4199-a6e7-d69bd176a022-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb838d78-ff4a-4199-a6e7-d69bd176a022-workload-socket\") pod \"fb838d78-ff4a-4199-a6e7-d69bd176a022\" (UID: \"fb838d78-ff4a-4199-a6e7-d69bd176a022\") "
Oct 31 01:55:41.738795 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:55:41.738442    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fb838d78-ff4a-4199-a6e7-d69bd176a022-kube-api-access-rntgc\" (UniqueName: \"kubernetes.io/projected/fb838d78-ff4a-4199-a6e7-d69bd176a022-kube-api-access-rntgc\") pod \"fb838d78-ff4a-4199-a6e7-d69bd176a022\" (UID: \"fb838d78-ff4a-4199-a6e7-d69bd176a022\") "
Oct 31 01:55:41.738795 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:55:41.738481    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fb838d78-ff4a-4199-a6e7-d69bd176a022-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb838d78-ff4a-4199-a6e7-d69bd176a022-credential-socket\") pod \"fb838d78-ff4a-4199-a6e7-d69bd176a022\" (UID: \"fb838d78-ff4a-4199-a6e7-d69bd176a022\") "
Oct 31 01:55:41.738795 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:55:41.738526    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fb838d78-ff4a-4199-a6e7-d69bd176a022-istio-token\" (UniqueName: \"kubernetes.io/projected/fb838d78-ff4a-4199-a6e7-d69bd176a022-istio-token\") pod \"fb838d78-ff4a-4199-a6e7-d69bd176a022\" (UID: \"fb838d78-ff4a-4199-a6e7-d69bd176a022\") "
Oct 31 01:55:41.738795 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:55:41.738585    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb838d78-ff4a-4199-a6e7-d69bd176a022-workload-certs" pod "fb838d78-ff4a-4199-a6e7-d69bd176a022" (UID: "fb838d78-ff4a-4199-a6e7-d69bd176a022"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:55:41.738795 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:55:41.738650    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb838d78-ff4a-4199-a6e7-d69bd176a022-workload-socket" pod "fb838d78-ff4a-4199-a6e7-d69bd176a022" (UID: "fb838d78-ff4a-4199-a6e7-d69bd176a022"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:55:41.738795 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:55:41.738677    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb838d78-ff4a-4199-a6e7-d69bd176a022-istio-data" pod "fb838d78-ff4a-4199-a6e7-d69bd176a022" (UID: "fb838d78-ff4a-4199-a6e7-d69bd176a022"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:55:41.738795 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:55:41.738734    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fb838d78-ff4a-4199-a6e7-d69bd176a022-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fb838d78-ff4a-4199-a6e7-d69bd176a022-istio-envoy\") pod \"fb838d78-ff4a-4199-a6e7-d69bd176a022\" (UID: \"fb838d78-ff4a-4199-a6e7-d69bd176a022\") "
Oct 31 01:55:41.738795 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:55:41.738744    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fb838d78-ff4a-4199-a6e7-d69bd176a022-istiod-ca-cert" pod "fb838d78-ff4a-4199-a6e7-d69bd176a022" (UID: "fb838d78-ff4a-4199-a6e7-d69bd176a022"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:55:41.738795 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:55:41.738786    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fb838d78-ff4a-4199-a6e7-d69bd176a022-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fb838d78-ff4a-4199-a6e7-d69bd176a022-istio-podinfo\") pod \"fb838d78-ff4a-4199-a6e7-d69bd176a022\" (UID: \"fb838d78-ff4a-4199-a6e7-d69bd176a022\") "
Oct 31 01:55:41.739505 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:55:41.738826    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb838d78-ff4a-4199-a6e7-d69bd176a022-credential-socket" pod "fb838d78-ff4a-4199-a6e7-d69bd176a022" (UID: "fb838d78-ff4a-4199-a6e7-d69bd176a022"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:55:41.739617 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:55:41.739597    4964 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fb838d78-ff4a-4199-a6e7-d69bd176a022-istio-data\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:55:41.739854 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:55:41.739669    4964 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb838d78-ff4a-4199-a6e7-d69bd176a022-workload-socket\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:55:41.739854 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:55:41.739687    4964 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb838d78-ff4a-4199-a6e7-d69bd176a022-credential-socket\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:55:41.739854 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:55:41.739703    4964 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fb838d78-ff4a-4199-a6e7-d69bd176a022-workload-certs\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:55:41.739854 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:55:41.739717    4964 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fb838d78-ff4a-4199-a6e7-d69bd176a022-istiod-ca-cert\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:55:41.741768 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:55:41.741429    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fb838d78-ff4a-4199-a6e7-d69bd176a022-istio-podinfo" pod "fb838d78-ff4a-4199-a6e7-d69bd176a022" (UID: "fb838d78-ff4a-4199-a6e7-d69bd176a022"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 01:55:41.741768 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:55:41.741632    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb838d78-ff4a-4199-a6e7-d69bd176a022-istio-envoy" pod "fb838d78-ff4a-4199-a6e7-d69bd176a022" (UID: "fb838d78-ff4a-4199-a6e7-d69bd176a022"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:55:41.741768 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:55:41.741702    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb838d78-ff4a-4199-a6e7-d69bd176a022-kube-api-access-rntgc" pod "fb838d78-ff4a-4199-a6e7-d69bd176a022" (UID: "fb838d78-ff4a-4199-a6e7-d69bd176a022"). InnerVolumeSpecName "kube-api-access-rntgc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:55:41.742069 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:55:41.742051    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb838d78-ff4a-4199-a6e7-d69bd176a022-istio-token" pod "fb838d78-ff4a-4199-a6e7-d69bd176a022" (UID: "fb838d78-ff4a-4199-a6e7-d69bd176a022"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:55:41.841043 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:55:41.840957    4964 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rntgc\" (UniqueName: \"kubernetes.io/projected/fb838d78-ff4a-4199-a6e7-d69bd176a022-kube-api-access-rntgc\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:55:41.841043 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:55:41.840989    4964 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fb838d78-ff4a-4199-a6e7-d69bd176a022-istio-token\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:55:41.841043 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:55:41.841000    4964 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fb838d78-ff4a-4199-a6e7-d69bd176a022-istio-envoy\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:55:41.841043 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:55:41.841009    4964 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fb838d78-ff4a-4199-a6e7-d69bd176a022-istio-podinfo\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:55:42.285259 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:55:42.285227    4964 generic.go:356] "Generic (PLEG): container finished" podID="fb838d78-ff4a-4199-a6e7-d69bd176a022" containerID="ea890ad2297c2b1c42671bdcbf1b3c333d54df0506082ee88e234310a7662fcb" exitCode=0
Oct 31 01:55:42.285477 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:55:42.285285    4964 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-qt5vs"
Oct 31 01:55:42.285477 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:55:42.285308    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-qt5vs" event={"ID":"fb838d78-ff4a-4199-a6e7-d69bd176a022","Type":"ContainerDied","Data":"ea890ad2297c2b1c42671bdcbf1b3c333d54df0506082ee88e234310a7662fcb"}
Oct 31 01:55:42.285477 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:55:42.285433    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-qt5vs" event={"ID":"fb838d78-ff4a-4199-a6e7-d69bd176a022","Type":"ContainerDied","Data":"1c48c5eade4ac2e01916a1c175db136e63e1ed915b7345bdaa2b3e47f2085b5f"}
Oct 31 01:55:42.285477 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:55:42.285463    4964 scope.go:122] "RemoveContainer" containerID="ea890ad2297c2b1c42671bdcbf1b3c333d54df0506082ee88e234310a7662fcb"
Oct 31 01:55:42.296489 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:55:42.296458    4964 scope.go:122] "RemoveContainer" containerID="ea890ad2297c2b1c42671bdcbf1b3c333d54df0506082ee88e234310a7662fcb"
Oct 31 01:55:42.296887 ip-10-0-133-197 kubenswrapper[4964]: E1031 01:55:42.296862    4964 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ea890ad2297c2b1c42671bdcbf1b3c333d54df0506082ee88e234310a7662fcb\": container with ID starting with ea890ad2297c2b1c42671bdcbf1b3c333d54df0506082ee88e234310a7662fcb not found: ID does not exist" containerID="ea890ad2297c2b1c42671bdcbf1b3c333d54df0506082ee88e234310a7662fcb"
Oct 31 01:55:42.296946 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:55:42.296906    4964 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ea890ad2297c2b1c42671bdcbf1b3c333d54df0506082ee88e234310a7662fcb"} err="failed to get container status \"ea890ad2297c2b1c42671bdcbf1b3c333d54df0506082ee88e234310a7662fcb\": rpc error: code = NotFound desc = could not find container \"ea890ad2297c2b1c42671bdcbf1b3c333d54df0506082ee88e234310a7662fcb\": container with ID starting with ea890ad2297c2b1c42671bdcbf1b3c333d54df0506082ee88e234310a7662fcb not found: ID does not exist"
Oct 31 01:55:42.450804 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:55:42.450763    4964 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-qt5vs"]
Oct 31 01:55:42.454693 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:55:42.454671    4964 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-qt5vs"]
Oct 31 01:55:43.778966 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:55:43.778926    4964 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fb838d78-ff4a-4199-a6e7-d69bd176a022" path="/var/lib/kubelet/pods/fb838d78-ff4a-4199-a6e7-d69bd176a022/volumes"
Oct 31 01:55:45.260956 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:55:45.260928    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-98b70c-55c5cd675b-ltbhs"
Oct 31 01:55:46.415510 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:55:46.415472    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:55:49.239807 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:55:49.239767    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:55:49.550736 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:55:49.550633    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:55:49.979784 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:55:49.979747    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:55:51.192912 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:55:51.192872    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:55:52.791901 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:55:52.791855    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:55:55.084124 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:55:55.084089    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:55:55.258475 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:55:55.258434    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:55:55.367338 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:55:55.367221    4964 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-c0daf4-openshift-default-84d68dc6f-g2858"]
Oct 31 01:55:55.367569 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:55:55.367519    4964 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-a-c0daf4-openshift-default-84d68dc6f-g2858" podUID="3d84fdf4-9d55-49fc-9784-0b008d97a20e" containerName="istio-proxy" containerID="cri-o://c8494344c3e8f9dea2ad9a895300a3f549b570e62facf3b6bb2cd85d58080748" gracePeriod=30
Oct 31 01:55:55.442699 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:55:55.442657    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:55:55.669109 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:55:55.664730    4964 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-051514-6d5fbc88-4xc4r"]
Oct 31 01:55:55.669109 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:55:55.665882    4964 cpu_manager.go:414] "containerMap: removing container" podUID="fb838d78-ff4a-4199-a6e7-d69bd176a022" containerName="istio-proxy"
Oct 31 01:55:55.669109 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:55:55.665904    4964 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fb838d78-ff4a-4199-a6e7-d69bd176a022" containerName="istio-proxy"
Oct 31 01:55:55.669109 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:55:55.666064    4964 memory_manager.go:365] "RemoveStaleState removing state" podUID="fb838d78-ff4a-4199-a6e7-d69bd176a022" containerName="istio-proxy"
Oct 31 01:55:55.675067 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:55:55.675037    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-051514-6d5fbc88-4xc4r"
Oct 31 01:55:55.733814 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:55:55.731601    4964 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w3-051514-serving-cert\""
Oct 31 01:55:55.734774 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:55:55.734743    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-051514-6d5fbc88-4xc4r"]
Oct 31 01:55:55.762023 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:55:55.758460    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/05dff628-00f6-4008-8eee-f7e7dcdf8729-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-051514-6d5fbc88-4xc4r\" (UID: \"05dff628-00f6-4008-8eee-f7e7dcdf8729\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-051514-6d5fbc88-4xc4r"
Oct 31 01:55:55.762023 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:55:55.758602    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/05dff628-00f6-4008-8eee-f7e7dcdf8729-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-051514-6d5fbc88-4xc4r\" (UID: \"05dff628-00f6-4008-8eee-f7e7dcdf8729\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-051514-6d5fbc88-4xc4r"
Oct 31 01:55:55.762023 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:55:55.758683    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nrg7c\" (UniqueName: \"kubernetes.io/projected/05dff628-00f6-4008-8eee-f7e7dcdf8729-kube-api-access-nrg7c\") pod \"maas-api-e2e-shared-a-w3-051514-6d5fbc88-4xc4r\" (UID: \"05dff628-00f6-4008-8eee-f7e7dcdf8729\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-051514-6d5fbc88-4xc4r"
Oct 31 01:55:55.762023 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:55:55.758721    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/05dff628-00f6-4008-8eee-f7e7dcdf8729-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-051514-6d5fbc88-4xc4r\" (UID: \"05dff628-00f6-4008-8eee-f7e7dcdf8729\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-051514-6d5fbc88-4xc4r"
Oct 31 01:55:55.860284 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:55:55.860237    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nrg7c\" (UniqueName: \"kubernetes.io/projected/05dff628-00f6-4008-8eee-f7e7dcdf8729-kube-api-access-nrg7c\") pod \"maas-api-e2e-shared-a-w3-051514-6d5fbc88-4xc4r\" (UID: \"05dff628-00f6-4008-8eee-f7e7dcdf8729\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-051514-6d5fbc88-4xc4r"
Oct 31 01:55:55.861177 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:55:55.860309    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/05dff628-00f6-4008-8eee-f7e7dcdf8729-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-051514-6d5fbc88-4xc4r\" (UID: \"05dff628-00f6-4008-8eee-f7e7dcdf8729\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-051514-6d5fbc88-4xc4r"
Oct 31 01:55:55.861177 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:55:55.860513    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/05dff628-00f6-4008-8eee-f7e7dcdf8729-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-051514-6d5fbc88-4xc4r\" (UID: \"05dff628-00f6-4008-8eee-f7e7dcdf8729\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-051514-6d5fbc88-4xc4r"
Oct 31 01:55:55.861177 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:55:55.860719    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/05dff628-00f6-4008-8eee-f7e7dcdf8729-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-051514-6d5fbc88-4xc4r\" (UID: \"05dff628-00f6-4008-8eee-f7e7dcdf8729\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-051514-6d5fbc88-4xc4r"
Oct 31 01:55:55.862292 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:55:55.862260    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/05dff628-00f6-4008-8eee-f7e7dcdf8729-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-051514-6d5fbc88-4xc4r\" (UID: \"05dff628-00f6-4008-8eee-f7e7dcdf8729\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-051514-6d5fbc88-4xc4r"
Oct 31 01:55:55.862796 ip-10-0-133-197 kubenswrapper[4964]: E1031 01:55:55.862769    4964 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-051514-serving-cert: secret "maas-api-e2e-shared-a-w3-051514-serving-cert" not found
Oct 31 01:55:55.862901 ip-10-0-133-197 kubenswrapper[4964]: E1031 01:55:55.862842    4964 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/05dff628-00f6-4008-8eee-f7e7dcdf8729-maas-api-tls podName:05dff628-00f6-4008-8eee-f7e7dcdf8729 nodeName:}" failed. No retries permitted until 2026-10-31 01:55:56.362822813 +0000 UTC m=+3124.839389189 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/05dff628-00f6-4008-8eee-f7e7dcdf8729-maas-api-tls") pod "maas-api-e2e-shared-a-w3-051514-6d5fbc88-4xc4r" (UID: "05dff628-00f6-4008-8eee-f7e7dcdf8729") : secret "maas-api-e2e-shared-a-w3-051514-serving-cert" not found
Oct 31 01:55:55.866835 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:55:55.866802    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/05dff628-00f6-4008-8eee-f7e7dcdf8729-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-051514-6d5fbc88-4xc4r\" (UID: \"05dff628-00f6-4008-8eee-f7e7dcdf8729\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-051514-6d5fbc88-4xc4r"
Oct 31 01:55:55.887178 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:55:55.886956    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:55:55.965443 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:55:55.964496    4964 patch_prober.go:29] interesting pod/e2e-praxis-a-c0daf4-openshift-default-84d68dc6f-g2858 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 01:55:55.965443 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:55:55.964560    4964 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-praxis-a-c0daf4-openshift-default-84d68dc6f-g2858" podUID="3d84fdf4-9d55-49fc-9784-0b008d97a20e" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 01:55:55.967549 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:55:55.966310    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nrg7c\" (UniqueName: \"kubernetes.io/projected/05dff628-00f6-4008-8eee-f7e7dcdf8729-kube-api-access-nrg7c\") pod \"maas-api-e2e-shared-a-w3-051514-6d5fbc88-4xc4r\" (UID: \"05dff628-00f6-4008-8eee-f7e7dcdf8729\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-051514-6d5fbc88-4xc4r"
Oct 31 01:55:56.122070 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:55:56.122034    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:55:56.372921 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:55:56.372878    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/05dff628-00f6-4008-8eee-f7e7dcdf8729-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-051514-6d5fbc88-4xc4r\" (UID: \"05dff628-00f6-4008-8eee-f7e7dcdf8729\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-051514-6d5fbc88-4xc4r"
Oct 31 01:55:56.373240 ip-10-0-133-197 kubenswrapper[4964]: E1031 01:55:56.373217    4964 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-051514-serving-cert: secret "maas-api-e2e-shared-a-w3-051514-serving-cert" not found
Oct 31 01:55:56.373357 ip-10-0-133-197 kubenswrapper[4964]: E1031 01:55:56.373302    4964 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/05dff628-00f6-4008-8eee-f7e7dcdf8729-maas-api-tls podName:05dff628-00f6-4008-8eee-f7e7dcdf8729 nodeName:}" failed. No retries permitted until 2026-10-31 01:55:57.373283152 +0000 UTC m=+3125.849849532 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/05dff628-00f6-4008-8eee-f7e7dcdf8729-maas-api-tls") pod "maas-api-e2e-shared-a-w3-051514-6d5fbc88-4xc4r" (UID: "05dff628-00f6-4008-8eee-f7e7dcdf8729") : secret "maas-api-e2e-shared-a-w3-051514-serving-cert" not found
Oct 31 01:55:56.580453 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:55:56.577452    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:55:57.388984 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:55:57.388948    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/05dff628-00f6-4008-8eee-f7e7dcdf8729-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-051514-6d5fbc88-4xc4r\" (UID: \"05dff628-00f6-4008-8eee-f7e7dcdf8729\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-051514-6d5fbc88-4xc4r"
Oct 31 01:55:57.411344 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:55:57.410659    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/05dff628-00f6-4008-8eee-f7e7dcdf8729-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-051514-6d5fbc88-4xc4r\" (UID: \"05dff628-00f6-4008-8eee-f7e7dcdf8729\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-051514-6d5fbc88-4xc4r"
Oct 31 01:55:57.489559 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:55:57.489526    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-051514-6d5fbc88-4xc4r"
Oct 31 01:55:58.031886 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:55:58.030915    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-051514-6d5fbc88-4xc4r"]
Oct 31 01:55:58.453896 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:55:58.453595    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-051514-6d5fbc88-4xc4r" event={"ID":"05dff628-00f6-4008-8eee-f7e7dcdf8729","Type":"ContainerStarted","Data":"ca5b2ea2d9f23a747055f01cb214bb8ea5fe31a2809b6be332b05941c21bf3a4"}
Oct 31 01:55:59.462927 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:55:59.462859    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-051514-6d5fbc88-4xc4r" event={"ID":"05dff628-00f6-4008-8eee-f7e7dcdf8729","Type":"ContainerStarted","Data":"ecb949a45b3da47bb0a0884365e71dff9e7d9b6558c71e228aa5d4882ae96dac"}
Oct 31 01:55:59.463441 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:55:59.463424    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-051514-6d5fbc88-4xc4r"
Oct 31 01:55:59.588468 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:55:59.588407    4964 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-051514-6d5fbc88-4xc4r" podStartSLOduration=3.724487354 podStartE2EDuration="4.588387533s" totalImagesPullingTime="863.900179ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:55:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:55:59.582965323 +0000 UTC m=+3128.059531699" watchObservedRunningTime="2026-10-31 01:55:59.588387533 +0000 UTC m=+3128.064953917"
Oct 31 01:56:00.745255 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:56:00.745228    4964 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-c0daf4-openshift-default-84d68dc6f-g2858"
Oct 31 01:56:00.836165 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:56:00.836123    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:56:00.847656 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:56:00.847597    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3d84fdf4-9d55-49fc-9784-0b008d97a20e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d84fdf4-9d55-49fc-9784-0b008d97a20e-workload-certs\") pod \"3d84fdf4-9d55-49fc-9784-0b008d97a20e\" (UID: \"3d84fdf4-9d55-49fc-9784-0b008d97a20e\") "
Oct 31 01:56:00.847906 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:56:00.847699    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3d84fdf4-9d55-49fc-9784-0b008d97a20e-istio-token\" (UniqueName: \"kubernetes.io/projected/3d84fdf4-9d55-49fc-9784-0b008d97a20e-istio-token\") pod \"3d84fdf4-9d55-49fc-9784-0b008d97a20e\" (UID: \"3d84fdf4-9d55-49fc-9784-0b008d97a20e\") "
Oct 31 01:56:00.848548 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:56:00.848102    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d84fdf4-9d55-49fc-9784-0b008d97a20e-workload-certs" pod "3d84fdf4-9d55-49fc-9784-0b008d97a20e" (UID: "3d84fdf4-9d55-49fc-9784-0b008d97a20e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:56:00.848548 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:56:00.848221    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3d84fdf4-9d55-49fc-9784-0b008d97a20e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3d84fdf4-9d55-49fc-9784-0b008d97a20e-istio-data\") pod \"3d84fdf4-9d55-49fc-9784-0b008d97a20e\" (UID: \"3d84fdf4-9d55-49fc-9784-0b008d97a20e\") "
Oct 31 01:56:00.848548 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:56:00.848257    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3d84fdf4-9d55-49fc-9784-0b008d97a20e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d84fdf4-9d55-49fc-9784-0b008d97a20e-credential-socket\") pod \"3d84fdf4-9d55-49fc-9784-0b008d97a20e\" (UID: \"3d84fdf4-9d55-49fc-9784-0b008d97a20e\") "
Oct 31 01:56:00.848548 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:56:00.848328    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3d84fdf4-9d55-49fc-9784-0b008d97a20e-kube-api-access-m8zkn\" (UniqueName: \"kubernetes.io/projected/3d84fdf4-9d55-49fc-9784-0b008d97a20e-kube-api-access-m8zkn\") pod \"3d84fdf4-9d55-49fc-9784-0b008d97a20e\" (UID: \"3d84fdf4-9d55-49fc-9784-0b008d97a20e\") "
Oct 31 01:56:00.848548 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:56:00.848365    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3d84fdf4-9d55-49fc-9784-0b008d97a20e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3d84fdf4-9d55-49fc-9784-0b008d97a20e-istio-envoy\") pod \"3d84fdf4-9d55-49fc-9784-0b008d97a20e\" (UID: \"3d84fdf4-9d55-49fc-9784-0b008d97a20e\") "
Oct 31 01:56:00.848548 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:56:00.848430    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3d84fdf4-9d55-49fc-9784-0b008d97a20e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d84fdf4-9d55-49fc-9784-0b008d97a20e-workload-socket\") pod \"3d84fdf4-9d55-49fc-9784-0b008d97a20e\" (UID: \"3d84fdf4-9d55-49fc-9784-0b008d97a20e\") "
Oct 31 01:56:00.848548 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:56:00.848491    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3d84fdf4-9d55-49fc-9784-0b008d97a20e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3d84fdf4-9d55-49fc-9784-0b008d97a20e-istio-podinfo\") pod \"3d84fdf4-9d55-49fc-9784-0b008d97a20e\" (UID: \"3d84fdf4-9d55-49fc-9784-0b008d97a20e\") "
Oct 31 01:56:00.848548 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:56:00.848543    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3d84fdf4-9d55-49fc-9784-0b008d97a20e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3d84fdf4-9d55-49fc-9784-0b008d97a20e-istiod-ca-cert\") pod \"3d84fdf4-9d55-49fc-9784-0b008d97a20e\" (UID: \"3d84fdf4-9d55-49fc-9784-0b008d97a20e\") "
Oct 31 01:56:00.849484 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:56:00.849458    4964 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d84fdf4-9d55-49fc-9784-0b008d97a20e-workload-certs\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:56:00.849825 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:56:00.849803    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d84fdf4-9d55-49fc-9784-0b008d97a20e-credential-socket" pod "3d84fdf4-9d55-49fc-9784-0b008d97a20e" (UID: "3d84fdf4-9d55-49fc-9784-0b008d97a20e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:56:00.850305 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:56:00.850286    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3d84fdf4-9d55-49fc-9784-0b008d97a20e-istiod-ca-cert" pod "3d84fdf4-9d55-49fc-9784-0b008d97a20e" (UID: "3d84fdf4-9d55-49fc-9784-0b008d97a20e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:56:00.850433 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:56:00.850359    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d84fdf4-9d55-49fc-9784-0b008d97a20e-workload-socket" pod "3d84fdf4-9d55-49fc-9784-0b008d97a20e" (UID: "3d84fdf4-9d55-49fc-9784-0b008d97a20e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:56:00.850550 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:56:00.850530    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d84fdf4-9d55-49fc-9784-0b008d97a20e-istio-data" pod "3d84fdf4-9d55-49fc-9784-0b008d97a20e" (UID: "3d84fdf4-9d55-49fc-9784-0b008d97a20e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:56:00.851717 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:56:00.851639    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d84fdf4-9d55-49fc-9784-0b008d97a20e-istio-token" pod "3d84fdf4-9d55-49fc-9784-0b008d97a20e" (UID: "3d84fdf4-9d55-49fc-9784-0b008d97a20e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:56:00.851717 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:56:00.851670    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3d84fdf4-9d55-49fc-9784-0b008d97a20e-istio-podinfo" pod "3d84fdf4-9d55-49fc-9784-0b008d97a20e" (UID: "3d84fdf4-9d55-49fc-9784-0b008d97a20e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 01:56:00.852263 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:56:00.852240    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d84fdf4-9d55-49fc-9784-0b008d97a20e-istio-envoy" pod "3d84fdf4-9d55-49fc-9784-0b008d97a20e" (UID: "3d84fdf4-9d55-49fc-9784-0b008d97a20e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:56:00.852732 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:56:00.852702    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d84fdf4-9d55-49fc-9784-0b008d97a20e-kube-api-access-m8zkn" pod "3d84fdf4-9d55-49fc-9784-0b008d97a20e" (UID: "3d84fdf4-9d55-49fc-9784-0b008d97a20e"). InnerVolumeSpecName "kube-api-access-m8zkn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:56:00.951160 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:56:00.951066    4964 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3d84fdf4-9d55-49fc-9784-0b008d97a20e-istio-token\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:56:00.951160 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:56:00.951097    4964 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3d84fdf4-9d55-49fc-9784-0b008d97a20e-istio-data\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:56:00.951160 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:56:00.951106    4964 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d84fdf4-9d55-49fc-9784-0b008d97a20e-credential-socket\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:56:00.951160 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:56:00.951116    4964 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m8zkn\" (UniqueName: \"kubernetes.io/projected/3d84fdf4-9d55-49fc-9784-0b008d97a20e-kube-api-access-m8zkn\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:56:00.951160 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:56:00.951125    4964 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3d84fdf4-9d55-49fc-9784-0b008d97a20e-istio-envoy\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:56:00.951160 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:56:00.951135    4964 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d84fdf4-9d55-49fc-9784-0b008d97a20e-workload-socket\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:56:00.951160 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:56:00.951143    4964 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3d84fdf4-9d55-49fc-9784-0b008d97a20e-istio-podinfo\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:56:00.951160 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:56:00.951152    4964 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3d84fdf4-9d55-49fc-9784-0b008d97a20e-istiod-ca-cert\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:56:01.484535 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:56:01.484433    4964 generic.go:356] "Generic (PLEG): container finished" podID="3d84fdf4-9d55-49fc-9784-0b008d97a20e" containerID="c8494344c3e8f9dea2ad9a895300a3f549b570e62facf3b6bb2cd85d58080748" exitCode=0
Oct 31 01:56:01.484535 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:56:01.484490    4964 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-c0daf4-openshift-default-84d68dc6f-g2858"
Oct 31 01:56:01.484535 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:56:01.484519    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-c0daf4-openshift-default-84d68dc6f-g2858" event={"ID":"3d84fdf4-9d55-49fc-9784-0b008d97a20e","Type":"ContainerDied","Data":"c8494344c3e8f9dea2ad9a895300a3f549b570e62facf3b6bb2cd85d58080748"}
Oct 31 01:56:01.484806 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:56:01.484585    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-c0daf4-openshift-default-84d68dc6f-g2858" event={"ID":"3d84fdf4-9d55-49fc-9784-0b008d97a20e","Type":"ContainerDied","Data":"6f4a6c50bc9ba0bfc73199b14c263b3da1f50df141359d1c82111a279f5f9b0d"}
Oct 31 01:56:01.484806 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:56:01.484614    4964 scope.go:122] "RemoveContainer" containerID="c8494344c3e8f9dea2ad9a895300a3f549b570e62facf3b6bb2cd85d58080748"
Oct 31 01:56:01.500040 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:56:01.499810    4964 scope.go:122] "RemoveContainer" containerID="c8494344c3e8f9dea2ad9a895300a3f549b570e62facf3b6bb2cd85d58080748"
Oct 31 01:56:01.500182 ip-10-0-133-197 kubenswrapper[4964]: E1031 01:56:01.500159    4964 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c8494344c3e8f9dea2ad9a895300a3f549b570e62facf3b6bb2cd85d58080748\": container with ID starting with c8494344c3e8f9dea2ad9a895300a3f549b570e62facf3b6bb2cd85d58080748 not found: ID does not exist" containerID="c8494344c3e8f9dea2ad9a895300a3f549b570e62facf3b6bb2cd85d58080748"
Oct 31 01:56:01.500254 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:56:01.500192    4964 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c8494344c3e8f9dea2ad9a895300a3f549b570e62facf3b6bb2cd85d58080748"} err="failed to get container status \"c8494344c3e8f9dea2ad9a895300a3f549b570e62facf3b6bb2cd85d58080748\": rpc error: code = NotFound desc = could not find container \"c8494344c3e8f9dea2ad9a895300a3f549b570e62facf3b6bb2cd85d58080748\": container with ID starting with c8494344c3e8f9dea2ad9a895300a3f549b570e62facf3b6bb2cd85d58080748 not found: ID does not exist"
Oct 31 01:56:01.691393 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:56:01.691352    4964 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-c0daf4-openshift-default-84d68dc6f-g2858"]
Oct 31 01:56:01.700375 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:56:01.699531    4964 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-a-c0daf4-openshift-default-84d68dc6f-g2858"]
Oct 31 01:56:01.782194 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:56:01.782106    4964 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3d84fdf4-9d55-49fc-9784-0b008d97a20e" path="/var/lib/kubelet/pods/3d84fdf4-9d55-49fc-9784-0b008d97a20e/volumes"
Oct 31 01:56:02.537531 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:56:02.537489    4964 patch_prober.go:29] interesting pod/e2e-praxis-b-476ce2-openshift-default-6df8c7ff5c-wqzzs container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.169:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 01:56:02.537744 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:56:02.537575    4964 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-praxis-b-476ce2-openshift-default-6df8c7ff5c-wqzzs" podUID="5a313b20-f6bb-446c-b23b-925c75a5df66" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.169:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:56:04.511420 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:56:04.511381    4964 generic.go:356] "Generic (PLEG): container finished" podID="55ee39ac-8129-4228-b0d7-a9b7841d3969" containerID="1395b33b38671b1bce5439e4857d67389b27b91047f8ad7debd534f9773edd3f" exitCode=0
Oct 31 01:56:04.511868 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:56:04.511458    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-98b70c-openshift-default-7b4cfc957-9ntnk" event={"ID":"55ee39ac-8129-4228-b0d7-a9b7841d3969","Type":"ContainerDied","Data":"1395b33b38671b1bce5439e4857d67389b27b91047f8ad7debd534f9773edd3f"}
Oct 31 01:56:04.511914 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:56:04.511893    4964 scope.go:122] "RemoveContainer" containerID="1395b33b38671b1bce5439e4857d67389b27b91047f8ad7debd534f9773edd3f"
Oct 31 01:56:04.513831 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:56:04.513799    4964 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 01:56:04.513916 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:56:04.513897    4964 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 01:56:04.513973 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:56:04.513957    4964 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 01:56:05.524464 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:56:05.524435    4964 generic.go:356] "Generic (PLEG): container finished" podID="5a313b20-f6bb-446c-b23b-925c75a5df66" containerID="a3cbedca00622e1ebaa8aae6974a651e953cb06626288052f24eed3e79af01a0" exitCode=0
Oct 31 01:56:05.525050 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:56:05.524515    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-476ce2-openshift-default-6df8c7ff5c-wqzzs" event={"ID":"5a313b20-f6bb-446c-b23b-925c75a5df66","Type":"ContainerDied","Data":"a3cbedca00622e1ebaa8aae6974a651e953cb06626288052f24eed3e79af01a0"}
Oct 31 01:56:05.525050 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:56:05.525011    4964 scope.go:122] "RemoveContainer" containerID="a3cbedca00622e1ebaa8aae6974a651e953cb06626288052f24eed3e79af01a0"
Oct 31 01:56:05.526703 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:56:05.526661    4964 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 01:56:05.526803 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:56:05.526756    4964 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 01:56:05.526863 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:56:05.526800    4964 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 01:56:05.526974 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:56:05.526957    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-98b70c-openshift-default-7b4cfc957-9ntnk" event={"ID":"55ee39ac-8129-4228-b0d7-a9b7841d3969","Type":"ContainerStarted","Data":"bd4e5278482b509af0d764b7c3d16f8d4a61ede0d0513849adf2ac1664727a5d"}
Oct 31 01:56:05.725340 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:56:05.719348    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:56:05.884145 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:56:05.884039    4964 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w0-98b70c-openshift-default-7b4cfc957-9ntnk"
Oct 31 01:56:05.885689 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:56:05.885652    4964 patch_prober.go:29] interesting pod/e2e-shared-a-w0-98b70c-openshift-default-7b4cfc957-9ntnk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.170:15021/healthz/ready\": dial tcp 10.132.0.170:15021: connect: connection refused" start-of-body=
Oct 31 01:56:05.885832 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:56:05.885728    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-98b70c-openshift-default-7b4cfc957-9ntnk" podUID="55ee39ac-8129-4228-b0d7-a9b7841d3969" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.170:15021/healthz/ready\": dial tcp 10.132.0.170:15021: connect: connection refused"
Oct 31 01:56:06.515548 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:56:06.515512    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-051514-6d5fbc88-4xc4r"
Oct 31 01:56:06.550950 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:56:06.549560    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-476ce2-openshift-default-6df8c7ff5c-wqzzs" event={"ID":"5a313b20-f6bb-446c-b23b-925c75a5df66","Type":"ContainerStarted","Data":"f0154e37e7309edbf51288e73c9ddffe9d1a6b6ec31f8f1592d51f40d4f270da"}
Oct 31 01:56:06.884140 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:56:06.884052    4964 patch_prober.go:29] interesting pod/e2e-shared-a-w0-98b70c-openshift-default-7b4cfc957-9ntnk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.170:15021/healthz/ready\": dial tcp 10.132.0.170:15021: connect: connection refused" start-of-body=
Oct 31 01:56:06.884140 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:56:06.884106    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-98b70c-openshift-default-7b4cfc957-9ntnk" podUID="55ee39ac-8129-4228-b0d7-a9b7841d3969" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.170:15021/healthz/ready\": dial tcp 10.132.0.170:15021: connect: connection refused"
Oct 31 01:56:07.211433 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:56:07.210845    4964 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-476ce2-openshift-default-6df8c7ff5c-wqzzs"
Oct 31 01:56:07.211433 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:56:07.210978    4964 patch_prober.go:29] interesting pod/e2e-praxis-b-476ce2-openshift-default-6df8c7ff5c-wqzzs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.169:15021/healthz/ready\": dial tcp 10.132.0.169:15021: connect: connection refused" start-of-body=
Oct 31 01:56:07.211433 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:56:07.211019    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-476ce2-openshift-default-6df8c7ff5c-wqzzs" podUID="5a313b20-f6bb-446c-b23b-925c75a5df66" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.169:15021/healthz/ready\": dial tcp 10.132.0.169:15021: connect: connection refused"
Oct 31 01:56:07.557432 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:56:07.557399    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w0-98b70c-openshift-default-7b4cfc957-9ntnk"
Oct 31 01:56:07.884467 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:56:07.884386    4964 patch_prober.go:29] interesting pod/e2e-shared-a-w0-98b70c-openshift-default-7b4cfc957-9ntnk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.170:15021/healthz/ready\": dial tcp 10.132.0.170:15021: connect: connection refused" start-of-body=
Oct 31 01:56:07.884467 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:56:07.884447    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-98b70c-openshift-default-7b4cfc957-9ntnk" podUID="55ee39ac-8129-4228-b0d7-a9b7841d3969" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.170:15021/healthz/ready\": dial tcp 10.132.0.170:15021: connect: connection refused"
Oct 31 01:56:08.208405 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:56:08.207782    4964 patch_prober.go:29] interesting pod/e2e-praxis-b-476ce2-openshift-default-6df8c7ff5c-wqzzs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.169:15021/healthz/ready\": dial tcp 10.132.0.169:15021: connect: connection refused" start-of-body=
Oct 31 01:56:08.208405 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:56:08.207843    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-476ce2-openshift-default-6df8c7ff5c-wqzzs" podUID="5a313b20-f6bb-446c-b23b-925c75a5df66" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.169:15021/healthz/ready\": dial tcp 10.132.0.169:15021: connect: connection refused"
Oct 31 01:56:08.883906 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:56:08.883872    4964 patch_prober.go:29] interesting pod/e2e-shared-a-w0-98b70c-openshift-default-7b4cfc957-9ntnk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.170:15021/healthz/ready\": dial tcp 10.132.0.170:15021: connect: connection refused" start-of-body=
Oct 31 01:56:08.884460 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:56:08.883932    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-98b70c-openshift-default-7b4cfc957-9ntnk" podUID="55ee39ac-8129-4228-b0d7-a9b7841d3969" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.170:15021/healthz/ready\": dial tcp 10.132.0.170:15021: connect: connection refused"
Oct 31 01:56:09.210847 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:56:09.210751    4964 patch_prober.go:29] interesting pod/e2e-praxis-b-476ce2-openshift-default-6df8c7ff5c-wqzzs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.169:15021/healthz/ready\": dial tcp 10.132.0.169:15021: connect: connection refused" start-of-body=
Oct 31 01:56:09.211112 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:56:09.211084    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-476ce2-openshift-default-6df8c7ff5c-wqzzs" podUID="5a313b20-f6bb-446c-b23b-925c75a5df66" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.169:15021/healthz/ready\": dial tcp 10.132.0.169:15021: connect: connection refused"
Oct 31 01:56:09.884687 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:56:09.884658    4964 patch_prober.go:29] interesting pod/e2e-shared-a-w0-98b70c-openshift-default-7b4cfc957-9ntnk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.170:15021/healthz/ready\": dial tcp 10.132.0.170:15021: connect: connection refused" start-of-body=
Oct 31 01:56:09.885162 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:56:09.885136    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-98b70c-openshift-default-7b4cfc957-9ntnk" podUID="55ee39ac-8129-4228-b0d7-a9b7841d3969" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.170:15021/healthz/ready\": dial tcp 10.132.0.170:15021: connect: connection refused"
Oct 31 01:56:10.209405 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:56:10.208522    4964 patch_prober.go:29] interesting pod/e2e-praxis-b-476ce2-openshift-default-6df8c7ff5c-wqzzs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.169:15021/healthz/ready\": dial tcp 10.132.0.169:15021: connect: connection refused" start-of-body=
Oct 31 01:56:10.209405 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:56:10.208582    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-476ce2-openshift-default-6df8c7ff5c-wqzzs" podUID="5a313b20-f6bb-446c-b23b-925c75a5df66" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.169:15021/healthz/ready\": dial tcp 10.132.0.169:15021: connect: connection refused"
Oct 31 01:56:10.521349 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:56:10.506400    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:56:10.953493 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:56:10.952555    4964 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w0-98b70c-openshift-default-7b4cfc957-9ntnk"
Oct 31 01:56:10.955170 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:56:10.955141    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w0-98b70c-openshift-default-7b4cfc957-9ntnk"
Oct 31 01:56:11.211781 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:56:11.211665    4964 patch_prober.go:29] interesting pod/e2e-praxis-b-476ce2-openshift-default-6df8c7ff5c-wqzzs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.169:15021/healthz/ready\": dial tcp 10.132.0.169:15021: connect: connection refused" start-of-body=
Oct 31 01:56:11.212069 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:56:11.212018    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-476ce2-openshift-default-6df8c7ff5c-wqzzs" podUID="5a313b20-f6bb-446c-b23b-925c75a5df66" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.169:15021/healthz/ready\": dial tcp 10.132.0.169:15021: connect: connection refused"
Oct 31 01:56:11.254297 ip-10-0-133-197 kubenswrapper[4964]: E1031 01:56:11.254264    4964 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q56x5" containerName="extractor"
Oct 31 01:56:12.211970 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:56:12.211939    4964 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-476ce2-openshift-default-6df8c7ff5c-wqzzs"
Oct 31 01:56:12.212521 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:56:12.212502    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-476ce2-openshift-default-6df8c7ff5c-wqzzs"
Oct 31 01:56:12.213212 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:56:12.213189    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-476ce2-openshift-default-6df8c7ff5c-wqzzs"
Oct 31 01:56:15.744692 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:56:15.744653    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:56:20.993792 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:56:20.993742    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:56:22.361266 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:56:22.361221    4964 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-cd0c98-7dd9d94dc-skdst"]
Oct 31 01:56:22.468162 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:56:22.468123    4964 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-cd0c98-747b757bc-xf68k"]
Oct 31 01:56:22.843138 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:56:22.843083    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:56:23.049938 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:56:23.049892    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:56:26.100011 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:56:26.099969    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:56:27.362433 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:56:27.362356    4964 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-b-cd0c98-7dd9d94dc-skdst" podUID="5eba908f-921b-45ce-8676-c027af587013" containerName="payload-processing" containerID="cri-o://10c4044bc5faec8e7ecc1969819dd03bbd0145eec16d7323ff277fc029994a60" gracePeriod=25
Oct 31 01:56:27.469047 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:56:27.469006    4964 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-cd0c98-747b757bc-xf68k" podUID="24c7c765-3123-429e-ad45-b95b7cf5e988" containerName="payload-pre-processing" containerID="cri-o://747402714dd7f6b363072cdb9d12517c4fb401db8046289ab2b40298015ccb0e" gracePeriod=25
Oct 31 01:56:27.691668 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:56:27.691628    4964 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-cd0c98-7dd9d94dc-skdst"
Oct 31 01:56:27.733048 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:56:27.732444    4964 generic.go:356] "Generic (PLEG): container finished" podID="5eba908f-921b-45ce-8676-c027af587013" containerID="10c4044bc5faec8e7ecc1969819dd03bbd0145eec16d7323ff277fc029994a60" exitCode=0
Oct 31 01:56:27.733048 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:56:27.732612    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-cd0c98-7dd9d94dc-skdst" event={"ID":"5eba908f-921b-45ce-8676-c027af587013","Type":"ContainerDied","Data":"10c4044bc5faec8e7ecc1969819dd03bbd0145eec16d7323ff277fc029994a60"}
Oct 31 01:56:27.733048 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:56:27.732665    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-cd0c98-7dd9d94dc-skdst" event={"ID":"5eba908f-921b-45ce-8676-c027af587013","Type":"ContainerDied","Data":"8e76102bc2629bac41a8bb56f711573b1d11be81b5462aee602b7aabb9cc90eb"}
Oct 31 01:56:27.733048 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:56:27.732696    4964 scope.go:122] "RemoveContainer" containerID="10c4044bc5faec8e7ecc1969819dd03bbd0145eec16d7323ff277fc029994a60"
Oct 31 01:56:27.733048 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:56:27.732854    4964 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-cd0c98-7dd9d94dc-skdst"
Oct 31 01:56:27.737297 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:56:27.737187    4964 generic.go:356] "Generic (PLEG): container finished" podID="24c7c765-3123-429e-ad45-b95b7cf5e988" containerID="747402714dd7f6b363072cdb9d12517c4fb401db8046289ab2b40298015ccb0e" exitCode=0
Oct 31 01:56:27.737297 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:56:27.737267    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-cd0c98-747b757bc-xf68k" event={"ID":"24c7c765-3123-429e-ad45-b95b7cf5e988","Type":"ContainerDied","Data":"747402714dd7f6b363072cdb9d12517c4fb401db8046289ab2b40298015ccb0e"}
Oct 31 01:56:27.751490 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:56:27.751358    4964 scope.go:122] "RemoveContainer" containerID="10c4044bc5faec8e7ecc1969819dd03bbd0145eec16d7323ff277fc029994a60"
Oct 31 01:56:27.751930 ip-10-0-133-197 kubenswrapper[4964]: E1031 01:56:27.751898    4964 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"10c4044bc5faec8e7ecc1969819dd03bbd0145eec16d7323ff277fc029994a60\": container with ID starting with 10c4044bc5faec8e7ecc1969819dd03bbd0145eec16d7323ff277fc029994a60 not found: ID does not exist" containerID="10c4044bc5faec8e7ecc1969819dd03bbd0145eec16d7323ff277fc029994a60"
Oct 31 01:56:27.752054 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:56:27.751940    4964 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"10c4044bc5faec8e7ecc1969819dd03bbd0145eec16d7323ff277fc029994a60"} err="failed to get container status \"10c4044bc5faec8e7ecc1969819dd03bbd0145eec16d7323ff277fc029994a60\": rpc error: code = NotFound desc = could not find container \"10c4044bc5faec8e7ecc1969819dd03bbd0145eec16d7323ff277fc029994a60\": container with ID starting with 10c4044bc5faec8e7ecc1969819dd03bbd0145eec16d7323ff277fc029994a60 not found: ID does not exist"
Oct 31 01:56:27.824265 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:56:27.824232    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5eba908f-921b-45ce-8676-c027af587013-kube-api-access-x7fgm\" (UniqueName: \"kubernetes.io/projected/5eba908f-921b-45ce-8676-c027af587013-kube-api-access-x7fgm\") pod \"5eba908f-921b-45ce-8676-c027af587013\" (UID: \"5eba908f-921b-45ce-8676-c027af587013\") "
Oct 31 01:56:27.824477 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:56:27.824413    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5eba908f-921b-45ce-8676-c027af587013-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5eba908f-921b-45ce-8676-c027af587013-plugins-config-volume\") pod \"5eba908f-921b-45ce-8676-c027af587013\" (UID: \"5eba908f-921b-45ce-8676-c027af587013\") "
Oct 31 01:56:27.826169 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:56:27.826139    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5eba908f-921b-45ce-8676-c027af587013-plugins-config-volume" pod "5eba908f-921b-45ce-8676-c027af587013" (UID: "5eba908f-921b-45ce-8676-c027af587013"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:56:27.826884 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:56:27.826821    4964 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-cd0c98-747b757bc-xf68k"
Oct 31 01:56:27.828574 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:56:27.828545    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5eba908f-921b-45ce-8676-c027af587013-kube-api-access-x7fgm" pod "5eba908f-921b-45ce-8676-c027af587013" (UID: "5eba908f-921b-45ce-8676-c027af587013"). InnerVolumeSpecName "kube-api-access-x7fgm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:56:27.926435 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:56:27.926344    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/24c7c765-3123-429e-ad45-b95b7cf5e988-kube-api-access-jxqdm\" (UniqueName: \"kubernetes.io/projected/24c7c765-3123-429e-ad45-b95b7cf5e988-kube-api-access-jxqdm\") pod \"24c7c765-3123-429e-ad45-b95b7cf5e988\" (UID: \"24c7c765-3123-429e-ad45-b95b7cf5e988\") "
Oct 31 01:56:27.926435 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:56:27.926405    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/24c7c765-3123-429e-ad45-b95b7cf5e988-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/24c7c765-3123-429e-ad45-b95b7cf5e988-plugins-config-volume\") pod \"24c7c765-3123-429e-ad45-b95b7cf5e988\" (UID: \"24c7c765-3123-429e-ad45-b95b7cf5e988\") "
Oct 31 01:56:27.926703 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:56:27.926690    4964 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x7fgm\" (UniqueName: \"kubernetes.io/projected/5eba908f-921b-45ce-8676-c027af587013-kube-api-access-x7fgm\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:56:27.926744 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:56:27.926705    4964 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5eba908f-921b-45ce-8676-c027af587013-plugins-config-volume\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:56:27.926835 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:56:27.926811    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/24c7c765-3123-429e-ad45-b95b7cf5e988-plugins-config-volume" pod "24c7c765-3123-429e-ad45-b95b7cf5e988" (UID: "24c7c765-3123-429e-ad45-b95b7cf5e988"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:56:27.928803 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:56:27.928776    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/24c7c765-3123-429e-ad45-b95b7cf5e988-kube-api-access-jxqdm" pod "24c7c765-3123-429e-ad45-b95b7cf5e988" (UID: "24c7c765-3123-429e-ad45-b95b7cf5e988"). InnerVolumeSpecName "kube-api-access-jxqdm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:56:28.028471 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:56:28.028434    4964 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jxqdm\" (UniqueName: \"kubernetes.io/projected/24c7c765-3123-429e-ad45-b95b7cf5e988-kube-api-access-jxqdm\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:56:28.028471 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:56:28.028467    4964 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/24c7c765-3123-429e-ad45-b95b7cf5e988-plugins-config-volume\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:56:28.154379 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:56:28.154344    4964 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-cd0c98-7dd9d94dc-skdst"]
Oct 31 01:56:28.159101 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:56:28.159068    4964 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-cd0c98-7dd9d94dc-skdst"]
Oct 31 01:56:28.746305 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:56:28.746272    4964 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-cd0c98-747b757bc-xf68k"
Oct 31 01:56:28.746809 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:56:28.746292    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-cd0c98-747b757bc-xf68k" event={"ID":"24c7c765-3123-429e-ad45-b95b7cf5e988","Type":"ContainerDied","Data":"86cb69f4bad4efe77802cd085c356d365cccb46a127bbdc441276dbb0346e03b"}
Oct 31 01:56:28.746809 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:56:28.746410    4964 scope.go:122] "RemoveContainer" containerID="747402714dd7f6b363072cdb9d12517c4fb401db8046289ab2b40298015ccb0e"
Oct 31 01:56:28.910370 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:56:28.910339    4964 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-cd0c98-747b757bc-xf68k"]
Oct 31 01:56:28.913815 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:56:28.913787    4964 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-cd0c98-747b757bc-xf68k"]
Oct 31 01:56:29.781158 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:56:29.781123    4964 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="24c7c765-3123-429e-ad45-b95b7cf5e988" path="/var/lib/kubelet/pods/24c7c765-3123-429e-ad45-b95b7cf5e988/volumes"
Oct 31 01:56:29.781722 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:56:29.781702    4964 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5eba908f-921b-45ce-8676-c027af587013" path="/var/lib/kubelet/pods/5eba908f-921b-45ce-8676-c027af587013/volumes"
Oct 31 01:56:30.212217 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:56:30.212125    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:56:31.398930 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:56:31.398898    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:56:36.253764 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:56:36.253728    4964 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-9e3b95-6654679bcb-dxw74"]
Oct 31 01:56:36.254645 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:56:36.254618    4964 cpu_manager.go:414] "containerMap: removing container" podUID="24c7c765-3123-429e-ad45-b95b7cf5e988" containerName="payload-pre-processing"
Oct 31 01:56:36.254645 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:56:36.254645    4964 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="24c7c765-3123-429e-ad45-b95b7cf5e988" containerName="payload-pre-processing"
Oct 31 01:56:36.254865 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:56:36.254672    4964 cpu_manager.go:414] "containerMap: removing container" podUID="3d84fdf4-9d55-49fc-9784-0b008d97a20e" containerName="istio-proxy"
Oct 31 01:56:36.254865 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:56:36.254680    4964 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3d84fdf4-9d55-49fc-9784-0b008d97a20e" containerName="istio-proxy"
Oct 31 01:56:36.254865 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:56:36.254695    4964 cpu_manager.go:414] "containerMap: removing container" podUID="5eba908f-921b-45ce-8676-c027af587013" containerName="payload-processing"
Oct 31 01:56:36.254865 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:56:36.254702    4964 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5eba908f-921b-45ce-8676-c027af587013" containerName="payload-processing"
Oct 31 01:56:36.254865 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:56:36.254856    4964 memory_manager.go:365] "RemoveStaleState removing state" podUID="3d84fdf4-9d55-49fc-9784-0b008d97a20e" containerName="istio-proxy"
Oct 31 01:56:36.255130 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:56:36.254872    4964 memory_manager.go:365] "RemoveStaleState removing state" podUID="5eba908f-921b-45ce-8676-c027af587013" containerName="payload-processing"
Oct 31 01:56:36.255130 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:56:36.254897    4964 memory_manager.go:365] "RemoveStaleState removing state" podUID="24c7c765-3123-429e-ad45-b95b7cf5e988" containerName="payload-pre-processing"
Oct 31 01:56:36.255130 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:56:36.255045    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:56:36.257945 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:56:36.257929    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-9e3b95-6654679bcb-dxw74"
Oct 31 01:56:36.346704 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:56:36.346664    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r4n64\" (UniqueName: \"kubernetes.io/projected/081cab7b-aa01-4d09-b244-92855e9b0249-kube-api-access-r4n64\") pod \"maas-api-e2e-shared-a-w2-9e3b95-6654679bcb-dxw74\" (UID: \"081cab7b-aa01-4d09-b244-92855e9b0249\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-9e3b95-6654679bcb-dxw74"
Oct 31 01:56:36.346911 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:56:36.346776    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/081cab7b-aa01-4d09-b244-92855e9b0249-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-9e3b95-6654679bcb-dxw74\" (UID: \"081cab7b-aa01-4d09-b244-92855e9b0249\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-9e3b95-6654679bcb-dxw74"
Oct 31 01:56:36.346984 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:56:36.346969    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/081cab7b-aa01-4d09-b244-92855e9b0249-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-9e3b95-6654679bcb-dxw74\" (UID: \"081cab7b-aa01-4d09-b244-92855e9b0249\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-9e3b95-6654679bcb-dxw74"
Oct 31 01:56:36.347190 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:56:36.347153    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/081cab7b-aa01-4d09-b244-92855e9b0249-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-9e3b95-6654679bcb-dxw74\" (UID: \"081cab7b-aa01-4d09-b244-92855e9b0249\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-9e3b95-6654679bcb-dxw74"
Oct 31 01:56:36.359659 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:56:36.359632    4964 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w2-9e3b95-serving-cert\""
Oct 31 01:56:36.363439 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:56:36.363411    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-9e3b95-6654679bcb-dxw74"]
Oct 31 01:56:36.447945 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:56:36.447917    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/081cab7b-aa01-4d09-b244-92855e9b0249-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-9e3b95-6654679bcb-dxw74\" (UID: \"081cab7b-aa01-4d09-b244-92855e9b0249\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-9e3b95-6654679bcb-dxw74"
Oct 31 01:56:36.448142 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:56:36.448010    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/081cab7b-aa01-4d09-b244-92855e9b0249-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-9e3b95-6654679bcb-dxw74\" (UID: \"081cab7b-aa01-4d09-b244-92855e9b0249\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-9e3b95-6654679bcb-dxw74"
Oct 31 01:56:36.448142 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:56:36.448056    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r4n64\" (UniqueName: \"kubernetes.io/projected/081cab7b-aa01-4d09-b244-92855e9b0249-kube-api-access-r4n64\") pod \"maas-api-e2e-shared-a-w2-9e3b95-6654679bcb-dxw74\" (UID: \"081cab7b-aa01-4d09-b244-92855e9b0249\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-9e3b95-6654679bcb-dxw74"
Oct 31 01:56:36.448142 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:56:36.448080    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/081cab7b-aa01-4d09-b244-92855e9b0249-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-9e3b95-6654679bcb-dxw74\" (UID: \"081cab7b-aa01-4d09-b244-92855e9b0249\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-9e3b95-6654679bcb-dxw74"
Oct 31 01:56:36.448379 ip-10-0-133-197 kubenswrapper[4964]: E1031 01:56:36.448231    4964 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-9e3b95-serving-cert: secret "maas-api-e2e-shared-a-w2-9e3b95-serving-cert" not found
Oct 31 01:56:36.448379 ip-10-0-133-197 kubenswrapper[4964]: E1031 01:56:36.448286    4964 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/081cab7b-aa01-4d09-b244-92855e9b0249-maas-api-tls podName:081cab7b-aa01-4d09-b244-92855e9b0249 nodeName:}" failed. No retries permitted until 2026-10-31 01:56:36.948269606 +0000 UTC m=+3165.424835981 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/081cab7b-aa01-4d09-b244-92855e9b0249-maas-api-tls") pod "maas-api-e2e-shared-a-w2-9e3b95-6654679bcb-dxw74" (UID: "081cab7b-aa01-4d09-b244-92855e9b0249") : secret "maas-api-e2e-shared-a-w2-9e3b95-serving-cert" not found
Oct 31 01:56:36.448699 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:56:36.448671    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/081cab7b-aa01-4d09-b244-92855e9b0249-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-9e3b95-6654679bcb-dxw74\" (UID: \"081cab7b-aa01-4d09-b244-92855e9b0249\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-9e3b95-6654679bcb-dxw74"
Oct 31 01:56:36.450833 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:56:36.450810    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/081cab7b-aa01-4d09-b244-92855e9b0249-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-9e3b95-6654679bcb-dxw74\" (UID: \"081cab7b-aa01-4d09-b244-92855e9b0249\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-9e3b95-6654679bcb-dxw74"
Oct 31 01:56:36.493081 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:56:36.493031    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:56:36.512194 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:56:36.512096    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r4n64\" (UniqueName: \"kubernetes.io/projected/081cab7b-aa01-4d09-b244-92855e9b0249-kube-api-access-r4n64\") pod \"maas-api-e2e-shared-a-w2-9e3b95-6654679bcb-dxw74\" (UID: \"081cab7b-aa01-4d09-b244-92855e9b0249\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-9e3b95-6654679bcb-dxw74"
Oct 31 01:56:36.668389 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:56:36.668351    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:56:36.842111 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:56:36.842023    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:56:36.954266 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:56:36.954229    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/081cab7b-aa01-4d09-b244-92855e9b0249-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-9e3b95-6654679bcb-dxw74\" (UID: \"081cab7b-aa01-4d09-b244-92855e9b0249\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-9e3b95-6654679bcb-dxw74"
Oct 31 01:56:36.957088 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:56:36.957064    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/081cab7b-aa01-4d09-b244-92855e9b0249-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-9e3b95-6654679bcb-dxw74\" (UID: \"081cab7b-aa01-4d09-b244-92855e9b0249\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-9e3b95-6654679bcb-dxw74"
Oct 31 01:56:37.171726 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:56:37.171637    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-9e3b95-6654679bcb-dxw74"
Oct 31 01:56:37.552961 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:56:37.552934    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-9e3b95-6654679bcb-dxw74"]
Oct 31 01:56:37.554030 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:56:37.554004    4964 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod081cab7b_aa01_4d09_b244_92855e9b0249.slice/crio-115ef4999c79691725b1b0ae9f3d541575e2984089453cb4efbd16b5f2b9f29c WatchSource:0}: Error finding container 115ef4999c79691725b1b0ae9f3d541575e2984089453cb4efbd16b5f2b9f29c: Status 404 returned error can't find the container with id 115ef4999c79691725b1b0ae9f3d541575e2984089453cb4efbd16b5f2b9f29c
Oct 31 01:56:37.827660 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:56:37.827572    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-9e3b95-6654679bcb-dxw74" event={"ID":"081cab7b-aa01-4d09-b244-92855e9b0249","Type":"ContainerStarted","Data":"115ef4999c79691725b1b0ae9f3d541575e2984089453cb4efbd16b5f2b9f29c"}
Oct 31 01:56:38.836812 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:56:38.836773    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-9e3b95-6654679bcb-dxw74" event={"ID":"081cab7b-aa01-4d09-b244-92855e9b0249","Type":"ContainerStarted","Data":"fecdedb8917aa829ca6b252f204f180caa58035295f108d71aac43d23eb0e1f4"}
Oct 31 01:56:38.837362 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:56:38.836859    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-9e3b95-6654679bcb-dxw74"
Oct 31 01:56:38.954608 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:56:38.954549    4964 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-9e3b95-6654679bcb-dxw74" podStartSLOduration=2.117456157 podStartE2EDuration="2.954534714s" totalImagesPullingTime="837.078557ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:56:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:56:38.951718255 +0000 UTC m=+3167.428284640" watchObservedRunningTime="2026-10-31 01:56:38.954534714 +0000 UTC m=+3167.431101135"
Oct 31 01:56:40.751427 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:56:40.751396    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:56:41.239743 ip-10-0-133-197 kubenswrapper[4964]: E1031 01:56:41.239706    4964 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q56x5" containerName="extractor"
Oct 31 01:56:41.875397 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:56:41.875305    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:56:42.325452 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:56:42.325415    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:56:44.849870 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:56:44.849834    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-9e3b95-6654679bcb-dxw74"
Oct 31 01:56:46.996972 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:56:46.996853    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:56:51.923059 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:56:51.923026    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:56:53.130835 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:56:53.130799    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:56:53.649161 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:56:53.649125    4964 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-3b7279-667b9df887-n49j9"]
Oct 31 01:56:53.758536 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:56:53.758500    4964 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-3b7279-6884d85fbc-fhmqm"]
Oct 31 01:56:54.043021 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:56:54.042990    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:56:54.231545 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:56:54.231509    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:56:56.368202 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:56:56.368144    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:56:56.909657 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:56:56.909612    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:56:57.106244 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:56:57.106209    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:56:58.649692 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:56:58.649633    4964 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-a-3b7279-667b9df887-n49j9" podUID="b6ecd194-e6e7-4b66-bdc3-b6166efd0045" containerName="payload-processing" containerID="cri-o://e0f96b8b6beb9cc8d29d68774840a3506ee664529628efe0cb47af80b0274ca0" gracePeriod=25
Oct 31 01:56:58.759850 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:56:58.759814    4964 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-3b7279-6884d85fbc-fhmqm" podUID="b1297c54-10e4-4992-9ae3-4d8d628dd0a2" containerName="payload-pre-processing" containerID="cri-o://f0ae72b88c1eefa98fc31fa16c5ff8bcec8cada3a1c5ee2d945763e41f028afc" gracePeriod=25
Oct 31 01:56:58.952194 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:56:58.952169    4964 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-3b7279-667b9df887-n49j9"
Oct 31 01:56:58.999694 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:56:58.999653    4964 generic.go:356] "Generic (PLEG): container finished" podID="b1297c54-10e4-4992-9ae3-4d8d628dd0a2" containerID="f0ae72b88c1eefa98fc31fa16c5ff8bcec8cada3a1c5ee2d945763e41f028afc" exitCode=0
Oct 31 01:56:58.999878 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:56:58.999798    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-3b7279-6884d85fbc-fhmqm" event={"ID":"b1297c54-10e4-4992-9ae3-4d8d628dd0a2","Type":"ContainerDied","Data":"f0ae72b88c1eefa98fc31fa16c5ff8bcec8cada3a1c5ee2d945763e41f028afc"}
Oct 31 01:56:59.001568 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:56:59.001536    4964 generic.go:356] "Generic (PLEG): container finished" podID="b6ecd194-e6e7-4b66-bdc3-b6166efd0045" containerID="e0f96b8b6beb9cc8d29d68774840a3506ee664529628efe0cb47af80b0274ca0" exitCode=0
Oct 31 01:56:59.001781 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:56:59.001761    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-3b7279-667b9df887-n49j9" event={"ID":"b6ecd194-e6e7-4b66-bdc3-b6166efd0045","Type":"ContainerDied","Data":"e0f96b8b6beb9cc8d29d68774840a3506ee664529628efe0cb47af80b0274ca0"}
Oct 31 01:56:59.001854 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:56:59.001818    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-3b7279-667b9df887-n49j9" event={"ID":"b6ecd194-e6e7-4b66-bdc3-b6166efd0045","Type":"ContainerDied","Data":"b6be67d466b7ae6000f4f31ce46c098d2400b0301307252aeb377f816e8ef44d"}
Oct 31 01:56:59.001854 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:56:59.001849    4964 scope.go:122] "RemoveContainer" containerID="e0f96b8b6beb9cc8d29d68774840a3506ee664529628efe0cb47af80b0274ca0"
Oct 31 01:56:59.002024 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:56:59.002005    4964 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-3b7279-667b9df887-n49j9"
Oct 31 01:56:59.018188 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:56:59.017950    4964 scope.go:122] "RemoveContainer" containerID="e0f96b8b6beb9cc8d29d68774840a3506ee664529628efe0cb47af80b0274ca0"
Oct 31 01:56:59.018490 ip-10-0-133-197 kubenswrapper[4964]: E1031 01:56:59.018401    4964 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e0f96b8b6beb9cc8d29d68774840a3506ee664529628efe0cb47af80b0274ca0\": container with ID starting with e0f96b8b6beb9cc8d29d68774840a3506ee664529628efe0cb47af80b0274ca0 not found: ID does not exist" containerID="e0f96b8b6beb9cc8d29d68774840a3506ee664529628efe0cb47af80b0274ca0"
Oct 31 01:56:59.018490 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:56:59.018436    4964 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e0f96b8b6beb9cc8d29d68774840a3506ee664529628efe0cb47af80b0274ca0"} err="failed to get container status \"e0f96b8b6beb9cc8d29d68774840a3506ee664529628efe0cb47af80b0274ca0\": rpc error: code = NotFound desc = could not find container \"e0f96b8b6beb9cc8d29d68774840a3506ee664529628efe0cb47af80b0274ca0\": container with ID starting with e0f96b8b6beb9cc8d29d68774840a3506ee664529628efe0cb47af80b0274ca0 not found: ID does not exist"
Oct 31 01:56:59.062717 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:56:59.062691    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b6ecd194-e6e7-4b66-bdc3-b6166efd0045-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b6ecd194-e6e7-4b66-bdc3-b6166efd0045-plugins-config-volume\") pod \"b6ecd194-e6e7-4b66-bdc3-b6166efd0045\" (UID: \"b6ecd194-e6e7-4b66-bdc3-b6166efd0045\") "
Oct 31 01:56:59.063026 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:56:59.063009    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b6ecd194-e6e7-4b66-bdc3-b6166efd0045-kube-api-access-6fb2f\" (UniqueName: \"kubernetes.io/projected/b6ecd194-e6e7-4b66-bdc3-b6166efd0045-kube-api-access-6fb2f\") pod \"b6ecd194-e6e7-4b66-bdc3-b6166efd0045\" (UID: \"b6ecd194-e6e7-4b66-bdc3-b6166efd0045\") "
Oct 31 01:56:59.063026 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:56:59.063015    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b6ecd194-e6e7-4b66-bdc3-b6166efd0045-plugins-config-volume" pod "b6ecd194-e6e7-4b66-bdc3-b6166efd0045" (UID: "b6ecd194-e6e7-4b66-bdc3-b6166efd0045"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:56:59.063622 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:56:59.063604    4964 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b6ecd194-e6e7-4b66-bdc3-b6166efd0045-plugins-config-volume\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:56:59.065144 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:56:59.065125    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b6ecd194-e6e7-4b66-bdc3-b6166efd0045-kube-api-access-6fb2f" pod "b6ecd194-e6e7-4b66-bdc3-b6166efd0045" (UID: "b6ecd194-e6e7-4b66-bdc3-b6166efd0045"). InnerVolumeSpecName "kube-api-access-6fb2f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:56:59.080302 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:56:59.080277    4964 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-3b7279-6884d85fbc-fhmqm"
Oct 31 01:56:59.164922 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:56:59.164838    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b1297c54-10e4-4992-9ae3-4d8d628dd0a2-kube-api-access-2j222\" (UniqueName: \"kubernetes.io/projected/b1297c54-10e4-4992-9ae3-4d8d628dd0a2-kube-api-access-2j222\") pod \"b1297c54-10e4-4992-9ae3-4d8d628dd0a2\" (UID: \"b1297c54-10e4-4992-9ae3-4d8d628dd0a2\") "
Oct 31 01:56:59.164922 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:56:59.164910    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b1297c54-10e4-4992-9ae3-4d8d628dd0a2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b1297c54-10e4-4992-9ae3-4d8d628dd0a2-plugins-config-volume\") pod \"b1297c54-10e4-4992-9ae3-4d8d628dd0a2\" (UID: \"b1297c54-10e4-4992-9ae3-4d8d628dd0a2\") "
Oct 31 01:56:59.165269 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:56:59.165245    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b1297c54-10e4-4992-9ae3-4d8d628dd0a2-plugins-config-volume" pod "b1297c54-10e4-4992-9ae3-4d8d628dd0a2" (UID: "b1297c54-10e4-4992-9ae3-4d8d628dd0a2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:56:59.165608 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:56:59.165591    4964 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6fb2f\" (UniqueName: \"kubernetes.io/projected/b6ecd194-e6e7-4b66-bdc3-b6166efd0045-kube-api-access-6fb2f\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:56:59.165663 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:56:59.165611    4964 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b1297c54-10e4-4992-9ae3-4d8d628dd0a2-plugins-config-volume\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:56:59.167062 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:56:59.167043    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b1297c54-10e4-4992-9ae3-4d8d628dd0a2-kube-api-access-2j222" pod "b1297c54-10e4-4992-9ae3-4d8d628dd0a2" (UID: "b1297c54-10e4-4992-9ae3-4d8d628dd0a2"). InnerVolumeSpecName "kube-api-access-2j222". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:56:59.266750 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:56:59.266715    4964 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2j222\" (UniqueName: \"kubernetes.io/projected/b1297c54-10e4-4992-9ae3-4d8d628dd0a2-kube-api-access-2j222\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:56:59.429123 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:56:59.429030    4964 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-3b7279-667b9df887-n49j9"]
Oct 31 01:56:59.432576 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:56:59.432546    4964 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-3b7279-667b9df887-n49j9"]
Oct 31 01:56:59.777989 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:56:59.777958    4964 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b6ecd194-e6e7-4b66-bdc3-b6166efd0045" path="/var/lib/kubelet/pods/b6ecd194-e6e7-4b66-bdc3-b6166efd0045/volumes"
Oct 31 01:57:00.011230 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:57:00.011204    4964 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-3b7279-6884d85fbc-fhmqm"
Oct 31 01:57:00.011458 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:57:00.011232    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-3b7279-6884d85fbc-fhmqm" event={"ID":"b1297c54-10e4-4992-9ae3-4d8d628dd0a2","Type":"ContainerDied","Data":"7f1f02dcb0a2b057a56b3c4dc84afd371bbf338ebf966008e6d0ea5a20c8cc32"}
Oct 31 01:57:00.011458 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:57:00.011301    4964 scope.go:122] "RemoveContainer" containerID="f0ae72b88c1eefa98fc31fa16c5ff8bcec8cada3a1c5ee2d945763e41f028afc"
Oct 31 01:57:00.177138 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:57:00.177026    4964 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-3b7279-6884d85fbc-fhmqm"]
Oct 31 01:57:00.180950 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:57:00.180925    4964 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-3b7279-6884d85fbc-fhmqm"]
Oct 31 01:57:00.226600 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:57:00.226562    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:57:00.400700 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:57:00.400665    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:57:01.780282 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:57:01.780246    4964 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b1297c54-10e4-4992-9ae3-4d8d628dd0a2" path="/var/lib/kubelet/pods/b1297c54-10e4-4992-9ae3-4d8d628dd0a2/volumes"
Oct 31 01:57:01.961961 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:57:01.961928    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:57:02.843576 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:57:02.843545    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:57:03.150757 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:57:03.150672    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:57:03.346659 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:57:03.346623    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:57:04.542212 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:57:04.542177    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:57:05.269073 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:57:05.269042    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:57:06.050391 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:57:06.050350    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:57:06.278612 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:57:06.278575    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:57:07.426598 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:57:07.426561    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:57:11.238804 ip-10-0-133-197 kubenswrapper[4964]: E1031 01:57:11.238769    4964 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q56x5" containerName="extractor"
Oct 31 01:57:11.923272 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:57:11.923245    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:57:13.759532 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:57:13.759492    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:57:14.000359 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:57:14.000303    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:57:14.184153 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:57:14.184057    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:57:14.510797 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:57:14.510769    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:57:14.839189 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:57:14.839101    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:57:15.005070 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:57:15.005030    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:57:18.938705 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:57:18.938670    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:57:20.504511 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:57:20.504471    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:57:20.733555 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:57:20.733521    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:57:20.928400 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:57:20.925075    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:57:23.531413 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:57:23.531375    4964 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-3b7279-7f85db968c-9qfcc"]
Oct 31 01:57:23.531881 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:57:23.531567    4964 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-3b7279-7f85db968c-9qfcc" podUID="3527b972-c696-4103-a8ce-434fddd6e167" containerName="maas-api" containerID="cri-o://8996522284a6957f7ba467e16dc2ea4b9602994f4868448d5e3058145d08625f" gracePeriod=30
Oct 31 01:57:23.675766 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:57:23.675727    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:57:23.825580 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:57:23.825550    4964 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-3b7279-7f85db968c-9qfcc"
Oct 31 01:57:23.910485 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:57:23.910450    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3527b972-c696-4103-a8ce-434fddd6e167-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3527b972-c696-4103-a8ce-434fddd6e167-openshift-service-ca\") pod \"3527b972-c696-4103-a8ce-434fddd6e167\" (UID: \"3527b972-c696-4103-a8ce-434fddd6e167\") "
Oct 31 01:57:23.910677 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:57:23.910585    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3527b972-c696-4103-a8ce-434fddd6e167-kube-api-access-mmb9d\" (UniqueName: \"kubernetes.io/projected/3527b972-c696-4103-a8ce-434fddd6e167-kube-api-access-mmb9d\") pod \"3527b972-c696-4103-a8ce-434fddd6e167\" (UID: \"3527b972-c696-4103-a8ce-434fddd6e167\") "
Oct 31 01:57:23.910677 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:57:23.910619    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3527b972-c696-4103-a8ce-434fddd6e167-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3527b972-c696-4103-a8ce-434fddd6e167-maas-api-tls\") pod \"3527b972-c696-4103-a8ce-434fddd6e167\" (UID: \"3527b972-c696-4103-a8ce-434fddd6e167\") "
Oct 31 01:57:23.910812 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:57:23.910677    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3527b972-c696-4103-a8ce-434fddd6e167-metrics-certs\" (UniqueName: \"kubernetes.io/secret/3527b972-c696-4103-a8ce-434fddd6e167-metrics-certs\") pod \"3527b972-c696-4103-a8ce-434fddd6e167\" (UID: \"3527b972-c696-4103-a8ce-434fddd6e167\") "
Oct 31 01:57:23.910871 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:57:23.910817    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3527b972-c696-4103-a8ce-434fddd6e167-openshift-service-ca" pod "3527b972-c696-4103-a8ce-434fddd6e167" (UID: "3527b972-c696-4103-a8ce-434fddd6e167"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:57:23.911405 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:57:23.911366    4964 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3527b972-c696-4103-a8ce-434fddd6e167-openshift-service-ca\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:57:23.913297 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:57:23.913260    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3527b972-c696-4103-a8ce-434fddd6e167-kube-api-access-mmb9d" pod "3527b972-c696-4103-a8ce-434fddd6e167" (UID: "3527b972-c696-4103-a8ce-434fddd6e167"). InnerVolumeSpecName "kube-api-access-mmb9d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:57:23.913443 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:57:23.913295    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3527b972-c696-4103-a8ce-434fddd6e167-metrics-certs" pod "3527b972-c696-4103-a8ce-434fddd6e167" (UID: "3527b972-c696-4103-a8ce-434fddd6e167"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:57:23.913544 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:57:23.913521    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3527b972-c696-4103-a8ce-434fddd6e167-maas-api-tls" pod "3527b972-c696-4103-a8ce-434fddd6e167" (UID: "3527b972-c696-4103-a8ce-434fddd6e167"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:57:24.012534 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:57:24.012492    4964 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3527b972-c696-4103-a8ce-434fddd6e167-metrics-certs\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:57:24.012534 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:57:24.012523    4964 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mmb9d\" (UniqueName: \"kubernetes.io/projected/3527b972-c696-4103-a8ce-434fddd6e167-kube-api-access-mmb9d\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:57:24.012534 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:57:24.012534    4964 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3527b972-c696-4103-a8ce-434fddd6e167-maas-api-tls\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:57:24.191606 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:57:24.191518    4964 generic.go:356] "Generic (PLEG): container finished" podID="3527b972-c696-4103-a8ce-434fddd6e167" containerID="8996522284a6957f7ba467e16dc2ea4b9602994f4868448d5e3058145d08625f" exitCode=0
Oct 31 01:57:24.191606 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:57:24.191560    4964 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-3b7279-7f85db968c-9qfcc"
Oct 31 01:57:24.191606 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:57:24.191597    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-3b7279-7f85db968c-9qfcc" event={"ID":"3527b972-c696-4103-a8ce-434fddd6e167","Type":"ContainerDied","Data":"8996522284a6957f7ba467e16dc2ea4b9602994f4868448d5e3058145d08625f"}
Oct 31 01:57:24.191899 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:57:24.191654    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-3b7279-7f85db968c-9qfcc" event={"ID":"3527b972-c696-4103-a8ce-434fddd6e167","Type":"ContainerDied","Data":"5ec927a29b4b7f56ee7bb2de918432c21e47ec3a1e8fff3ba43b7e4f2dba4eda"}
Oct 31 01:57:24.191899 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:57:24.191682    4964 scope.go:122] "RemoveContainer" containerID="8996522284a6957f7ba467e16dc2ea4b9602994f4868448d5e3058145d08625f"
Oct 31 01:57:24.202977 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:57:24.202955    4964 scope.go:122] "RemoveContainer" containerID="8996522284a6957f7ba467e16dc2ea4b9602994f4868448d5e3058145d08625f"
Oct 31 01:57:24.203247 ip-10-0-133-197 kubenswrapper[4964]: E1031 01:57:24.203228    4964 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8996522284a6957f7ba467e16dc2ea4b9602994f4868448d5e3058145d08625f\": container with ID starting with 8996522284a6957f7ba467e16dc2ea4b9602994f4868448d5e3058145d08625f not found: ID does not exist" containerID="8996522284a6957f7ba467e16dc2ea4b9602994f4868448d5e3058145d08625f"
Oct 31 01:57:24.203304 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:57:24.203256    4964 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8996522284a6957f7ba467e16dc2ea4b9602994f4868448d5e3058145d08625f"} err="failed to get container status \"8996522284a6957f7ba467e16dc2ea4b9602994f4868448d5e3058145d08625f\": rpc error: code = NotFound desc = could not find container \"8996522284a6957f7ba467e16dc2ea4b9602994f4868448d5e3058145d08625f\": container with ID starting with 8996522284a6957f7ba467e16dc2ea4b9602994f4868448d5e3058145d08625f not found: ID does not exist"
Oct 31 01:57:24.355161 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:57:24.355130    4964 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-3b7279-7f85db968c-9qfcc"]
Oct 31 01:57:24.404202 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:57:24.404170    4964 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-3b7279-7f85db968c-9qfcc"]
Oct 31 01:57:25.215098 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:57:25.215058    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:57:25.440936 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:57:25.440902    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:57:25.697546 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:57:25.697517    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:57:25.777564 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:57:25.777531    4964 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3527b972-c696-4103-a8ce-434fddd6e167" path="/var/lib/kubelet/pods/3527b972-c696-4103-a8ce-434fddd6e167/volumes"
Oct 31 01:57:25.798997 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:57:25.798971    4964 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w3-c4d438-openshift-default-6d9d56f475-c995r"]
Oct 31 01:57:25.799630 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:57:25.799614    4964 cpu_manager.go:414] "containerMap: removing container" podUID="b6ecd194-e6e7-4b66-bdc3-b6166efd0045" containerName="payload-processing"
Oct 31 01:57:25.799676 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:57:25.799633    4964 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b6ecd194-e6e7-4b66-bdc3-b6166efd0045" containerName="payload-processing"
Oct 31 01:57:25.799676 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:57:25.799654    4964 cpu_manager.go:414] "containerMap: removing container" podUID="3527b972-c696-4103-a8ce-434fddd6e167" containerName="maas-api"
Oct 31 01:57:25.799676 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:57:25.799661    4964 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3527b972-c696-4103-a8ce-434fddd6e167" containerName="maas-api"
Oct 31 01:57:25.799784 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:57:25.799678    4964 cpu_manager.go:414] "containerMap: removing container" podUID="b1297c54-10e4-4992-9ae3-4d8d628dd0a2" containerName="payload-pre-processing"
Oct 31 01:57:25.799784 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:57:25.799683    4964 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b1297c54-10e4-4992-9ae3-4d8d628dd0a2" containerName="payload-pre-processing"
Oct 31 01:57:25.799891 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:57:25.799789    4964 memory_manager.go:365] "RemoveStaleState removing state" podUID="b6ecd194-e6e7-4b66-bdc3-b6166efd0045" containerName="payload-processing"
Oct 31 01:57:25.799891 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:57:25.799800    4964 memory_manager.go:365] "RemoveStaleState removing state" podUID="3527b972-c696-4103-a8ce-434fddd6e167" containerName="maas-api"
Oct 31 01:57:25.799891 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:57:25.799807    4964 memory_manager.go:365] "RemoveStaleState removing state" podUID="b1297c54-10e4-4992-9ae3-4d8d628dd0a2" containerName="payload-pre-processing"
Oct 31 01:57:25.804625 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:57:25.804606    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-c4d438-openshift-default-6d9d56f475-c995r"
Oct 31 01:57:25.886667 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:57:25.886636    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-c4d438-openshift-default-6d9d56f475-c995r"]
Oct 31 01:57:25.926131 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:57:25.926103    4964 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w3-c4d438-openshift-default-dockercfg-n8945\""
Oct 31 01:57:25.927727 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:57:25.927701    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:57:25.943244 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:57:25.943207    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ec66b87-7d07-45fd-8469-e1bd2c83f651-credential-socket\") pod \"e2e-shared-b-w3-c4d438-openshift-default-6d9d56f475-c995r\" (UID: \"1ec66b87-7d07-45fd-8469-e1bd2c83f651\") " pod="openshift-ingress/e2e-shared-b-w3-c4d438-openshift-default-6d9d56f475-c995r"
Oct 31 01:57:25.943516 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:57:25.943493    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1ec66b87-7d07-45fd-8469-e1bd2c83f651-istio-token\") pod \"e2e-shared-b-w3-c4d438-openshift-default-6d9d56f475-c995r\" (UID: \"1ec66b87-7d07-45fd-8469-e1bd2c83f651\") " pod="openshift-ingress/e2e-shared-b-w3-c4d438-openshift-default-6d9d56f475-c995r"
Oct 31 01:57:25.943694 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:57:25.943670    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1ec66b87-7d07-45fd-8469-e1bd2c83f651-istio-data\") pod \"e2e-shared-b-w3-c4d438-openshift-default-6d9d56f475-c995r\" (UID: \"1ec66b87-7d07-45fd-8469-e1bd2c83f651\") " pod="openshift-ingress/e2e-shared-b-w3-c4d438-openshift-default-6d9d56f475-c995r"
Oct 31 01:57:25.943902 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:57:25.943793    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8pwdt\" (UniqueName: \"kubernetes.io/projected/1ec66b87-7d07-45fd-8469-e1bd2c83f651-kube-api-access-8pwdt\") pod \"e2e-shared-b-w3-c4d438-openshift-default-6d9d56f475-c995r\" (UID: \"1ec66b87-7d07-45fd-8469-e1bd2c83f651\") " pod="openshift-ingress/e2e-shared-b-w3-c4d438-openshift-default-6d9d56f475-c995r"
Oct 31 01:57:25.944030 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:57:25.944008    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1ec66b87-7d07-45fd-8469-e1bd2c83f651-istio-envoy\") pod \"e2e-shared-b-w3-c4d438-openshift-default-6d9d56f475-c995r\" (UID: \"1ec66b87-7d07-45fd-8469-e1bd2c83f651\") " pod="openshift-ingress/e2e-shared-b-w3-c4d438-openshift-default-6d9d56f475-c995r"
Oct 31 01:57:25.944176 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:57:25.944130    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1ec66b87-7d07-45fd-8469-e1bd2c83f651-istio-podinfo\") pod \"e2e-shared-b-w3-c4d438-openshift-default-6d9d56f475-c995r\" (UID: \"1ec66b87-7d07-45fd-8469-e1bd2c83f651\") " pod="openshift-ingress/e2e-shared-b-w3-c4d438-openshift-default-6d9d56f475-c995r"
Oct 31 01:57:25.944254 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:57:25.944244    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1ec66b87-7d07-45fd-8469-e1bd2c83f651-workload-certs\") pod \"e2e-shared-b-w3-c4d438-openshift-default-6d9d56f475-c995r\" (UID: \"1ec66b87-7d07-45fd-8469-e1bd2c83f651\") " pod="openshift-ingress/e2e-shared-b-w3-c4d438-openshift-default-6d9d56f475-c995r"
Oct 31 01:57:25.944426 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:57:25.944399    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1ec66b87-7d07-45fd-8469-e1bd2c83f651-istiod-ca-cert\") pod \"e2e-shared-b-w3-c4d438-openshift-default-6d9d56f475-c995r\" (UID: \"1ec66b87-7d07-45fd-8469-e1bd2c83f651\") " pod="openshift-ingress/e2e-shared-b-w3-c4d438-openshift-default-6d9d56f475-c995r"
Oct 31 01:57:25.944526 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:57:25.944473    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ec66b87-7d07-45fd-8469-e1bd2c83f651-workload-socket\") pod \"e2e-shared-b-w3-c4d438-openshift-default-6d9d56f475-c995r\" (UID: \"1ec66b87-7d07-45fd-8469-e1bd2c83f651\") " pod="openshift-ingress/e2e-shared-b-w3-c4d438-openshift-default-6d9d56f475-c995r"
Oct 31 01:57:26.046179 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:57:26.046145    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ec66b87-7d07-45fd-8469-e1bd2c83f651-credential-socket\") pod \"e2e-shared-b-w3-c4d438-openshift-default-6d9d56f475-c995r\" (UID: \"1ec66b87-7d07-45fd-8469-e1bd2c83f651\") " pod="openshift-ingress/e2e-shared-b-w3-c4d438-openshift-default-6d9d56f475-c995r"
Oct 31 01:57:26.046442 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:57:26.046195    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1ec66b87-7d07-45fd-8469-e1bd2c83f651-istio-token\") pod \"e2e-shared-b-w3-c4d438-openshift-default-6d9d56f475-c995r\" (UID: \"1ec66b87-7d07-45fd-8469-e1bd2c83f651\") " pod="openshift-ingress/e2e-shared-b-w3-c4d438-openshift-default-6d9d56f475-c995r"
Oct 31 01:57:26.046442 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:57:26.046228    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1ec66b87-7d07-45fd-8469-e1bd2c83f651-istio-data\") pod \"e2e-shared-b-w3-c4d438-openshift-default-6d9d56f475-c995r\" (UID: \"1ec66b87-7d07-45fd-8469-e1bd2c83f651\") " pod="openshift-ingress/e2e-shared-b-w3-c4d438-openshift-default-6d9d56f475-c995r"
Oct 31 01:57:26.046442 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:57:26.046305    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8pwdt\" (UniqueName: \"kubernetes.io/projected/1ec66b87-7d07-45fd-8469-e1bd2c83f651-kube-api-access-8pwdt\") pod \"e2e-shared-b-w3-c4d438-openshift-default-6d9d56f475-c995r\" (UID: \"1ec66b87-7d07-45fd-8469-e1bd2c83f651\") " pod="openshift-ingress/e2e-shared-b-w3-c4d438-openshift-default-6d9d56f475-c995r"
Oct 31 01:57:26.046442 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:57:26.046394    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1ec66b87-7d07-45fd-8469-e1bd2c83f651-istio-envoy\") pod \"e2e-shared-b-w3-c4d438-openshift-default-6d9d56f475-c995r\" (UID: \"1ec66b87-7d07-45fd-8469-e1bd2c83f651\") " pod="openshift-ingress/e2e-shared-b-w3-c4d438-openshift-default-6d9d56f475-c995r"
Oct 31 01:57:26.046442 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:57:26.046434    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1ec66b87-7d07-45fd-8469-e1bd2c83f651-istio-podinfo\") pod \"e2e-shared-b-w3-c4d438-openshift-default-6d9d56f475-c995r\" (UID: \"1ec66b87-7d07-45fd-8469-e1bd2c83f651\") " pod="openshift-ingress/e2e-shared-b-w3-c4d438-openshift-default-6d9d56f475-c995r"
Oct 31 01:57:26.046773 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:57:26.046501    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1ec66b87-7d07-45fd-8469-e1bd2c83f651-workload-certs\") pod \"e2e-shared-b-w3-c4d438-openshift-default-6d9d56f475-c995r\" (UID: \"1ec66b87-7d07-45fd-8469-e1bd2c83f651\") " pod="openshift-ingress/e2e-shared-b-w3-c4d438-openshift-default-6d9d56f475-c995r"
Oct 31 01:57:26.046773 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:57:26.046568    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1ec66b87-7d07-45fd-8469-e1bd2c83f651-istiod-ca-cert\") pod \"e2e-shared-b-w3-c4d438-openshift-default-6d9d56f475-c995r\" (UID: \"1ec66b87-7d07-45fd-8469-e1bd2c83f651\") " pod="openshift-ingress/e2e-shared-b-w3-c4d438-openshift-default-6d9d56f475-c995r"
Oct 31 01:57:26.046773 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:57:26.046626    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ec66b87-7d07-45fd-8469-e1bd2c83f651-workload-socket\") pod \"e2e-shared-b-w3-c4d438-openshift-default-6d9d56f475-c995r\" (UID: \"1ec66b87-7d07-45fd-8469-e1bd2c83f651\") " pod="openshift-ingress/e2e-shared-b-w3-c4d438-openshift-default-6d9d56f475-c995r"
Oct 31 01:57:26.046773 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:57:26.046648    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ec66b87-7d07-45fd-8469-e1bd2c83f651-credential-socket\") pod \"e2e-shared-b-w3-c4d438-openshift-default-6d9d56f475-c995r\" (UID: \"1ec66b87-7d07-45fd-8469-e1bd2c83f651\") " pod="openshift-ingress/e2e-shared-b-w3-c4d438-openshift-default-6d9d56f475-c995r"
Oct 31 01:57:26.047031 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:57:26.046989    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1ec66b87-7d07-45fd-8469-e1bd2c83f651-istio-data\") pod \"e2e-shared-b-w3-c4d438-openshift-default-6d9d56f475-c995r\" (UID: \"1ec66b87-7d07-45fd-8469-e1bd2c83f651\") " pod="openshift-ingress/e2e-shared-b-w3-c4d438-openshift-default-6d9d56f475-c995r"
Oct 31 01:57:26.047110 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:57:26.047089    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1ec66b87-7d07-45fd-8469-e1bd2c83f651-workload-certs\") pod \"e2e-shared-b-w3-c4d438-openshift-default-6d9d56f475-c995r\" (UID: \"1ec66b87-7d07-45fd-8469-e1bd2c83f651\") " pod="openshift-ingress/e2e-shared-b-w3-c4d438-openshift-default-6d9d56f475-c995r"
Oct 31 01:57:26.047191 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:57:26.047122    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ec66b87-7d07-45fd-8469-e1bd2c83f651-workload-socket\") pod \"e2e-shared-b-w3-c4d438-openshift-default-6d9d56f475-c995r\" (UID: \"1ec66b87-7d07-45fd-8469-e1bd2c83f651\") " pod="openshift-ingress/e2e-shared-b-w3-c4d438-openshift-default-6d9d56f475-c995r"
Oct 31 01:57:26.047351 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:57:26.047303    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1ec66b87-7d07-45fd-8469-e1bd2c83f651-istiod-ca-cert\") pod \"e2e-shared-b-w3-c4d438-openshift-default-6d9d56f475-c995r\" (UID: \"1ec66b87-7d07-45fd-8469-e1bd2c83f651\") " pod="openshift-ingress/e2e-shared-b-w3-c4d438-openshift-default-6d9d56f475-c995r"
Oct 31 01:57:26.049328 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:57:26.049288    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1ec66b87-7d07-45fd-8469-e1bd2c83f651-istio-envoy\") pod \"e2e-shared-b-w3-c4d438-openshift-default-6d9d56f475-c995r\" (UID: \"1ec66b87-7d07-45fd-8469-e1bd2c83f651\") " pod="openshift-ingress/e2e-shared-b-w3-c4d438-openshift-default-6d9d56f475-c995r"
Oct 31 01:57:26.049846 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:57:26.049828    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1ec66b87-7d07-45fd-8469-e1bd2c83f651-istio-podinfo\") pod \"e2e-shared-b-w3-c4d438-openshift-default-6d9d56f475-c995r\" (UID: \"1ec66b87-7d07-45fd-8469-e1bd2c83f651\") " pod="openshift-ingress/e2e-shared-b-w3-c4d438-openshift-default-6d9d56f475-c995r"
Oct 31 01:57:26.102907 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:57:26.102865    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1ec66b87-7d07-45fd-8469-e1bd2c83f651-istio-token\") pod \"e2e-shared-b-w3-c4d438-openshift-default-6d9d56f475-c995r\" (UID: \"1ec66b87-7d07-45fd-8469-e1bd2c83f651\") " pod="openshift-ingress/e2e-shared-b-w3-c4d438-openshift-default-6d9d56f475-c995r"
Oct 31 01:57:26.103064 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:57:26.102918    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8pwdt\" (UniqueName: \"kubernetes.io/projected/1ec66b87-7d07-45fd-8469-e1bd2c83f651-kube-api-access-8pwdt\") pod \"e2e-shared-b-w3-c4d438-openshift-default-6d9d56f475-c995r\" (UID: \"1ec66b87-7d07-45fd-8469-e1bd2c83f651\") " pod="openshift-ingress/e2e-shared-b-w3-c4d438-openshift-default-6d9d56f475-c995r"
Oct 31 01:57:26.118637 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:57:26.118605    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-c4d438-openshift-default-6d9d56f475-c995r"
Oct 31 01:57:26.136248 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:57:26.136201    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:57:26.314711 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:57:26.314627    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:57:26.556340 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:57:26.555845    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-c4d438-openshift-default-6d9d56f475-c995r"]
Oct 31 01:57:26.592607 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:57:26.592580    4964 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ec66b87_7d07_45fd_8469_e1bd2c83f651.slice/crio-93d5d2e82295d3e412063cd592affd5f8b9b5598ae8f38b3957c04eebca75112 WatchSource:0}: Error finding container 93d5d2e82295d3e412063cd592affd5f8b9b5598ae8f38b3957c04eebca75112: Status 404 returned error can't find the container with id 93d5d2e82295d3e412063cd592affd5f8b9b5598ae8f38b3957c04eebca75112
Oct 31 01:57:26.595762 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:57:26.595734    4964 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 01:57:26.595840 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:57:26.595794    4964 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 01:57:26.595840 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:57:26.595821    4964 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 01:57:27.220681 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:57:27.220644    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-c4d438-openshift-default-6d9d56f475-c995r" event={"ID":"1ec66b87-7d07-45fd-8469-e1bd2c83f651","Type":"ContainerStarted","Data":"224f2098b2772bf59ef5570eb3e4e2d7efb5d0551b47d9ed009178abe42408bf"}
Oct 31 01:57:27.220855 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:57:27.220703    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-c4d438-openshift-default-6d9d56f475-c995r" event={"ID":"1ec66b87-7d07-45fd-8469-e1bd2c83f651","Type":"ContainerStarted","Data":"93d5d2e82295d3e412063cd592affd5f8b9b5598ae8f38b3957c04eebca75112"}
Oct 31 01:57:27.338952 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:57:27.338905    4964 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w3-c4d438-openshift-default-6d9d56f475-c995r" podStartSLOduration=2.338892752 podStartE2EDuration="2.338892752s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:57:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:57:27.337482561 +0000 UTC m=+3215.814048942" watchObservedRunningTime="2026-10-31 01:57:27.338892752 +0000 UTC m=+3215.815459131"
Oct 31 01:57:28.119556 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:57:28.119522    4964 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-c4d438-openshift-default-6d9d56f475-c995r"
Oct 31 01:57:28.125269 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:57:28.125242    4964 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-c4d438-openshift-default-6d9d56f475-c995r"
Oct 31 01:57:28.227567 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:57:28.227533    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-c4d438-openshift-default-6d9d56f475-c995r"
Oct 31 01:57:28.228515 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:57:28.228498    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-c4d438-openshift-default-6d9d56f475-c995r"
Oct 31 01:57:30.275545 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:57:30.275513    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:57:30.513968 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:57:30.513904    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:57:30.763843 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:57:30.763806    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:57:32.125351 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:57:32.125282    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:57:32.333354 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:57:32.333302    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:57:32.527340 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:57:32.526762    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:57:33.489464 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:57:33.489429    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:57:33.674094 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:57:33.674063    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:57:34.158076 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:57:34.158039    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:57:36.089562 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:57:36.089531    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:57:36.568343 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:57:36.568283    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:57:40.544544 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:57:40.544510    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:57:40.896425 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:57:40.896347    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:57:41.238448 ip-10-0-133-197 kubenswrapper[4964]: E1031 01:57:41.238408    4964 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q56x5" containerName="extractor"
Oct 31 01:57:45.147201 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:57:45.147159    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:57:45.322791 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:57:45.322734    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:57:45.796485 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:57:45.796446    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:57:49.231709 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:57:49.231667    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:57:49.674384 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:57:49.674284    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:57:50.604446 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:57:50.604407    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:57:50.917101 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:57:50.917017    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:57:51.098195 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:57:51.098160    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:57:53.468515 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:57:53.468472    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:57:54.357665 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:57:54.357628    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:57:54.636648 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:57:54.636567    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:57:54.825933 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:57:54.825894    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:57:56.123138 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:57:56.123102    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:57:57.591643 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:57:57.591606    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:57:57.816588 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:57:57.816552    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:57:58.287497 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:57:58.287465    4964 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-5632ea-678cc4999-rv754"]
Oct 31 01:57:58.293439 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:57:58.293407    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-5632ea-678cc4999-rv754"
Oct 31 01:57:58.346848 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:57:58.346824    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-5632ea-678cc4999-rv754"]
Oct 31 01:57:58.390472 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:57:58.390441    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xmm5l\" (UniqueName: \"kubernetes.io/projected/ba3f39ff-dde5-4b50-b624-d3144586f11f-kube-api-access-xmm5l\") pod \"payload-pre-processing-e2e-shared-b-w2-5632ea-678cc4999-rv754\" (UID: \"ba3f39ff-dde5-4b50-b624-d3144586f11f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-5632ea-678cc4999-rv754"
Oct 31 01:57:58.390647 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:57:58.390627    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ba3f39ff-dde5-4b50-b624-d3144586f11f-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-5632ea-678cc4999-rv754\" (UID: \"ba3f39ff-dde5-4b50-b624-d3144586f11f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-5632ea-678cc4999-rv754"
Oct 31 01:57:58.391784 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:57:58.391757    4964 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w2-5632ea\""
Oct 31 01:57:58.391927 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:57:58.391840    4964 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w2-5632ea-dockercfg-nbckf\""
Oct 31 01:57:58.491551 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:57:58.491518    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ba3f39ff-dde5-4b50-b624-d3144586f11f-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-5632ea-678cc4999-rv754\" (UID: \"ba3f39ff-dde5-4b50-b624-d3144586f11f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-5632ea-678cc4999-rv754"
Oct 31 01:57:58.491731 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:57:58.491707    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xmm5l\" (UniqueName: \"kubernetes.io/projected/ba3f39ff-dde5-4b50-b624-d3144586f11f-kube-api-access-xmm5l\") pod \"payload-pre-processing-e2e-shared-b-w2-5632ea-678cc4999-rv754\" (UID: \"ba3f39ff-dde5-4b50-b624-d3144586f11f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-5632ea-678cc4999-rv754"
Oct 31 01:57:58.492209 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:57:58.492188    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ba3f39ff-dde5-4b50-b624-d3144586f11f-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-5632ea-678cc4999-rv754\" (UID: \"ba3f39ff-dde5-4b50-b624-d3144586f11f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-5632ea-678cc4999-rv754"
Oct 31 01:57:58.548516 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:57:58.548445    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xmm5l\" (UniqueName: \"kubernetes.io/projected/ba3f39ff-dde5-4b50-b624-d3144586f11f-kube-api-access-xmm5l\") pod \"payload-pre-processing-e2e-shared-b-w2-5632ea-678cc4999-rv754\" (UID: \"ba3f39ff-dde5-4b50-b624-d3144586f11f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-5632ea-678cc4999-rv754"
Oct 31 01:57:58.608035 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:57:58.607990    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-5632ea-678cc4999-rv754"
Oct 31 01:57:58.992363 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:57:58.992305    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-5632ea-678cc4999-rv754"]
Oct 31 01:57:58.993840 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:57:58.993814    4964 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba3f39ff_dde5_4b50_b624_d3144586f11f.slice/crio-ceb93aad076f6057f4c7cbf3ab3c06620cccf5846bf2781ffdf91e29ff92df8e WatchSource:0}: Error finding container ceb93aad076f6057f4c7cbf3ab3c06620cccf5846bf2781ffdf91e29ff92df8e: Status 404 returned error can't find the container with id ceb93aad076f6057f4c7cbf3ab3c06620cccf5846bf2781ffdf91e29ff92df8e
Oct 31 01:57:59.451025 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:57:59.450945    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-5632ea-678cc4999-rv754" event={"ID":"ba3f39ff-dde5-4b50-b624-d3144586f11f","Type":"ContainerStarted","Data":"d4f4abaa61972841995520a7c02102eb71efad52528a827861636cfb769319f1"}
Oct 31 01:57:59.451025 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:57:59.450993    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-5632ea-678cc4999-rv754" event={"ID":"ba3f39ff-dde5-4b50-b624-d3144586f11f","Type":"ContainerStarted","Data":"ceb93aad076f6057f4c7cbf3ab3c06620cccf5846bf2781ffdf91e29ff92df8e"}
Oct 31 01:57:59.451253 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:57:59.451039    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-5632ea-678cc4999-rv754"
Oct 31 01:57:59.568235 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:57:59.568178    4964 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-5632ea-678cc4999-rv754" podStartSLOduration=1.568166642 podStartE2EDuration="1.568166642s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:57:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:57:59.567373904 +0000 UTC m=+3248.043940284" watchObservedRunningTime="2026-10-31 01:57:59.568166642 +0000 UTC m=+3248.044733021"
Oct 31 01:58:01.180907 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:58:01.180879    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:58:06.159528 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:58:06.159491    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:58:06.370195 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:58:06.370156    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:58:10.462291 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:58:10.462264    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-5632ea-678cc4999-rv754"
Oct 31 01:58:11.184056 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:58:11.184023    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:58:11.243246 ip-10-0-133-197 kubenswrapper[4964]: E1031 01:58:11.243212    4964 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q56x5" containerName="extractor"
Oct 31 01:58:15.938487 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:58:15.938447    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:58:16.210114 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:58:16.210041    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:58:20.703106 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:58:20.703071    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:58:25.441894 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:58:25.441859    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:58:30.273358 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:58:30.273227    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:58:34.886376 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:58:34.886312    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:58:35.065635 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:58:35.065598    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:58:35.243198 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:58:35.243166    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:58:36.903466 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:58:36.903417    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:58:39.892590 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:58:39.892558    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:58:40.896709 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:58:40.896675    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:58:41.240081 ip-10-0-133-197 kubenswrapper[4964]: E1031 01:58:41.240047    4964 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q56x5" containerName="extractor"
Oct 31 01:58:43.960203 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:58:43.960119    4964 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-476ce2-6f4f4c79d5-zv7hs"]
Oct 31 01:58:43.960656 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:58:43.960327    4964 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-476ce2-6f4f4c79d5-zv7hs" podUID="0c20b1e6-06d4-4fcc-b595-588ca57b04e9" containerName="maas-api" containerID="cri-o://598e89b73c06c0bff1aa140da31b5e498339284dfd9bdc67d0c0c8e234aabb57" gracePeriod=30
Oct 31 01:58:44.098933 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:58:44.098892    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:58:44.275210 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:58:44.275185    4964 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-476ce2-6f4f4c79d5-zv7hs"
Oct 31 01:58:44.376453 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:58:44.376409    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0c20b1e6-06d4-4fcc-b595-588ca57b04e9-kube-api-access-ndnkj\" (UniqueName: \"kubernetes.io/projected/0c20b1e6-06d4-4fcc-b595-588ca57b04e9-kube-api-access-ndnkj\") pod \"0c20b1e6-06d4-4fcc-b595-588ca57b04e9\" (UID: \"0c20b1e6-06d4-4fcc-b595-588ca57b04e9\") "
Oct 31 01:58:44.376660 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:58:44.376465    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0c20b1e6-06d4-4fcc-b595-588ca57b04e9-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0c20b1e6-06d4-4fcc-b595-588ca57b04e9-openshift-service-ca\") pod \"0c20b1e6-06d4-4fcc-b595-588ca57b04e9\" (UID: \"0c20b1e6-06d4-4fcc-b595-588ca57b04e9\") "
Oct 31 01:58:44.376660 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:58:44.376618    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0c20b1e6-06d4-4fcc-b595-588ca57b04e9-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0c20b1e6-06d4-4fcc-b595-588ca57b04e9-maas-api-tls\") pod \"0c20b1e6-06d4-4fcc-b595-588ca57b04e9\" (UID: \"0c20b1e6-06d4-4fcc-b595-588ca57b04e9\") "
Oct 31 01:58:44.376781 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:58:44.376725    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0c20b1e6-06d4-4fcc-b595-588ca57b04e9-metrics-certs\" (UniqueName: \"kubernetes.io/secret/0c20b1e6-06d4-4fcc-b595-588ca57b04e9-metrics-certs\") pod \"0c20b1e6-06d4-4fcc-b595-588ca57b04e9\" (UID: \"0c20b1e6-06d4-4fcc-b595-588ca57b04e9\") "
Oct 31 01:58:44.376894 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:58:44.376862    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0c20b1e6-06d4-4fcc-b595-588ca57b04e9-openshift-service-ca" pod "0c20b1e6-06d4-4fcc-b595-588ca57b04e9" (UID: "0c20b1e6-06d4-4fcc-b595-588ca57b04e9"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:58:44.377522 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:58:44.377500    4964 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0c20b1e6-06d4-4fcc-b595-588ca57b04e9-openshift-service-ca\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:58:44.378959 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:58:44.378929    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c20b1e6-06d4-4fcc-b595-588ca57b04e9-kube-api-access-ndnkj" pod "0c20b1e6-06d4-4fcc-b595-588ca57b04e9" (UID: "0c20b1e6-06d4-4fcc-b595-588ca57b04e9"). InnerVolumeSpecName "kube-api-access-ndnkj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:58:44.379062 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:58:44.379005    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0c20b1e6-06d4-4fcc-b595-588ca57b04e9-maas-api-tls" pod "0c20b1e6-06d4-4fcc-b595-588ca57b04e9" (UID: "0c20b1e6-06d4-4fcc-b595-588ca57b04e9"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:58:44.379120 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:58:44.379041    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0c20b1e6-06d4-4fcc-b595-588ca57b04e9-metrics-certs" pod "0c20b1e6-06d4-4fcc-b595-588ca57b04e9" (UID: "0c20b1e6-06d4-4fcc-b595-588ca57b04e9"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:58:44.479088 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:58:44.479055    4964 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ndnkj\" (UniqueName: \"kubernetes.io/projected/0c20b1e6-06d4-4fcc-b595-588ca57b04e9-kube-api-access-ndnkj\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:58:44.479088 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:58:44.479085    4964 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0c20b1e6-06d4-4fcc-b595-588ca57b04e9-maas-api-tls\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:58:44.479088 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:58:44.479095    4964 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0c20b1e6-06d4-4fcc-b595-588ca57b04e9-metrics-certs\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:58:44.780998 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:58:44.780961    4964 generic.go:356] "Generic (PLEG): container finished" podID="0c20b1e6-06d4-4fcc-b595-588ca57b04e9" containerID="598e89b73c06c0bff1aa140da31b5e498339284dfd9bdc67d0c0c8e234aabb57" exitCode=0
Oct 31 01:58:44.781182 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:58:44.781023    4964 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-476ce2-6f4f4c79d5-zv7hs"
Oct 31 01:58:44.781182 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:58:44.781022    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-476ce2-6f4f4c79d5-zv7hs" event={"ID":"0c20b1e6-06d4-4fcc-b595-588ca57b04e9","Type":"ContainerDied","Data":"598e89b73c06c0bff1aa140da31b5e498339284dfd9bdc67d0c0c8e234aabb57"}
Oct 31 01:58:44.781182 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:58:44.781160    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-476ce2-6f4f4c79d5-zv7hs" event={"ID":"0c20b1e6-06d4-4fcc-b595-588ca57b04e9","Type":"ContainerDied","Data":"cb3b7f14b85532f2e9c48cf0ff0a8140a29b9208148fc415bcf77a2916a57412"}
Oct 31 01:58:44.781435 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:58:44.781194    4964 scope.go:122] "RemoveContainer" containerID="598e89b73c06c0bff1aa140da31b5e498339284dfd9bdc67d0c0c8e234aabb57"
Oct 31 01:58:44.793780 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:58:44.793749    4964 scope.go:122] "RemoveContainer" containerID="598e89b73c06c0bff1aa140da31b5e498339284dfd9bdc67d0c0c8e234aabb57"
Oct 31 01:58:44.794156 ip-10-0-133-197 kubenswrapper[4964]: E1031 01:58:44.794129    4964 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"598e89b73c06c0bff1aa140da31b5e498339284dfd9bdc67d0c0c8e234aabb57\": container with ID starting with 598e89b73c06c0bff1aa140da31b5e498339284dfd9bdc67d0c0c8e234aabb57 not found: ID does not exist" containerID="598e89b73c06c0bff1aa140da31b5e498339284dfd9bdc67d0c0c8e234aabb57"
Oct 31 01:58:44.794244 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:58:44.794169    4964 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"598e89b73c06c0bff1aa140da31b5e498339284dfd9bdc67d0c0c8e234aabb57"} err="failed to get container status \"598e89b73c06c0bff1aa140da31b5e498339284dfd9bdc67d0c0c8e234aabb57\": rpc error: code = NotFound desc = could not find container \"598e89b73c06c0bff1aa140da31b5e498339284dfd9bdc67d0c0c8e234aabb57\": container with ID starting with 598e89b73c06c0bff1aa140da31b5e498339284dfd9bdc67d0c0c8e234aabb57 not found: ID does not exist"
Oct 31 01:58:44.947103 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:58:44.947020    4964 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-476ce2-6f4f4c79d5-zv7hs"]
Oct 31 01:58:44.950821 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:58:44.950794    4964 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-476ce2-6f4f4c79d5-zv7hs"]
Oct 31 01:58:45.778372 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:58:45.778340    4964 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0c20b1e6-06d4-4fcc-b595-588ca57b04e9" path="/var/lib/kubelet/pods/0c20b1e6-06d4-4fcc-b595-588ca57b04e9/volumes"
Oct 31 01:58:45.798362 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:58:45.797649    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:58:45.969165 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:58:45.969114    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:58:46.621144 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:58:46.621107    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:58:47.460224 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:58:47.460190    4964 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w1-71bc98/auth-test-model-71bc98-kserve-77dd77fdfc-qln45"]
Oct 31 01:58:47.461083 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:58:47.461063    4964 cpu_manager.go:414] "containerMap: removing container" podUID="0c20b1e6-06d4-4fcc-b595-588ca57b04e9" containerName="maas-api"
Oct 31 01:58:47.461157 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:58:47.461086    4964 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0c20b1e6-06d4-4fcc-b595-588ca57b04e9" containerName="maas-api"
Oct 31 01:58:47.461242 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:58:47.461233    4964 memory_manager.go:365] "RemoveStaleState removing state" podUID="0c20b1e6-06d4-4fcc-b595-588ca57b04e9" containerName="maas-api"
Oct 31 01:58:47.465056 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:58:47.465038    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-71bc98/auth-test-model-71bc98-kserve-77dd77fdfc-qln45"
Oct 31 01:58:47.523949 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:58:47.523909    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3d0e640a-41cb-48fe-9b7f-8927fd534ca4-dshm\") pod \"auth-test-model-71bc98-kserve-77dd77fdfc-qln45\" (UID: \"3d0e640a-41cb-48fe-9b7f-8927fd534ca4\") " pod="ai-tenant-e2e-shared-b-w1-71bc98/auth-test-model-71bc98-kserve-77dd77fdfc-qln45"
Oct 31 01:58:47.524136 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:58:47.523994    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3d0e640a-41cb-48fe-9b7f-8927fd534ca4-tls-certs\") pod \"auth-test-model-71bc98-kserve-77dd77fdfc-qln45\" (UID: \"3d0e640a-41cb-48fe-9b7f-8927fd534ca4\") " pod="ai-tenant-e2e-shared-b-w1-71bc98/auth-test-model-71bc98-kserve-77dd77fdfc-qln45"
Oct 31 01:58:47.524136 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:58:47.524080    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3d0e640a-41cb-48fe-9b7f-8927fd534ca4-home\") pod \"auth-test-model-71bc98-kserve-77dd77fdfc-qln45\" (UID: \"3d0e640a-41cb-48fe-9b7f-8927fd534ca4\") " pod="ai-tenant-e2e-shared-b-w1-71bc98/auth-test-model-71bc98-kserve-77dd77fdfc-qln45"
Oct 31 01:58:47.524263 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:58:47.524146    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3d0e640a-41cb-48fe-9b7f-8927fd534ca4-tmp-dir\") pod \"auth-test-model-71bc98-kserve-77dd77fdfc-qln45\" (UID: \"3d0e640a-41cb-48fe-9b7f-8927fd534ca4\") " pod="ai-tenant-e2e-shared-b-w1-71bc98/auth-test-model-71bc98-kserve-77dd77fdfc-qln45"
Oct 31 01:58:47.524263 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:58:47.524242    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cbz52\" (UniqueName: \"kubernetes.io/projected/3d0e640a-41cb-48fe-9b7f-8927fd534ca4-kube-api-access-cbz52\") pod \"auth-test-model-71bc98-kserve-77dd77fdfc-qln45\" (UID: \"3d0e640a-41cb-48fe-9b7f-8927fd534ca4\") " pod="ai-tenant-e2e-shared-b-w1-71bc98/auth-test-model-71bc98-kserve-77dd77fdfc-qln45"
Oct 31 01:58:47.524429 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:58:47.524368    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3d0e640a-41cb-48fe-9b7f-8927fd534ca4-model-cache\") pod \"auth-test-model-71bc98-kserve-77dd77fdfc-qln45\" (UID: \"3d0e640a-41cb-48fe-9b7f-8927fd534ca4\") " pod="ai-tenant-e2e-shared-b-w1-71bc98/auth-test-model-71bc98-kserve-77dd77fdfc-qln45"
Oct 31 01:58:47.536122 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:58:47.536091    4964 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w1-71bc98\"/\"openshift-service-ca.crt\""
Oct 31 01:58:47.538792 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:58:47.538763    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w1-71bc98/auth-test-model-71bc98-kserve-77dd77fdfc-qln45"]
Oct 31 01:58:47.566680 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:58:47.566645    4964 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w1-71bc98\"/\"kube-root-ca.crt\""
Oct 31 01:58:47.566680 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:58:47.566683    4964 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w1-71bc98\"/\"default-dockercfg-t26rh\""
Oct 31 01:58:47.566914 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:58:47.566750    4964 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w1-71bc98\"/\"auth-test-model-71bc98-kserve-self-signed-certs\""
Oct 31 01:58:47.607198 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:58:47.600699    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:58:47.638252 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:58:47.638212    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cbz52\" (UniqueName: \"kubernetes.io/projected/3d0e640a-41cb-48fe-9b7f-8927fd534ca4-kube-api-access-cbz52\") pod \"auth-test-model-71bc98-kserve-77dd77fdfc-qln45\" (UID: \"3d0e640a-41cb-48fe-9b7f-8927fd534ca4\") " pod="ai-tenant-e2e-shared-b-w1-71bc98/auth-test-model-71bc98-kserve-77dd77fdfc-qln45"
Oct 31 01:58:47.638502 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:58:47.638347    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3d0e640a-41cb-48fe-9b7f-8927fd534ca4-model-cache\") pod \"auth-test-model-71bc98-kserve-77dd77fdfc-qln45\" (UID: \"3d0e640a-41cb-48fe-9b7f-8927fd534ca4\") " pod="ai-tenant-e2e-shared-b-w1-71bc98/auth-test-model-71bc98-kserve-77dd77fdfc-qln45"
Oct 31 01:58:47.638502 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:58:47.638384    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3d0e640a-41cb-48fe-9b7f-8927fd534ca4-dshm\") pod \"auth-test-model-71bc98-kserve-77dd77fdfc-qln45\" (UID: \"3d0e640a-41cb-48fe-9b7f-8927fd534ca4\") " pod="ai-tenant-e2e-shared-b-w1-71bc98/auth-test-model-71bc98-kserve-77dd77fdfc-qln45"
Oct 31 01:58:47.638502 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:58:47.638439    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3d0e640a-41cb-48fe-9b7f-8927fd534ca4-tls-certs\") pod \"auth-test-model-71bc98-kserve-77dd77fdfc-qln45\" (UID: \"3d0e640a-41cb-48fe-9b7f-8927fd534ca4\") " pod="ai-tenant-e2e-shared-b-w1-71bc98/auth-test-model-71bc98-kserve-77dd77fdfc-qln45"
Oct 31 01:58:47.638673 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:58:47.638520    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3d0e640a-41cb-48fe-9b7f-8927fd534ca4-home\") pod \"auth-test-model-71bc98-kserve-77dd77fdfc-qln45\" (UID: \"3d0e640a-41cb-48fe-9b7f-8927fd534ca4\") " pod="ai-tenant-e2e-shared-b-w1-71bc98/auth-test-model-71bc98-kserve-77dd77fdfc-qln45"
Oct 31 01:58:47.638673 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:58:47.638581    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3d0e640a-41cb-48fe-9b7f-8927fd534ca4-tmp-dir\") pod \"auth-test-model-71bc98-kserve-77dd77fdfc-qln45\" (UID: \"3d0e640a-41cb-48fe-9b7f-8927fd534ca4\") " pod="ai-tenant-e2e-shared-b-w1-71bc98/auth-test-model-71bc98-kserve-77dd77fdfc-qln45"
Oct 31 01:58:47.639056 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:58:47.639033    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3d0e640a-41cb-48fe-9b7f-8927fd534ca4-tmp-dir\") pod \"auth-test-model-71bc98-kserve-77dd77fdfc-qln45\" (UID: \"3d0e640a-41cb-48fe-9b7f-8927fd534ca4\") " pod="ai-tenant-e2e-shared-b-w1-71bc98/auth-test-model-71bc98-kserve-77dd77fdfc-qln45"
Oct 31 01:58:47.639644 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:58:47.639622    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3d0e640a-41cb-48fe-9b7f-8927fd534ca4-model-cache\") pod \"auth-test-model-71bc98-kserve-77dd77fdfc-qln45\" (UID: \"3d0e640a-41cb-48fe-9b7f-8927fd534ca4\") " pod="ai-tenant-e2e-shared-b-w1-71bc98/auth-test-model-71bc98-kserve-77dd77fdfc-qln45"
Oct 31 01:58:47.639859 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:58:47.639833    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3d0e640a-41cb-48fe-9b7f-8927fd534ca4-home\") pod \"auth-test-model-71bc98-kserve-77dd77fdfc-qln45\" (UID: \"3d0e640a-41cb-48fe-9b7f-8927fd534ca4\") " pod="ai-tenant-e2e-shared-b-w1-71bc98/auth-test-model-71bc98-kserve-77dd77fdfc-qln45"
Oct 31 01:58:47.642913 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:58:47.642787    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3d0e640a-41cb-48fe-9b7f-8927fd534ca4-dshm\") pod \"auth-test-model-71bc98-kserve-77dd77fdfc-qln45\" (UID: \"3d0e640a-41cb-48fe-9b7f-8927fd534ca4\") " pod="ai-tenant-e2e-shared-b-w1-71bc98/auth-test-model-71bc98-kserve-77dd77fdfc-qln45"
Oct 31 01:58:47.643128 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:58:47.642999    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3d0e640a-41cb-48fe-9b7f-8927fd534ca4-tls-certs\") pod \"auth-test-model-71bc98-kserve-77dd77fdfc-qln45\" (UID: \"3d0e640a-41cb-48fe-9b7f-8927fd534ca4\") " pod="ai-tenant-e2e-shared-b-w1-71bc98/auth-test-model-71bc98-kserve-77dd77fdfc-qln45"
Oct 31 01:58:47.696121 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:58:47.696085    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cbz52\" (UniqueName: \"kubernetes.io/projected/3d0e640a-41cb-48fe-9b7f-8927fd534ca4-kube-api-access-cbz52\") pod \"auth-test-model-71bc98-kserve-77dd77fdfc-qln45\" (UID: \"3d0e640a-41cb-48fe-9b7f-8927fd534ca4\") " pod="ai-tenant-e2e-shared-b-w1-71bc98/auth-test-model-71bc98-kserve-77dd77fdfc-qln45"
Oct 31 01:58:47.778900 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:58:47.778868    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-71bc98/auth-test-model-71bc98-kserve-77dd77fdfc-qln45"
Oct 31 01:58:48.173365 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:58:48.173339    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w1-71bc98/auth-test-model-71bc98-kserve-77dd77fdfc-qln45"]
Oct 31 01:58:48.174886 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:58:48.174856    4964 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3d0e640a_41cb_48fe_9b7f_8927fd534ca4.slice/crio-7a641b433b44a56cff93fee57e1cc92930be87d2352cf267a13c0d87baaa86a4 WatchSource:0}: Error finding container 7a641b433b44a56cff93fee57e1cc92930be87d2352cf267a13c0d87baaa86a4: Status 404 returned error can't find the container with id 7a641b433b44a56cff93fee57e1cc92930be87d2352cf267a13c0d87baaa86a4
Oct 31 01:58:48.817883 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:58:48.817839    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-71bc98/auth-test-model-71bc98-kserve-77dd77fdfc-qln45" event={"ID":"3d0e640a-41cb-48fe-9b7f-8927fd534ca4","Type":"ContainerStarted","Data":"6155efa6a78befb0bca4b3f34af9276dbee8545d7973d851807fce8433268cb3"}
Oct 31 01:58:48.818450 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:58:48.817901    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-71bc98/auth-test-model-71bc98-kserve-77dd77fdfc-qln45" event={"ID":"3d0e640a-41cb-48fe-9b7f-8927fd534ca4","Type":"ContainerStarted","Data":"7a641b433b44a56cff93fee57e1cc92930be87d2352cf267a13c0d87baaa86a4"}
Oct 31 01:58:48.935561 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:58:48.935498    4964 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w1-71bc98/auth-test-model-71bc98-kserve-77dd77fdfc-qln45" podStartSLOduration=1.935483612 podStartE2EDuration="1.935483612s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:58:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:58:48.932773628 +0000 UTC m=+3297.409340009" watchObservedRunningTime="2026-10-31 01:58:48.935483612 +0000 UTC m=+3297.412049992"
Oct 31 01:58:49.779014 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:58:49.778981    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w1-71bc98/auth-test-model-71bc98-kserve-77dd77fdfc-qln45"
Oct 31 01:58:50.853416 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:58:50.849057    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:58:52.283835 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:58:52.283805    4964 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-9257n_f0c0ada7-7688-4dc2-97fc-813e4d0c4ef3/console-operator/2.log"
Oct 31 01:58:52.294774 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:58:52.294750    4964 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-9257n_f0c0ada7-7688-4dc2-97fc-813e4d0c4ef3/console-operator/2.log"
Oct 31 01:58:52.441536 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:58:52.441443    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:58:52.767123 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:58:52.766812    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:58:52.824625 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:58:52.824593    4964 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-476ce2-openshift-default-6df8c7ff5c-wqzzs"]
Oct 31 01:58:52.824821 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:58:52.824795    4964 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-b-476ce2-openshift-default-6df8c7ff5c-wqzzs" podUID="5a313b20-f6bb-446c-b23b-925c75a5df66" containerName="istio-proxy" containerID="cri-o://f0154e37e7309edbf51288e73c9ddffe9d1a6b6ec31f8f1592d51f40d4f270da" gracePeriod=30
Oct 31 01:58:52.953983 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:58:52.953949    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:58:53.798395 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:58:53.798354    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:58:55.694838 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:58:55.689834    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:58:57.219919 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:58:57.219862    4964 patch_prober.go:29] interesting pod/e2e-praxis-b-476ce2-openshift-default-6df8c7ff5c-wqzzs container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 01:58:57.220453 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:58:57.219999    4964 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-praxis-b-476ce2-openshift-default-6df8c7ff5c-wqzzs" podUID="5a313b20-f6bb-446c-b23b-925c75a5df66" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 01:58:57.779227 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:58:57.779191    4964 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w1-71bc98/auth-test-model-71bc98-kserve-77dd77fdfc-qln45"
Oct 31 01:58:57.818249 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:58:57.818213    4964 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w1-71bc98/auth-test-model-71bc98-kserve-77dd77fdfc-qln45"
Oct 31 01:58:57.923470 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:58:57.923436    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w1-71bc98/auth-test-model-71bc98-kserve-77dd77fdfc-qln45"
Oct 31 01:58:58.157768 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:58:58.154796    4964 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-476ce2-openshift-default-6df8c7ff5c-wqzzs"
Oct 31 01:58:58.221425 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:58:58.197722    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5a313b20-f6bb-446c-b23b-925c75a5df66-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a313b20-f6bb-446c-b23b-925c75a5df66-istio-podinfo\") pod \"5a313b20-f6bb-446c-b23b-925c75a5df66\" (UID: \"5a313b20-f6bb-446c-b23b-925c75a5df66\") "
Oct 31 01:58:58.221425 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:58:58.197799    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a313b20-f6bb-446c-b23b-925c75a5df66-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a313b20-f6bb-446c-b23b-925c75a5df66-workload-certs\") pod \"5a313b20-f6bb-446c-b23b-925c75a5df66\" (UID: \"5a313b20-f6bb-446c-b23b-925c75a5df66\") "
Oct 31 01:58:58.221425 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:58:58.197843    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a313b20-f6bb-446c-b23b-925c75a5df66-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a313b20-f6bb-446c-b23b-925c75a5df66-credential-socket\") pod \"5a313b20-f6bb-446c-b23b-925c75a5df66\" (UID: \"5a313b20-f6bb-446c-b23b-925c75a5df66\") "
Oct 31 01:58:58.221425 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:58:58.197887    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5a313b20-f6bb-446c-b23b-925c75a5df66-istio-token\" (UniqueName: \"kubernetes.io/projected/5a313b20-f6bb-446c-b23b-925c75a5df66-istio-token\") pod \"5a313b20-f6bb-446c-b23b-925c75a5df66\" (UID: \"5a313b20-f6bb-446c-b23b-925c75a5df66\") "
Oct 31 01:58:58.221425 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:58:58.197921    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a313b20-f6bb-446c-b23b-925c75a5df66-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a313b20-f6bb-446c-b23b-925c75a5df66-istio-data\") pod \"5a313b20-f6bb-446c-b23b-925c75a5df66\" (UID: \"5a313b20-f6bb-446c-b23b-925c75a5df66\") "
Oct 31 01:58:58.221425 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:58:58.198082    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5a313b20-f6bb-446c-b23b-925c75a5df66-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a313b20-f6bb-446c-b23b-925c75a5df66-istiod-ca-cert\") pod \"5a313b20-f6bb-446c-b23b-925c75a5df66\" (UID: \"5a313b20-f6bb-446c-b23b-925c75a5df66\") "
Oct 31 01:58:58.221425 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:58:58.198171    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a313b20-f6bb-446c-b23b-925c75a5df66-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a313b20-f6bb-446c-b23b-925c75a5df66-workload-socket\") pod \"5a313b20-f6bb-446c-b23b-925c75a5df66\" (UID: \"5a313b20-f6bb-446c-b23b-925c75a5df66\") "
Oct 31 01:58:58.221425 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:58:58.198213    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a313b20-f6bb-446c-b23b-925c75a5df66-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a313b20-f6bb-446c-b23b-925c75a5df66-istio-envoy\") pod \"5a313b20-f6bb-446c-b23b-925c75a5df66\" (UID: \"5a313b20-f6bb-446c-b23b-925c75a5df66\") "
Oct 31 01:58:58.221425 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:58:58.198292    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5a313b20-f6bb-446c-b23b-925c75a5df66-kube-api-access-x95wq\" (UniqueName: \"kubernetes.io/projected/5a313b20-f6bb-446c-b23b-925c75a5df66-kube-api-access-x95wq\") pod \"5a313b20-f6bb-446c-b23b-925c75a5df66\" (UID: \"5a313b20-f6bb-446c-b23b-925c75a5df66\") "
Oct 31 01:58:58.221425 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:58:58.213759    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a313b20-f6bb-446c-b23b-925c75a5df66-istio-data" pod "5a313b20-f6bb-446c-b23b-925c75a5df66" (UID: "5a313b20-f6bb-446c-b23b-925c75a5df66"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:58:58.221425 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:58:58.214053    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5a313b20-f6bb-446c-b23b-925c75a5df66-istiod-ca-cert" pod "5a313b20-f6bb-446c-b23b-925c75a5df66" (UID: "5a313b20-f6bb-446c-b23b-925c75a5df66"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:58:58.222560 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:58:58.222228    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a313b20-f6bb-446c-b23b-925c75a5df66-credential-socket" pod "5a313b20-f6bb-446c-b23b-925c75a5df66" (UID: "5a313b20-f6bb-446c-b23b-925c75a5df66"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:58:58.222625 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:58:58.222607    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a313b20-f6bb-446c-b23b-925c75a5df66-workload-socket" pod "5a313b20-f6bb-446c-b23b-925c75a5df66" (UID: "5a313b20-f6bb-446c-b23b-925c75a5df66"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:58:58.222690 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:58:58.222672    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a313b20-f6bb-446c-b23b-925c75a5df66-workload-certs" pod "5a313b20-f6bb-446c-b23b-925c75a5df66" (UID: "5a313b20-f6bb-446c-b23b-925c75a5df66"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:58:58.226784 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:58:58.226713    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a313b20-f6bb-446c-b23b-925c75a5df66-kube-api-access-x95wq" pod "5a313b20-f6bb-446c-b23b-925c75a5df66" (UID: "5a313b20-f6bb-446c-b23b-925c75a5df66"). InnerVolumeSpecName "kube-api-access-x95wq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:58:58.227160 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:58:58.226850    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a313b20-f6bb-446c-b23b-925c75a5df66-istio-token" pod "5a313b20-f6bb-446c-b23b-925c75a5df66" (UID: "5a313b20-f6bb-446c-b23b-925c75a5df66"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:58:58.227160 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:58:58.227089    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a313b20-f6bb-446c-b23b-925c75a5df66-istio-envoy" pod "5a313b20-f6bb-446c-b23b-925c75a5df66" (UID: "5a313b20-f6bb-446c-b23b-925c75a5df66"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:58:58.227693 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:58:58.227633    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5a313b20-f6bb-446c-b23b-925c75a5df66-istio-podinfo" pod "5a313b20-f6bb-446c-b23b-925c75a5df66" (UID: "5a313b20-f6bb-446c-b23b-925c75a5df66"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 01:58:58.300457 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:58:58.300132    4964 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a313b20-f6bb-446c-b23b-925c75a5df66-workload-socket\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:58:58.300457 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:58:58.300162    4964 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a313b20-f6bb-446c-b23b-925c75a5df66-istio-envoy\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:58:58.300457 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:58:58.300172    4964 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x95wq\" (UniqueName: \"kubernetes.io/projected/5a313b20-f6bb-446c-b23b-925c75a5df66-kube-api-access-x95wq\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:58:58.300457 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:58:58.300182    4964 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a313b20-f6bb-446c-b23b-925c75a5df66-istio-podinfo\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:58:58.300457 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:58:58.300190    4964 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a313b20-f6bb-446c-b23b-925c75a5df66-workload-certs\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:58:58.300457 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:58:58.300199    4964 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a313b20-f6bb-446c-b23b-925c75a5df66-credential-socket\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:58:58.300457 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:58:58.300208    4964 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5a313b20-f6bb-446c-b23b-925c75a5df66-istio-token\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:58:58.300457 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:58:58.300216    4964 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a313b20-f6bb-446c-b23b-925c75a5df66-istio-data\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:58:58.300457 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:58:58.300226    4964 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a313b20-f6bb-446c-b23b-925c75a5df66-istiod-ca-cert\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:58:58.926339 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:58:58.925555    4964 generic.go:356] "Generic (PLEG): container finished" podID="5a313b20-f6bb-446c-b23b-925c75a5df66" containerID="f0154e37e7309edbf51288e73c9ddffe9d1a6b6ec31f8f1592d51f40d4f270da" exitCode=0
Oct 31 01:58:58.926339 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:58:58.925991    4964 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-476ce2-openshift-default-6df8c7ff5c-wqzzs"
Oct 31 01:58:58.928159 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:58:58.928036    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-476ce2-openshift-default-6df8c7ff5c-wqzzs" event={"ID":"5a313b20-f6bb-446c-b23b-925c75a5df66","Type":"ContainerDied","Data":"f0154e37e7309edbf51288e73c9ddffe9d1a6b6ec31f8f1592d51f40d4f270da"}
Oct 31 01:58:58.928159 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:58:58.928103    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-476ce2-openshift-default-6df8c7ff5c-wqzzs" event={"ID":"5a313b20-f6bb-446c-b23b-925c75a5df66","Type":"ContainerDied","Data":"06ab6d1c1655cbdd3e17454fb4bde664f16e000348e935f8984d4e961271a5d2"}
Oct 31 01:58:58.928159 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:58:58.928130    4964 scope.go:122] "RemoveContainer" containerID="f0154e37e7309edbf51288e73c9ddffe9d1a6b6ec31f8f1592d51f40d4f270da"
Oct 31 01:58:58.944922 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:58:58.944898    4964 scope.go:122] "RemoveContainer" containerID="a3cbedca00622e1ebaa8aae6974a651e953cb06626288052f24eed3e79af01a0"
Oct 31 01:58:58.955658 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:58:58.955637    4964 scope.go:122] "RemoveContainer" containerID="f0154e37e7309edbf51288e73c9ddffe9d1a6b6ec31f8f1592d51f40d4f270da"
Oct 31 01:58:58.956013 ip-10-0-133-197 kubenswrapper[4964]: E1031 01:58:58.955995    4964 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f0154e37e7309edbf51288e73c9ddffe9d1a6b6ec31f8f1592d51f40d4f270da\": container with ID starting with f0154e37e7309edbf51288e73c9ddffe9d1a6b6ec31f8f1592d51f40d4f270da not found: ID does not exist" containerID="f0154e37e7309edbf51288e73c9ddffe9d1a6b6ec31f8f1592d51f40d4f270da"
Oct 31 01:58:58.956067 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:58:58.956025    4964 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f0154e37e7309edbf51288e73c9ddffe9d1a6b6ec31f8f1592d51f40d4f270da"} err="failed to get container status \"f0154e37e7309edbf51288e73c9ddffe9d1a6b6ec31f8f1592d51f40d4f270da\": rpc error: code = NotFound desc = could not find container \"f0154e37e7309edbf51288e73c9ddffe9d1a6b6ec31f8f1592d51f40d4f270da\": container with ID starting with f0154e37e7309edbf51288e73c9ddffe9d1a6b6ec31f8f1592d51f40d4f270da not found: ID does not exist"
Oct 31 01:58:58.956067 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:58:58.956052    4964 scope.go:122] "RemoveContainer" containerID="a3cbedca00622e1ebaa8aae6974a651e953cb06626288052f24eed3e79af01a0"
Oct 31 01:58:58.956457 ip-10-0-133-197 kubenswrapper[4964]: E1031 01:58:58.956428    4964 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a3cbedca00622e1ebaa8aae6974a651e953cb06626288052f24eed3e79af01a0\": container with ID starting with a3cbedca00622e1ebaa8aae6974a651e953cb06626288052f24eed3e79af01a0 not found: ID does not exist" containerID="a3cbedca00622e1ebaa8aae6974a651e953cb06626288052f24eed3e79af01a0"
Oct 31 01:58:58.956503 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:58:58.956464    4964 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a3cbedca00622e1ebaa8aae6974a651e953cb06626288052f24eed3e79af01a0"} err="failed to get container status \"a3cbedca00622e1ebaa8aae6974a651e953cb06626288052f24eed3e79af01a0\": rpc error: code = NotFound desc = could not find container \"a3cbedca00622e1ebaa8aae6974a651e953cb06626288052f24eed3e79af01a0\": container with ID starting with a3cbedca00622e1ebaa8aae6974a651e953cb06626288052f24eed3e79af01a0 not found: ID does not exist"
Oct 31 01:58:59.113344 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:58:59.105196    4964 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-476ce2-openshift-default-6df8c7ff5c-wqzzs"]
Oct 31 01:58:59.148829 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:58:59.148793    4964 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-b-476ce2-openshift-default-6df8c7ff5c-wqzzs"]
Oct 31 01:58:59.777954 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:58:59.777917    4964 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5a313b20-f6bb-446c-b23b-925c75a5df66" path="/var/lib/kubelet/pods/5a313b20-f6bb-446c-b23b-925c75a5df66/volumes"
Oct 31 01:59:00.127142 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:59:00.127005    4964 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-5632ea-b5b6857b8-5zj46"]
Oct 31 01:59:00.127988 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:59:00.127968    4964 cpu_manager.go:414] "containerMap: removing container" podUID="5a313b20-f6bb-446c-b23b-925c75a5df66" containerName="istio-proxy"
Oct 31 01:59:00.128079 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:59:00.127991    4964 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5a313b20-f6bb-446c-b23b-925c75a5df66" containerName="istio-proxy"
Oct 31 01:59:00.128079 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:59:00.128028    4964 cpu_manager.go:414] "containerMap: removing container" podUID="5a313b20-f6bb-446c-b23b-925c75a5df66" containerName="istio-proxy"
Oct 31 01:59:00.128079 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:59:00.128036    4964 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5a313b20-f6bb-446c-b23b-925c75a5df66" containerName="istio-proxy"
Oct 31 01:59:00.128255 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:59:00.128243    4964 memory_manager.go:365] "RemoveStaleState removing state" podUID="5a313b20-f6bb-446c-b23b-925c75a5df66" containerName="istio-proxy"
Oct 31 01:59:00.128292 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:59:00.128263    4964 memory_manager.go:365] "RemoveStaleState removing state" podUID="5a313b20-f6bb-446c-b23b-925c75a5df66" containerName="istio-proxy"
Oct 31 01:59:00.134243 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:59:00.134218    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-5632ea-b5b6857b8-5zj46"
Oct 31 01:59:00.188534 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:59:00.188498    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-5632ea-b5b6857b8-5zj46"]
Oct 31 01:59:00.232504 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:59:00.232468    4964 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w2-5632ea-serving-cert\""
Oct 31 01:59:00.249670 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:59:00.249636    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/290631aa-9208-4dbb-8a0c-aa79cea9e6f8-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-5632ea-b5b6857b8-5zj46\" (UID: \"290631aa-9208-4dbb-8a0c-aa79cea9e6f8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-5632ea-b5b6857b8-5zj46"
Oct 31 01:59:00.249968 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:59:00.249950    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4xcb7\" (UniqueName: \"kubernetes.io/projected/290631aa-9208-4dbb-8a0c-aa79cea9e6f8-kube-api-access-4xcb7\") pod \"maas-api-e2e-shared-b-w2-5632ea-b5b6857b8-5zj46\" (UID: \"290631aa-9208-4dbb-8a0c-aa79cea9e6f8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-5632ea-b5b6857b8-5zj46"
Oct 31 01:59:00.250145 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:59:00.250126    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/290631aa-9208-4dbb-8a0c-aa79cea9e6f8-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-5632ea-b5b6857b8-5zj46\" (UID: \"290631aa-9208-4dbb-8a0c-aa79cea9e6f8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-5632ea-b5b6857b8-5zj46"
Oct 31 01:59:00.250303 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:59:00.250288    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/290631aa-9208-4dbb-8a0c-aa79cea9e6f8-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-5632ea-b5b6857b8-5zj46\" (UID: \"290631aa-9208-4dbb-8a0c-aa79cea9e6f8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-5632ea-b5b6857b8-5zj46"
Oct 31 01:59:00.347289 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:59:00.347253    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:59:00.351467 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:59:00.351434    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/290631aa-9208-4dbb-8a0c-aa79cea9e6f8-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-5632ea-b5b6857b8-5zj46\" (UID: \"290631aa-9208-4dbb-8a0c-aa79cea9e6f8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-5632ea-b5b6857b8-5zj46"
Oct 31 01:59:00.351629 ip-10-0-133-197 kubenswrapper[4964]: E1031 01:59:00.351595    4964 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-5632ea-serving-cert: secret "maas-api-e2e-shared-b-w2-5632ea-serving-cert" not found
Oct 31 01:59:00.351699 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:59:00.351598    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/290631aa-9208-4dbb-8a0c-aa79cea9e6f8-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-5632ea-b5b6857b8-5zj46\" (UID: \"290631aa-9208-4dbb-8a0c-aa79cea9e6f8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-5632ea-b5b6857b8-5zj46"
Oct 31 01:59:00.351699 ip-10-0-133-197 kubenswrapper[4964]: E1031 01:59:00.351668    4964 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/290631aa-9208-4dbb-8a0c-aa79cea9e6f8-maas-api-tls podName:290631aa-9208-4dbb-8a0c-aa79cea9e6f8 nodeName:}" failed. No retries permitted until 2026-10-31 01:59:00.851647223 +0000 UTC m=+3309.328213611 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/290631aa-9208-4dbb-8a0c-aa79cea9e6f8-maas-api-tls") pod "maas-api-e2e-shared-b-w2-5632ea-b5b6857b8-5zj46" (UID: "290631aa-9208-4dbb-8a0c-aa79cea9e6f8") : secret "maas-api-e2e-shared-b-w2-5632ea-serving-cert" not found
Oct 31 01:59:00.351816 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:59:00.351753    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4xcb7\" (UniqueName: \"kubernetes.io/projected/290631aa-9208-4dbb-8a0c-aa79cea9e6f8-kube-api-access-4xcb7\") pod \"maas-api-e2e-shared-b-w2-5632ea-b5b6857b8-5zj46\" (UID: \"290631aa-9208-4dbb-8a0c-aa79cea9e6f8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-5632ea-b5b6857b8-5zj46"
Oct 31 01:59:00.351816 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:59:00.351773    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/290631aa-9208-4dbb-8a0c-aa79cea9e6f8-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-5632ea-b5b6857b8-5zj46\" (UID: \"290631aa-9208-4dbb-8a0c-aa79cea9e6f8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-5632ea-b5b6857b8-5zj46"
Oct 31 01:59:00.352491 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:59:00.352468    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/290631aa-9208-4dbb-8a0c-aa79cea9e6f8-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-5632ea-b5b6857b8-5zj46\" (UID: \"290631aa-9208-4dbb-8a0c-aa79cea9e6f8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-5632ea-b5b6857b8-5zj46"
Oct 31 01:59:00.354272 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:59:00.354245    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/290631aa-9208-4dbb-8a0c-aa79cea9e6f8-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-5632ea-b5b6857b8-5zj46\" (UID: \"290631aa-9208-4dbb-8a0c-aa79cea9e6f8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-5632ea-b5b6857b8-5zj46"
Oct 31 01:59:00.437232 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:59:00.437145    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4xcb7\" (UniqueName: \"kubernetes.io/projected/290631aa-9208-4dbb-8a0c-aa79cea9e6f8-kube-api-access-4xcb7\") pod \"maas-api-e2e-shared-b-w2-5632ea-b5b6857b8-5zj46\" (UID: \"290631aa-9208-4dbb-8a0c-aa79cea9e6f8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-5632ea-b5b6857b8-5zj46"
Oct 31 01:59:00.524015 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:59:00.523972    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:59:00.696276 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:59:00.696186    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:59:00.860258 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:59:00.860223    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/290631aa-9208-4dbb-8a0c-aa79cea9e6f8-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-5632ea-b5b6857b8-5zj46\" (UID: \"290631aa-9208-4dbb-8a0c-aa79cea9e6f8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-5632ea-b5b6857b8-5zj46"
Oct 31 01:59:00.862871 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:59:00.862845    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/290631aa-9208-4dbb-8a0c-aa79cea9e6f8-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-5632ea-b5b6857b8-5zj46\" (UID: \"290631aa-9208-4dbb-8a0c-aa79cea9e6f8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-5632ea-b5b6857b8-5zj46"
Oct 31 01:59:01.049062 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:59:01.049027    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-5632ea-b5b6857b8-5zj46"
Oct 31 01:59:01.442923 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:59:01.441958    4964 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod290631aa_9208_4dbb_8a0c_aa79cea9e6f8.slice/crio-40b8119b50769dff2377c83ea574e47f48e504f2a07254e66a5f7aeb072de6a9 WatchSource:0}: Error finding container 40b8119b50769dff2377c83ea574e47f48e504f2a07254e66a5f7aeb072de6a9: Status 404 returned error can't find the container with id 40b8119b50769dff2377c83ea574e47f48e504f2a07254e66a5f7aeb072de6a9
Oct 31 01:59:01.443758 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:59:01.443740    4964 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 31 01:59:01.456763 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:59:01.456741    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-5632ea-b5b6857b8-5zj46"]
Oct 31 01:59:01.532631 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:59:01.532598    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:59:01.956294 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:59:01.956254    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-5632ea-b5b6857b8-5zj46" event={"ID":"290631aa-9208-4dbb-8a0c-aa79cea9e6f8","Type":"ContainerStarted","Data":"40b8119b50769dff2377c83ea574e47f48e504f2a07254e66a5f7aeb072de6a9"}
Oct 31 01:59:02.966802 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:59:02.966766    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-5632ea-b5b6857b8-5zj46" event={"ID":"290631aa-9208-4dbb-8a0c-aa79cea9e6f8","Type":"ContainerStarted","Data":"46cd860ac553eca1c24bf50488224f62e68e46a1d98903afd843b6b95b2cf31e"}
Oct 31 01:59:02.967263 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:59:02.967006    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-5632ea-b5b6857b8-5zj46"
Oct 31 01:59:03.089732 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:59:03.089666    4964 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-5632ea-b5b6857b8-5zj46" podStartSLOduration=2.246802755 podStartE2EDuration="3.089647636s" totalImagesPullingTime="842.844881ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:59:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:59:03.083879941 +0000 UTC m=+3311.560446313" watchObservedRunningTime="2026-10-31 01:59:03.089647636 +0000 UTC m=+3311.566214019"
Oct 31 01:59:04.981841 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:59:04.981810    4964 generic.go:356] "Generic (PLEG): container finished" podID="1ec66b87-7d07-45fd-8469-e1bd2c83f651" containerID="224f2098b2772bf59ef5570eb3e4e2d7efb5d0551b47d9ed009178abe42408bf" exitCode=0
Oct 31 01:59:04.982271 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:59:04.981883    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-c4d438-openshift-default-6d9d56f475-c995r" event={"ID":"1ec66b87-7d07-45fd-8469-e1bd2c83f651","Type":"ContainerDied","Data":"224f2098b2772bf59ef5570eb3e4e2d7efb5d0551b47d9ed009178abe42408bf"}
Oct 31 01:59:04.982354 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:59:04.982309    4964 scope.go:122] "RemoveContainer" containerID="224f2098b2772bf59ef5570eb3e4e2d7efb5d0551b47d9ed009178abe42408bf"
Oct 31 01:59:04.983977 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:59:04.983952    4964 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 01:59:04.984061 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:59:04.984016    4964 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 01:59:04.984061 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:59:04.984045    4964 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 01:59:05.284925 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:59:05.284821    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:59:06.014125 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:59:06.014092    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-c4d438-openshift-default-6d9d56f475-c995r" event={"ID":"1ec66b87-7d07-45fd-8469-e1bd2c83f651","Type":"ContainerStarted","Data":"4b04497f96486b8416976008199fb69286b68711327136c14cb456dc1ea3e4fd"}
Oct 31 01:59:06.125139 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:59:06.125099    4964 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-c4d438-openshift-default-6d9d56f475-c995r"
Oct 31 01:59:06.125585 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:59:06.125491    4964 patch_prober.go:29] interesting pod/e2e-shared-b-w3-c4d438-openshift-default-6d9d56f475-c995r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.180:15021/healthz/ready\": dial tcp 10.132.0.180:15021: connect: connection refused" start-of-body=
Oct 31 01:59:06.125759 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:59:06.125738    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-c4d438-openshift-default-6d9d56f475-c995r" podUID="1ec66b87-7d07-45fd-8469-e1bd2c83f651" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.180:15021/healthz/ready\": dial tcp 10.132.0.180:15021: connect: connection refused"
Oct 31 01:59:07.120011 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:59:07.119978    4964 patch_prober.go:29] interesting pod/e2e-shared-b-w3-c4d438-openshift-default-6d9d56f475-c995r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.180:15021/healthz/ready\": dial tcp 10.132.0.180:15021: connect: connection refused" start-of-body=
Oct 31 01:59:07.120662 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:59:07.120623    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-c4d438-openshift-default-6d9d56f475-c995r" podUID="1ec66b87-7d07-45fd-8469-e1bd2c83f651" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.180:15021/healthz/ready\": dial tcp 10.132.0.180:15021: connect: connection refused"
Oct 31 01:59:08.119899 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:59:08.119864    4964 patch_prober.go:29] interesting pod/e2e-shared-b-w3-c4d438-openshift-default-6d9d56f475-c995r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.180:15021/healthz/ready\": dial tcp 10.132.0.180:15021: connect: connection refused" start-of-body=
Oct 31 01:59:08.120113 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:59:08.119933    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-c4d438-openshift-default-6d9d56f475-c995r" podUID="1ec66b87-7d07-45fd-8469-e1bd2c83f651" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.180:15021/healthz/ready\": dial tcp 10.132.0.180:15021: connect: connection refused"
Oct 31 01:59:08.980337 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:59:08.980083    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-5632ea-b5b6857b8-5zj46"
Oct 31 01:59:09.124365 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:59:09.119680    4964 patch_prober.go:29] interesting pod/e2e-shared-b-w3-c4d438-openshift-default-6d9d56f475-c995r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.180:15021/healthz/ready\": dial tcp 10.132.0.180:15021: connect: connection refused" start-of-body=
Oct 31 01:59:09.124365 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:59:09.119755    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-c4d438-openshift-default-6d9d56f475-c995r" podUID="1ec66b87-7d07-45fd-8469-e1bd2c83f651" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.180:15021/healthz/ready\": dial tcp 10.132.0.180:15021: connect: connection refused"
Oct 31 01:59:10.120370 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:59:10.119713    4964 patch_prober.go:29] interesting pod/e2e-shared-b-w3-c4d438-openshift-default-6d9d56f475-c995r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.180:15021/healthz/ready\": dial tcp 10.132.0.180:15021: connect: connection refused" start-of-body=
Oct 31 01:59:10.120370 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:59:10.119788    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-c4d438-openshift-default-6d9d56f475-c995r" podUID="1ec66b87-7d07-45fd-8469-e1bd2c83f651" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.180:15021/healthz/ready\": dial tcp 10.132.0.180:15021: connect: connection refused"
Oct 31 01:59:10.131253 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:59:10.131217    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:59:10.755456 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:59:10.755423    4964 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-5jgz8"]
Oct 31 01:59:10.755643 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:59:10.755619    4964 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-5jgz8" podUID="59e32fbb-3cc2-40cc-b2f5-03b3c1d71870" containerName="istio-proxy" containerID="cri-o://910e66802e5f0d58fde77ba312626cc5b6b776d4112bae6c68076da98709d1fc" gracePeriod=30
Oct 31 01:59:10.827349 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:59:10.824879    4964 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-sbp55"]
Oct 31 01:59:10.827349 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:59:10.825077    4964 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-sbp55" podUID="31fe1f3a-23fe-4999-a8fc-d82b2185f0a7" containerName="istio-proxy" containerID="cri-o://e314062d6f06a9ccea3fa6eb68b01f515ffcfdc936fc7aae9d35bdd960fa44e6" gracePeriod=30
Oct 31 01:59:11.119258 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:59:11.119169    4964 patch_prober.go:29] interesting pod/e2e-shared-b-w3-c4d438-openshift-default-6d9d56f475-c995r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.180:15021/healthz/ready\": dial tcp 10.132.0.180:15021: connect: connection refused" start-of-body=
Oct 31 01:59:11.119258 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:59:11.119226    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-c4d438-openshift-default-6d9d56f475-c995r" podUID="1ec66b87-7d07-45fd-8469-e1bd2c83f651" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.180:15021/healthz/ready\": dial tcp 10.132.0.180:15021: connect: connection refused"
Oct 31 01:59:11.242367 ip-10-0-133-197 kubenswrapper[4964]: E1031 01:59:11.242303    4964 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q56x5" containerName="extractor"
Oct 31 01:59:12.124582 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:59:12.124552    4964 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-c4d438-openshift-default-6d9d56f475-c995r"
Oct 31 01:59:12.125031 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:59:12.125001    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-c4d438-openshift-default-6d9d56f475-c995r"
Oct 31 01:59:12.125750 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:59:12.125732    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-c4d438-openshift-default-6d9d56f475-c995r"
Oct 31 01:59:14.938022 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:59:14.937986    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:59:16.095920 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:59:16.095877    4964 generic.go:356] "Generic (PLEG): container finished" podID="59e32fbb-3cc2-40cc-b2f5-03b3c1d71870" containerID="910e66802e5f0d58fde77ba312626cc5b6b776d4112bae6c68076da98709d1fc" exitCode=0
Oct 31 01:59:16.096410 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:59:16.096122    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-5jgz8" event={"ID":"59e32fbb-3cc2-40cc-b2f5-03b3c1d71870","Type":"ContainerDied","Data":"910e66802e5f0d58fde77ba312626cc5b6b776d4112bae6c68076da98709d1fc"}
Oct 31 01:59:16.096410 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:59:16.096179    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-5jgz8" event={"ID":"59e32fbb-3cc2-40cc-b2f5-03b3c1d71870","Type":"ContainerDied","Data":"c3fa5861612e2b6a0f6a90100104409f96c115c9ead0a3233d3299a173f570ba"}
Oct 31 01:59:16.096410 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:59:16.096205    4964 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c3fa5861612e2b6a0f6a90100104409f96c115c9ead0a3233d3299a173f570ba"
Oct 31 01:59:16.098938 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:59:16.098914    4964 generic.go:356] "Generic (PLEG): container finished" podID="31fe1f3a-23fe-4999-a8fc-d82b2185f0a7" containerID="e314062d6f06a9ccea3fa6eb68b01f515ffcfdc936fc7aae9d35bdd960fa44e6" exitCode=0
Oct 31 01:59:16.099076 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:59:16.098987    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-sbp55" event={"ID":"31fe1f3a-23fe-4999-a8fc-d82b2185f0a7","Type":"ContainerDied","Data":"e314062d6f06a9ccea3fa6eb68b01f515ffcfdc936fc7aae9d35bdd960fa44e6"}
Oct 31 01:59:16.116632 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:59:16.116590    4964 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-5jgz8"
Oct 31 01:59:16.160041 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:59:16.160013    4964 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-sbp55"
Oct 31 01:59:16.296728 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:59:16.296645    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/59e32fbb-3cc2-40cc-b2f5-03b3c1d71870-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/59e32fbb-3cc2-40cc-b2f5-03b3c1d71870-workload-certs\") pod \"59e32fbb-3cc2-40cc-b2f5-03b3c1d71870\" (UID: \"59e32fbb-3cc2-40cc-b2f5-03b3c1d71870\") "
Oct 31 01:59:16.296728 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:59:16.296705    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/59e32fbb-3cc2-40cc-b2f5-03b3c1d71870-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/59e32fbb-3cc2-40cc-b2f5-03b3c1d71870-credential-socket\") pod \"59e32fbb-3cc2-40cc-b2f5-03b3c1d71870\" (UID: \"59e32fbb-3cc2-40cc-b2f5-03b3c1d71870\") "
Oct 31 01:59:16.297039 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:59:16.296770    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/59e32fbb-3cc2-40cc-b2f5-03b3c1d71870-kube-api-access-5z2q6\" (UniqueName: \"kubernetes.io/projected/59e32fbb-3cc2-40cc-b2f5-03b3c1d71870-kube-api-access-5z2q6\") pod \"59e32fbb-3cc2-40cc-b2f5-03b3c1d71870\" (UID: \"59e32fbb-3cc2-40cc-b2f5-03b3c1d71870\") "
Oct 31 01:59:16.297039 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:59:16.296826    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/59e32fbb-3cc2-40cc-b2f5-03b3c1d71870-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/59e32fbb-3cc2-40cc-b2f5-03b3c1d71870-istio-envoy\") pod \"59e32fbb-3cc2-40cc-b2f5-03b3c1d71870\" (UID: \"59e32fbb-3cc2-40cc-b2f5-03b3c1d71870\") "
Oct 31 01:59:16.297039 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:59:16.296944    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59e32fbb-3cc2-40cc-b2f5-03b3c1d71870-workload-certs" pod "59e32fbb-3cc2-40cc-b2f5-03b3c1d71870" (UID: "59e32fbb-3cc2-40cc-b2f5-03b3c1d71870"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:59:16.297225 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:59:16.297080    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59e32fbb-3cc2-40cc-b2f5-03b3c1d71870-credential-socket" pod "59e32fbb-3cc2-40cc-b2f5-03b3c1d71870" (UID: "59e32fbb-3cc2-40cc-b2f5-03b3c1d71870"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:59:16.297290 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:59:16.297249    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/59e32fbb-3cc2-40cc-b2f5-03b3c1d71870-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/59e32fbb-3cc2-40cc-b2f5-03b3c1d71870-istio-podinfo\") pod \"59e32fbb-3cc2-40cc-b2f5-03b3c1d71870\" (UID: \"59e32fbb-3cc2-40cc-b2f5-03b3c1d71870\") "
Oct 31 01:59:16.297417 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:59:16.297356    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/31fe1f3a-23fe-4999-a8fc-d82b2185f0a7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/31fe1f3a-23fe-4999-a8fc-d82b2185f0a7-istiod-ca-cert\") pod \"31fe1f3a-23fe-4999-a8fc-d82b2185f0a7\" (UID: \"31fe1f3a-23fe-4999-a8fc-d82b2185f0a7\") "
Oct 31 01:59:16.297482 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:59:16.297423    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/31fe1f3a-23fe-4999-a8fc-d82b2185f0a7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/31fe1f3a-23fe-4999-a8fc-d82b2185f0a7-workload-certs\") pod \"31fe1f3a-23fe-4999-a8fc-d82b2185f0a7\" (UID: \"31fe1f3a-23fe-4999-a8fc-d82b2185f0a7\") "
Oct 31 01:59:16.297551 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:59:16.297533    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/59e32fbb-3cc2-40cc-b2f5-03b3c1d71870-istio-token\" (UniqueName: \"kubernetes.io/projected/59e32fbb-3cc2-40cc-b2f5-03b3c1d71870-istio-token\") pod \"59e32fbb-3cc2-40cc-b2f5-03b3c1d71870\" (UID: \"59e32fbb-3cc2-40cc-b2f5-03b3c1d71870\") "
Oct 31 01:59:16.297609 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:59:16.297572    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/31fe1f3a-23fe-4999-a8fc-d82b2185f0a7-istio-token\" (UniqueName: \"kubernetes.io/projected/31fe1f3a-23fe-4999-a8fc-d82b2185f0a7-istio-token\") pod \"31fe1f3a-23fe-4999-a8fc-d82b2185f0a7\" (UID: \"31fe1f3a-23fe-4999-a8fc-d82b2185f0a7\") "
Oct 31 01:59:16.297688 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:59:16.297674    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/59e32fbb-3cc2-40cc-b2f5-03b3c1d71870-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/59e32fbb-3cc2-40cc-b2f5-03b3c1d71870-istiod-ca-cert\") pod \"59e32fbb-3cc2-40cc-b2f5-03b3c1d71870\" (UID: \"59e32fbb-3cc2-40cc-b2f5-03b3c1d71870\") "
Oct 31 01:59:16.297747 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:59:16.297718    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/31fe1f3a-23fe-4999-a8fc-d82b2185f0a7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/31fe1f3a-23fe-4999-a8fc-d82b2185f0a7-istio-envoy\") pod \"31fe1f3a-23fe-4999-a8fc-d82b2185f0a7\" (UID: \"31fe1f3a-23fe-4999-a8fc-d82b2185f0a7\") "
Oct 31 01:59:16.297819 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:59:16.297801    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/31fe1f3a-23fe-4999-a8fc-d82b2185f0a7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/31fe1f3a-23fe-4999-a8fc-d82b2185f0a7-workload-socket\") pod \"31fe1f3a-23fe-4999-a8fc-d82b2185f0a7\" (UID: \"31fe1f3a-23fe-4999-a8fc-d82b2185f0a7\") "
Oct 31 01:59:16.297880 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:59:16.297865    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/31fe1f3a-23fe-4999-a8fc-d82b2185f0a7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/31fe1f3a-23fe-4999-a8fc-d82b2185f0a7-credential-socket\") pod \"31fe1f3a-23fe-4999-a8fc-d82b2185f0a7\" (UID: \"31fe1f3a-23fe-4999-a8fc-d82b2185f0a7\") "
Oct 31 01:59:16.297958 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:59:16.297946    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/59e32fbb-3cc2-40cc-b2f5-03b3c1d71870-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/59e32fbb-3cc2-40cc-b2f5-03b3c1d71870-istio-data\") pod \"59e32fbb-3cc2-40cc-b2f5-03b3c1d71870\" (UID: \"59e32fbb-3cc2-40cc-b2f5-03b3c1d71870\") "
Oct 31 01:59:16.298026 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:59:16.298005    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/31fe1f3a-23fe-4999-a8fc-d82b2185f0a7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/31fe1f3a-23fe-4999-a8fc-d82b2185f0a7-istio-data\") pod \"31fe1f3a-23fe-4999-a8fc-d82b2185f0a7\" (UID: \"31fe1f3a-23fe-4999-a8fc-d82b2185f0a7\") "
Oct 31 01:59:16.298090 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:59:16.298055    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/59e32fbb-3cc2-40cc-b2f5-03b3c1d71870-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/59e32fbb-3cc2-40cc-b2f5-03b3c1d71870-workload-socket\") pod \"59e32fbb-3cc2-40cc-b2f5-03b3c1d71870\" (UID: \"59e32fbb-3cc2-40cc-b2f5-03b3c1d71870\") "
Oct 31 01:59:16.298168 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:59:16.298136    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/31fe1f3a-23fe-4999-a8fc-d82b2185f0a7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/31fe1f3a-23fe-4999-a8fc-d82b2185f0a7-istio-podinfo\") pod \"31fe1f3a-23fe-4999-a8fc-d82b2185f0a7\" (UID: \"31fe1f3a-23fe-4999-a8fc-d82b2185f0a7\") "
Oct 31 01:59:16.298227 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:59:16.298209    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/59e32fbb-3cc2-40cc-b2f5-03b3c1d71870-istiod-ca-cert" pod "59e32fbb-3cc2-40cc-b2f5-03b3c1d71870" (UID: "59e32fbb-3cc2-40cc-b2f5-03b3c1d71870"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:59:16.298360 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:59:16.298292    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/31fe1f3a-23fe-4999-a8fc-d82b2185f0a7-kube-api-access-4k7vl\" (UniqueName: \"kubernetes.io/projected/31fe1f3a-23fe-4999-a8fc-d82b2185f0a7-kube-api-access-4k7vl\") pod \"31fe1f3a-23fe-4999-a8fc-d82b2185f0a7\" (UID: \"31fe1f3a-23fe-4999-a8fc-d82b2185f0a7\") "
Oct 31 01:59:16.298703 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:59:16.298677    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/31fe1f3a-23fe-4999-a8fc-d82b2185f0a7-istiod-ca-cert" pod "31fe1f3a-23fe-4999-a8fc-d82b2185f0a7" (UID: "31fe1f3a-23fe-4999-a8fc-d82b2185f0a7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:59:16.298947 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:59:16.298922    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31fe1f3a-23fe-4999-a8fc-d82b2185f0a7-workload-certs" pod "31fe1f3a-23fe-4999-a8fc-d82b2185f0a7" (UID: "31fe1f3a-23fe-4999-a8fc-d82b2185f0a7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:59:16.299148 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:59:16.299124    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59e32fbb-3cc2-40cc-b2f5-03b3c1d71870-istio-data" pod "59e32fbb-3cc2-40cc-b2f5-03b3c1d71870" (UID: "59e32fbb-3cc2-40cc-b2f5-03b3c1d71870"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:59:16.299420 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:59:16.299153    4964 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/59e32fbb-3cc2-40cc-b2f5-03b3c1d71870-istiod-ca-cert\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:59:16.299420 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:59:16.299175    4964 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/59e32fbb-3cc2-40cc-b2f5-03b3c1d71870-workload-certs\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:59:16.299420 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:59:16.299190    4964 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/59e32fbb-3cc2-40cc-b2f5-03b3c1d71870-credential-socket\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:59:16.299420 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:59:16.299204    4964 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/31fe1f3a-23fe-4999-a8fc-d82b2185f0a7-istiod-ca-cert\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:59:16.299420 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:59:16.299237    4964 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/31fe1f3a-23fe-4999-a8fc-d82b2185f0a7-workload-certs\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:59:16.299712 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:59:16.299475    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31fe1f3a-23fe-4999-a8fc-d82b2185f0a7-workload-socket" pod "31fe1f3a-23fe-4999-a8fc-d82b2185f0a7" (UID: "31fe1f3a-23fe-4999-a8fc-d82b2185f0a7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:59:16.299847 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:59:16.299806    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31fe1f3a-23fe-4999-a8fc-d82b2185f0a7-istio-data" pod "31fe1f3a-23fe-4999-a8fc-d82b2185f0a7" (UID: "31fe1f3a-23fe-4999-a8fc-d82b2185f0a7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:59:16.299847 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:59:16.299836    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59e32fbb-3cc2-40cc-b2f5-03b3c1d71870-workload-socket" pod "59e32fbb-3cc2-40cc-b2f5-03b3c1d71870" (UID: "59e32fbb-3cc2-40cc-b2f5-03b3c1d71870"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:59:16.300040 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:59:16.299943    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31fe1f3a-23fe-4999-a8fc-d82b2185f0a7-credential-socket" pod "31fe1f3a-23fe-4999-a8fc-d82b2185f0a7" (UID: "31fe1f3a-23fe-4999-a8fc-d82b2185f0a7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:59:16.301754 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:59:16.301696    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/31fe1f3a-23fe-4999-a8fc-d82b2185f0a7-kube-api-access-4k7vl" pod "31fe1f3a-23fe-4999-a8fc-d82b2185f0a7" (UID: "31fe1f3a-23fe-4999-a8fc-d82b2185f0a7"). InnerVolumeSpecName "kube-api-access-4k7vl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:59:16.302381 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:59:16.302356    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/31fe1f3a-23fe-4999-a8fc-d82b2185f0a7-istio-token" pod "31fe1f3a-23fe-4999-a8fc-d82b2185f0a7" (UID: "31fe1f3a-23fe-4999-a8fc-d82b2185f0a7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:59:16.302797 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:59:16.302749    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59e32fbb-3cc2-40cc-b2f5-03b3c1d71870-istio-envoy" pod "59e32fbb-3cc2-40cc-b2f5-03b3c1d71870" (UID: "59e32fbb-3cc2-40cc-b2f5-03b3c1d71870"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:59:16.304009 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:59:16.303976    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31fe1f3a-23fe-4999-a8fc-d82b2185f0a7-istio-envoy" pod "31fe1f3a-23fe-4999-a8fc-d82b2185f0a7" (UID: "31fe1f3a-23fe-4999-a8fc-d82b2185f0a7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:59:16.304110 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:59:16.304013    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/59e32fbb-3cc2-40cc-b2f5-03b3c1d71870-kube-api-access-5z2q6" pod "59e32fbb-3cc2-40cc-b2f5-03b3c1d71870" (UID: "59e32fbb-3cc2-40cc-b2f5-03b3c1d71870"). InnerVolumeSpecName "kube-api-access-5z2q6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:59:16.304177 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:59:16.304140    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/31fe1f3a-23fe-4999-a8fc-d82b2185f0a7-istio-podinfo" pod "31fe1f3a-23fe-4999-a8fc-d82b2185f0a7" (UID: "31fe1f3a-23fe-4999-a8fc-d82b2185f0a7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 01:59:16.304271 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:59:16.304238    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/59e32fbb-3cc2-40cc-b2f5-03b3c1d71870-istio-token" pod "59e32fbb-3cc2-40cc-b2f5-03b3c1d71870" (UID: "59e32fbb-3cc2-40cc-b2f5-03b3c1d71870"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:59:16.304670 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:59:16.304645    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/59e32fbb-3cc2-40cc-b2f5-03b3c1d71870-istio-podinfo" pod "59e32fbb-3cc2-40cc-b2f5-03b3c1d71870" (UID: "59e32fbb-3cc2-40cc-b2f5-03b3c1d71870"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 01:59:16.400538 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:59:16.400505    4964 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/59e32fbb-3cc2-40cc-b2f5-03b3c1d71870-istio-token\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:59:16.400538 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:59:16.400538    4964 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/31fe1f3a-23fe-4999-a8fc-d82b2185f0a7-istio-token\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:59:16.400804 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:59:16.400549    4964 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/31fe1f3a-23fe-4999-a8fc-d82b2185f0a7-istio-envoy\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:59:16.400804 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:59:16.400562    4964 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/31fe1f3a-23fe-4999-a8fc-d82b2185f0a7-workload-socket\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:59:16.400804 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:59:16.400574    4964 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/31fe1f3a-23fe-4999-a8fc-d82b2185f0a7-credential-socket\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:59:16.400804 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:59:16.400586    4964 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/59e32fbb-3cc2-40cc-b2f5-03b3c1d71870-istio-data\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:59:16.400804 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:59:16.400597    4964 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/31fe1f3a-23fe-4999-a8fc-d82b2185f0a7-istio-data\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:59:16.400804 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:59:16.400608    4964 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/59e32fbb-3cc2-40cc-b2f5-03b3c1d71870-workload-socket\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:59:16.400804 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:59:16.400619    4964 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/31fe1f3a-23fe-4999-a8fc-d82b2185f0a7-istio-podinfo\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:59:16.400804 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:59:16.400630    4964 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4k7vl\" (UniqueName: \"kubernetes.io/projected/31fe1f3a-23fe-4999-a8fc-d82b2185f0a7-kube-api-access-4k7vl\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:59:16.400804 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:59:16.400643    4964 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5z2q6\" (UniqueName: \"kubernetes.io/projected/59e32fbb-3cc2-40cc-b2f5-03b3c1d71870-kube-api-access-5z2q6\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:59:16.400804 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:59:16.400654    4964 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/59e32fbb-3cc2-40cc-b2f5-03b3c1d71870-istio-envoy\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:59:16.400804 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:59:16.400667    4964 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/59e32fbb-3cc2-40cc-b2f5-03b3c1d71870-istio-podinfo\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:59:17.106998 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:59:17.106918    4964 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-sbp55"
Oct 31 01:59:17.106998 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:59:17.106936    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-sbp55" event={"ID":"31fe1f3a-23fe-4999-a8fc-d82b2185f0a7","Type":"ContainerDied","Data":"dbed79999a734addb7df91c92771720443546bba4748514a8f2939fc4582ba15"}
Oct 31 01:59:17.107536 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:59:17.107048    4964 scope.go:122] "RemoveContainer" containerID="e314062d6f06a9ccea3fa6eb68b01f515ffcfdc936fc7aae9d35bdd960fa44e6"
Oct 31 01:59:17.107536 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:59:17.107263    4964 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-5jgz8"
Oct 31 01:59:17.270622 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:59:17.270592    4964 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-sbp55"]
Oct 31 01:59:17.320424 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:59:17.320388    4964 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-sbp55"]
Oct 31 01:59:17.428236 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:59:17.428148    4964 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-5jgz8"]
Oct 31 01:59:17.432328 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:59:17.432297    4964 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-5jgz8"]
Oct 31 01:59:17.779523 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:59:17.779492    4964 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="31fe1f3a-23fe-4999-a8fc-d82b2185f0a7" path="/var/lib/kubelet/pods/31fe1f3a-23fe-4999-a8fc-d82b2185f0a7/volumes"
Oct 31 01:59:17.780032 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:59:17.780013    4964 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="59e32fbb-3cc2-40cc-b2f5-03b3c1d71870" path="/var/lib/kubelet/pods/59e32fbb-3cc2-40cc-b2f5-03b3c1d71870/volumes"
Oct 31 01:59:19.634028 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:59:19.633995    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:59:21.009682 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:59:21.009648    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:59:24.702057 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:59:24.702018    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:59:29.506366 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:59:29.506300    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:59:34.224774 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:59:34.224742    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:59:39.096559 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:59:39.096523    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:59:41.239644 ip-10-0-133-197 kubenswrapper[4964]: E1031 01:59:41.239605    4964 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q56x5" containerName="extractor"
Oct 31 01:59:41.240021 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:59:41.239814    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:59:43.904245 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:59:43.904117    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:59:47.283398 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:59:47.283362    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:59:48.791593 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:59:48.791547    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:59:51.729916 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:59:51.729885    4964 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w0-98b70c/auth-test-model-98b70c-kserve-57ddcb7c85-5wdxx"]
Oct 31 01:59:51.730598 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:59:51.730582    4964 cpu_manager.go:414] "containerMap: removing container" podUID="31fe1f3a-23fe-4999-a8fc-d82b2185f0a7" containerName="istio-proxy"
Oct 31 01:59:51.730663 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:59:51.730601    4964 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="31fe1f3a-23fe-4999-a8fc-d82b2185f0a7" containerName="istio-proxy"
Oct 31 01:59:51.730663 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:59:51.730641    4964 cpu_manager.go:414] "containerMap: removing container" podUID="59e32fbb-3cc2-40cc-b2f5-03b3c1d71870" containerName="istio-proxy"
Oct 31 01:59:51.730663 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:59:51.730646    4964 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="59e32fbb-3cc2-40cc-b2f5-03b3c1d71870" containerName="istio-proxy"
Oct 31 01:59:51.730780 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:59:51.730767    4964 memory_manager.go:365] "RemoveStaleState removing state" podUID="59e32fbb-3cc2-40cc-b2f5-03b3c1d71870" containerName="istio-proxy"
Oct 31 01:59:51.730819 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:59:51.730785    4964 memory_manager.go:365] "RemoveStaleState removing state" podUID="31fe1f3a-23fe-4999-a8fc-d82b2185f0a7" containerName="istio-proxy"
Oct 31 01:59:51.734225 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:59:51.734208    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-98b70c/auth-test-model-98b70c-kserve-57ddcb7c85-5wdxx"
Oct 31 01:59:51.827755 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:59:51.827723    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5b8433e9-df92-402c-9873-83646d9bfee5-tmp-dir\") pod \"auth-test-model-98b70c-kserve-57ddcb7c85-5wdxx\" (UID: \"5b8433e9-df92-402c-9873-83646d9bfee5\") " pod="ai-tenant-e2e-shared-a-w0-98b70c/auth-test-model-98b70c-kserve-57ddcb7c85-5wdxx"
Oct 31 01:59:51.827922 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:59:51.827779    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5b8433e9-df92-402c-9873-83646d9bfee5-model-cache\") pod \"auth-test-model-98b70c-kserve-57ddcb7c85-5wdxx\" (UID: \"5b8433e9-df92-402c-9873-83646d9bfee5\") " pod="ai-tenant-e2e-shared-a-w0-98b70c/auth-test-model-98b70c-kserve-57ddcb7c85-5wdxx"
Oct 31 01:59:51.827922 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:59:51.827907    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5b8433e9-df92-402c-9873-83646d9bfee5-dshm\") pod \"auth-test-model-98b70c-kserve-57ddcb7c85-5wdxx\" (UID: \"5b8433e9-df92-402c-9873-83646d9bfee5\") " pod="ai-tenant-e2e-shared-a-w0-98b70c/auth-test-model-98b70c-kserve-57ddcb7c85-5wdxx"
Oct 31 01:59:51.828006 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:59:51.827994    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5b8433e9-df92-402c-9873-83646d9bfee5-tls-certs\") pod \"auth-test-model-98b70c-kserve-57ddcb7c85-5wdxx\" (UID: \"5b8433e9-df92-402c-9873-83646d9bfee5\") " pod="ai-tenant-e2e-shared-a-w0-98b70c/auth-test-model-98b70c-kserve-57ddcb7c85-5wdxx"
Oct 31 01:59:51.828215 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:59:51.828196    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5b8433e9-df92-402c-9873-83646d9bfee5-home\") pod \"auth-test-model-98b70c-kserve-57ddcb7c85-5wdxx\" (UID: \"5b8433e9-df92-402c-9873-83646d9bfee5\") " pod="ai-tenant-e2e-shared-a-w0-98b70c/auth-test-model-98b70c-kserve-57ddcb7c85-5wdxx"
Oct 31 01:59:51.828254 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:59:51.828233    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wld8g\" (UniqueName: \"kubernetes.io/projected/5b8433e9-df92-402c-9873-83646d9bfee5-kube-api-access-wld8g\") pod \"auth-test-model-98b70c-kserve-57ddcb7c85-5wdxx\" (UID: \"5b8433e9-df92-402c-9873-83646d9bfee5\") " pod="ai-tenant-e2e-shared-a-w0-98b70c/auth-test-model-98b70c-kserve-57ddcb7c85-5wdxx"
Oct 31 01:59:51.835653 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:59:51.835628    4964 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w0-98b70c\"/\"kube-root-ca.crt\""
Oct 31 01:59:51.835653 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:59:51.835631    4964 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w0-98b70c\"/\"openshift-service-ca.crt\""
Oct 31 01:59:51.835906 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:59:51.835739    4964 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w0-98b70c\"/\"default-dockercfg-46qdr\""
Oct 31 01:59:51.835906 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:59:51.835836    4964 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w0-98b70c\"/\"auth-test-model-98b70c-kserve-self-signed-certs\""
Oct 31 01:59:51.839134 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:59:51.839116    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w0-98b70c/auth-test-model-98b70c-kserve-57ddcb7c85-5wdxx"]
Oct 31 01:59:51.893869 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:59:51.893804    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:59:51.931569 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:59:51.930229    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5b8433e9-df92-402c-9873-83646d9bfee5-dshm\") pod \"auth-test-model-98b70c-kserve-57ddcb7c85-5wdxx\" (UID: \"5b8433e9-df92-402c-9873-83646d9bfee5\") " pod="ai-tenant-e2e-shared-a-w0-98b70c/auth-test-model-98b70c-kserve-57ddcb7c85-5wdxx"
Oct 31 01:59:51.931569 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:59:51.930356    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5b8433e9-df92-402c-9873-83646d9bfee5-tls-certs\") pod \"auth-test-model-98b70c-kserve-57ddcb7c85-5wdxx\" (UID: \"5b8433e9-df92-402c-9873-83646d9bfee5\") " pod="ai-tenant-e2e-shared-a-w0-98b70c/auth-test-model-98b70c-kserve-57ddcb7c85-5wdxx"
Oct 31 01:59:51.931569 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:59:51.930467    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5b8433e9-df92-402c-9873-83646d9bfee5-home\") pod \"auth-test-model-98b70c-kserve-57ddcb7c85-5wdxx\" (UID: \"5b8433e9-df92-402c-9873-83646d9bfee5\") " pod="ai-tenant-e2e-shared-a-w0-98b70c/auth-test-model-98b70c-kserve-57ddcb7c85-5wdxx"
Oct 31 01:59:51.931569 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:59:51.930500    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wld8g\" (UniqueName: \"kubernetes.io/projected/5b8433e9-df92-402c-9873-83646d9bfee5-kube-api-access-wld8g\") pod \"auth-test-model-98b70c-kserve-57ddcb7c85-5wdxx\" (UID: \"5b8433e9-df92-402c-9873-83646d9bfee5\") " pod="ai-tenant-e2e-shared-a-w0-98b70c/auth-test-model-98b70c-kserve-57ddcb7c85-5wdxx"
Oct 31 01:59:51.931569 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:59:51.930563    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5b8433e9-df92-402c-9873-83646d9bfee5-tmp-dir\") pod \"auth-test-model-98b70c-kserve-57ddcb7c85-5wdxx\" (UID: \"5b8433e9-df92-402c-9873-83646d9bfee5\") " pod="ai-tenant-e2e-shared-a-w0-98b70c/auth-test-model-98b70c-kserve-57ddcb7c85-5wdxx"
Oct 31 01:59:51.931569 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:59:51.930630    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5b8433e9-df92-402c-9873-83646d9bfee5-model-cache\") pod \"auth-test-model-98b70c-kserve-57ddcb7c85-5wdxx\" (UID: \"5b8433e9-df92-402c-9873-83646d9bfee5\") " pod="ai-tenant-e2e-shared-a-w0-98b70c/auth-test-model-98b70c-kserve-57ddcb7c85-5wdxx"
Oct 31 01:59:51.931569 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:59:51.931136    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5b8433e9-df92-402c-9873-83646d9bfee5-model-cache\") pod \"auth-test-model-98b70c-kserve-57ddcb7c85-5wdxx\" (UID: \"5b8433e9-df92-402c-9873-83646d9bfee5\") " pod="ai-tenant-e2e-shared-a-w0-98b70c/auth-test-model-98b70c-kserve-57ddcb7c85-5wdxx"
Oct 31 01:59:51.941485 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:59:51.934602    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5b8433e9-df92-402c-9873-83646d9bfee5-tls-certs\") pod \"auth-test-model-98b70c-kserve-57ddcb7c85-5wdxx\" (UID: \"5b8433e9-df92-402c-9873-83646d9bfee5\") " pod="ai-tenant-e2e-shared-a-w0-98b70c/auth-test-model-98b70c-kserve-57ddcb7c85-5wdxx"
Oct 31 01:59:51.941485 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:59:51.934893    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5b8433e9-df92-402c-9873-83646d9bfee5-home\") pod \"auth-test-model-98b70c-kserve-57ddcb7c85-5wdxx\" (UID: \"5b8433e9-df92-402c-9873-83646d9bfee5\") " pod="ai-tenant-e2e-shared-a-w0-98b70c/auth-test-model-98b70c-kserve-57ddcb7c85-5wdxx"
Oct 31 01:59:51.941485 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:59:51.935473    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5b8433e9-df92-402c-9873-83646d9bfee5-tmp-dir\") pod \"auth-test-model-98b70c-kserve-57ddcb7c85-5wdxx\" (UID: \"5b8433e9-df92-402c-9873-83646d9bfee5\") " pod="ai-tenant-e2e-shared-a-w0-98b70c/auth-test-model-98b70c-kserve-57ddcb7c85-5wdxx"
Oct 31 01:59:51.942153 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:59:51.942127    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5b8433e9-df92-402c-9873-83646d9bfee5-dshm\") pod \"auth-test-model-98b70c-kserve-57ddcb7c85-5wdxx\" (UID: \"5b8433e9-df92-402c-9873-83646d9bfee5\") " pod="ai-tenant-e2e-shared-a-w0-98b70c/auth-test-model-98b70c-kserve-57ddcb7c85-5wdxx"
Oct 31 01:59:51.991353 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:59:51.991275    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wld8g\" (UniqueName: \"kubernetes.io/projected/5b8433e9-df92-402c-9873-83646d9bfee5-kube-api-access-wld8g\") pod \"auth-test-model-98b70c-kserve-57ddcb7c85-5wdxx\" (UID: \"5b8433e9-df92-402c-9873-83646d9bfee5\") " pod="ai-tenant-e2e-shared-a-w0-98b70c/auth-test-model-98b70c-kserve-57ddcb7c85-5wdxx"
Oct 31 01:59:52.046369 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:59:52.046347    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-98b70c/auth-test-model-98b70c-kserve-57ddcb7c85-5wdxx"
Oct 31 01:59:52.429731 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:59:52.429705    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w0-98b70c/auth-test-model-98b70c-kserve-57ddcb7c85-5wdxx"]
Oct 31 01:59:52.431742 ip-10-0-133-197 kubenswrapper[4964]: W1031 01:59:52.431715    4964 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5b8433e9_df92_402c_9873_83646d9bfee5.slice/crio-29e76f1963dcc8c136738558208bddf61d11306563db50e9f0fedf4223536d8d WatchSource:0}: Error finding container 29e76f1963dcc8c136738558208bddf61d11306563db50e9f0fedf4223536d8d: Status 404 returned error can't find the container with id 29e76f1963dcc8c136738558208bddf61d11306563db50e9f0fedf4223536d8d
Oct 31 01:59:52.775724 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:59:52.775683    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:59:53.356045 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:59:53.355956    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-98b70c/auth-test-model-98b70c-kserve-57ddcb7c85-5wdxx" event={"ID":"5b8433e9-df92-402c-9873-83646d9bfee5","Type":"ContainerStarted","Data":"c4bdb29cfd0649fdd6b43e6f66b33166f42516fd9a1c5be67cce65aaac84fba2"}
Oct 31 01:59:53.356045 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:59:53.356004    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-98b70c/auth-test-model-98b70c-kserve-57ddcb7c85-5wdxx" event={"ID":"5b8433e9-df92-402c-9873-83646d9bfee5","Type":"ContainerStarted","Data":"29e76f1963dcc8c136738558208bddf61d11306563db50e9f0fedf4223536d8d"}
Oct 31 01:59:53.471292 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:59:53.471235    4964 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w0-98b70c/auth-test-model-98b70c-kserve-57ddcb7c85-5wdxx" podStartSLOduration=2.47121668 podStartE2EDuration="2.47121668s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:59:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:59:53.46865852 +0000 UTC m=+3361.945224900" watchObservedRunningTime="2026-10-31 01:59:53.47121668 +0000 UTC m=+3361.947783065"
Oct 31 01:59:53.815870 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:59:53.815193    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 01:59:54.047516 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:59:54.047483    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w0-98b70c/auth-test-model-98b70c-kserve-57ddcb7c85-5wdxx"
Oct 31 01:59:58.608653 ip-10-0-133-197 kubenswrapper[4964]: I1031 01:59:58.607969    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 02:00:02.046586 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:00:02.046500    4964 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w0-98b70c/auth-test-model-98b70c-kserve-57ddcb7c85-5wdxx"
Oct 31 02:00:02.059279 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:00:02.059254    4964 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w0-98b70c/auth-test-model-98b70c-kserve-57ddcb7c85-5wdxx"
Oct 31 02:00:02.435584 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:00:02.435504    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w0-98b70c/auth-test-model-98b70c-kserve-57ddcb7c85-5wdxx"
Oct 31 02:00:03.435302 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:00:03.435259    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 02:00:08.218995 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:00:08.218957    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 02:00:09.712664 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:00:09.712617    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 02:00:10.261294 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:00:10.261259    4964 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-explicit-c4be41-openshift-default-686574f97c-phvtc"]
Oct 31 02:00:10.267150 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:00:10.267125    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-c4be41-openshift-default-686574f97c-phvtc"
Oct 31 02:00:10.319394 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:00:10.319362    4964 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-explicit-c4be41-openshift-default-dockercfg-z2ztp\""
Oct 31 02:00:10.373833 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:00:10.373795    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-explicit-c4be41-openshift-default-686574f97c-phvtc"]
Oct 31 02:00:10.396791 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:00:10.396731    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 02:00:10.469840 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:00:10.469801    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8508d596-d89b-4f67-9a57-f6ba4890ce5e-istio-podinfo\") pod \"e2e-explicit-c4be41-openshift-default-686574f97c-phvtc\" (UID: \"8508d596-d89b-4f67-9a57-f6ba4890ce5e\") " pod="openshift-ingress/e2e-explicit-c4be41-openshift-default-686574f97c-phvtc"
Oct 31 02:00:10.475336 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:00:10.474381    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8508d596-d89b-4f67-9a57-f6ba4890ce5e-istiod-ca-cert\") pod \"e2e-explicit-c4be41-openshift-default-686574f97c-phvtc\" (UID: \"8508d596-d89b-4f67-9a57-f6ba4890ce5e\") " pod="openshift-ingress/e2e-explicit-c4be41-openshift-default-686574f97c-phvtc"
Oct 31 02:00:10.481396 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:00:10.481366    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8508d596-d89b-4f67-9a57-f6ba4890ce5e-workload-socket\") pod \"e2e-explicit-c4be41-openshift-default-686574f97c-phvtc\" (UID: \"8508d596-d89b-4f67-9a57-f6ba4890ce5e\") " pod="openshift-ingress/e2e-explicit-c4be41-openshift-default-686574f97c-phvtc"
Oct 31 02:00:10.481695 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:00:10.481670    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8508d596-d89b-4f67-9a57-f6ba4890ce5e-credential-socket\") pod \"e2e-explicit-c4be41-openshift-default-686574f97c-phvtc\" (UID: \"8508d596-d89b-4f67-9a57-f6ba4890ce5e\") " pod="openshift-ingress/e2e-explicit-c4be41-openshift-default-686574f97c-phvtc"
Oct 31 02:00:10.481890 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:00:10.481870    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8508d596-d89b-4f67-9a57-f6ba4890ce5e-istio-data\") pod \"e2e-explicit-c4be41-openshift-default-686574f97c-phvtc\" (UID: \"8508d596-d89b-4f67-9a57-f6ba4890ce5e\") " pod="openshift-ingress/e2e-explicit-c4be41-openshift-default-686574f97c-phvtc"
Oct 31 02:00:10.482119 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:00:10.482101    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hzdn9\" (UniqueName: \"kubernetes.io/projected/8508d596-d89b-4f67-9a57-f6ba4890ce5e-kube-api-access-hzdn9\") pod \"e2e-explicit-c4be41-openshift-default-686574f97c-phvtc\" (UID: \"8508d596-d89b-4f67-9a57-f6ba4890ce5e\") " pod="openshift-ingress/e2e-explicit-c4be41-openshift-default-686574f97c-phvtc"
Oct 31 02:00:10.482308 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:00:10.482293    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8508d596-d89b-4f67-9a57-f6ba4890ce5e-istio-envoy\") pod \"e2e-explicit-c4be41-openshift-default-686574f97c-phvtc\" (UID: \"8508d596-d89b-4f67-9a57-f6ba4890ce5e\") " pod="openshift-ingress/e2e-explicit-c4be41-openshift-default-686574f97c-phvtc"
Oct 31 02:00:10.482540 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:00:10.482524    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8508d596-d89b-4f67-9a57-f6ba4890ce5e-istio-token\") pod \"e2e-explicit-c4be41-openshift-default-686574f97c-phvtc\" (UID: \"8508d596-d89b-4f67-9a57-f6ba4890ce5e\") " pod="openshift-ingress/e2e-explicit-c4be41-openshift-default-686574f97c-phvtc"
Oct 31 02:00:10.482689 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:00:10.482676    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8508d596-d89b-4f67-9a57-f6ba4890ce5e-workload-certs\") pod \"e2e-explicit-c4be41-openshift-default-686574f97c-phvtc\" (UID: \"8508d596-d89b-4f67-9a57-f6ba4890ce5e\") " pod="openshift-ingress/e2e-explicit-c4be41-openshift-default-686574f97c-phvtc"
Oct 31 02:00:10.584814 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:00:10.584713    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8508d596-d89b-4f67-9a57-f6ba4890ce5e-workload-socket\") pod \"e2e-explicit-c4be41-openshift-default-686574f97c-phvtc\" (UID: \"8508d596-d89b-4f67-9a57-f6ba4890ce5e\") " pod="openshift-ingress/e2e-explicit-c4be41-openshift-default-686574f97c-phvtc"
Oct 31 02:00:10.584814 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:00:10.584790    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8508d596-d89b-4f67-9a57-f6ba4890ce5e-credential-socket\") pod \"e2e-explicit-c4be41-openshift-default-686574f97c-phvtc\" (UID: \"8508d596-d89b-4f67-9a57-f6ba4890ce5e\") " pod="openshift-ingress/e2e-explicit-c4be41-openshift-default-686574f97c-phvtc"
Oct 31 02:00:10.585138 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:00:10.584814    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8508d596-d89b-4f67-9a57-f6ba4890ce5e-istio-data\") pod \"e2e-explicit-c4be41-openshift-default-686574f97c-phvtc\" (UID: \"8508d596-d89b-4f67-9a57-f6ba4890ce5e\") " pod="openshift-ingress/e2e-explicit-c4be41-openshift-default-686574f97c-phvtc"
Oct 31 02:00:10.585138 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:00:10.585129    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hzdn9\" (UniqueName: \"kubernetes.io/projected/8508d596-d89b-4f67-9a57-f6ba4890ce5e-kube-api-access-hzdn9\") pod \"e2e-explicit-c4be41-openshift-default-686574f97c-phvtc\" (UID: \"8508d596-d89b-4f67-9a57-f6ba4890ce5e\") " pod="openshift-ingress/e2e-explicit-c4be41-openshift-default-686574f97c-phvtc"
Oct 31 02:00:10.585368 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:00:10.585192    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8508d596-d89b-4f67-9a57-f6ba4890ce5e-istio-envoy\") pod \"e2e-explicit-c4be41-openshift-default-686574f97c-phvtc\" (UID: \"8508d596-d89b-4f67-9a57-f6ba4890ce5e\") " pod="openshift-ingress/e2e-explicit-c4be41-openshift-default-686574f97c-phvtc"
Oct 31 02:00:10.585368 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:00:10.585141    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8508d596-d89b-4f67-9a57-f6ba4890ce5e-workload-socket\") pod \"e2e-explicit-c4be41-openshift-default-686574f97c-phvtc\" (UID: \"8508d596-d89b-4f67-9a57-f6ba4890ce5e\") " pod="openshift-ingress/e2e-explicit-c4be41-openshift-default-686574f97c-phvtc"
Oct 31 02:00:10.585368 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:00:10.585196    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8508d596-d89b-4f67-9a57-f6ba4890ce5e-credential-socket\") pod \"e2e-explicit-c4be41-openshift-default-686574f97c-phvtc\" (UID: \"8508d596-d89b-4f67-9a57-f6ba4890ce5e\") " pod="openshift-ingress/e2e-explicit-c4be41-openshift-default-686574f97c-phvtc"
Oct 31 02:00:10.585368 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:00:10.585270    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8508d596-d89b-4f67-9a57-f6ba4890ce5e-istio-token\") pod \"e2e-explicit-c4be41-openshift-default-686574f97c-phvtc\" (UID: \"8508d596-d89b-4f67-9a57-f6ba4890ce5e\") " pod="openshift-ingress/e2e-explicit-c4be41-openshift-default-686574f97c-phvtc"
Oct 31 02:00:10.585368 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:00:10.585304    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8508d596-d89b-4f67-9a57-f6ba4890ce5e-workload-certs\") pod \"e2e-explicit-c4be41-openshift-default-686574f97c-phvtc\" (UID: \"8508d596-d89b-4f67-9a57-f6ba4890ce5e\") " pod="openshift-ingress/e2e-explicit-c4be41-openshift-default-686574f97c-phvtc"
Oct 31 02:00:10.585368 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:00:10.585368    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8508d596-d89b-4f67-9a57-f6ba4890ce5e-istio-podinfo\") pod \"e2e-explicit-c4be41-openshift-default-686574f97c-phvtc\" (UID: \"8508d596-d89b-4f67-9a57-f6ba4890ce5e\") " pod="openshift-ingress/e2e-explicit-c4be41-openshift-default-686574f97c-phvtc"
Oct 31 02:00:10.590171 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:00:10.585591    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8508d596-d89b-4f67-9a57-f6ba4890ce5e-istiod-ca-cert\") pod \"e2e-explicit-c4be41-openshift-default-686574f97c-phvtc\" (UID: \"8508d596-d89b-4f67-9a57-f6ba4890ce5e\") " pod="openshift-ingress/e2e-explicit-c4be41-openshift-default-686574f97c-phvtc"
Oct 31 02:00:10.590171 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:00:10.586392    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8508d596-d89b-4f67-9a57-f6ba4890ce5e-istio-data\") pod \"e2e-explicit-c4be41-openshift-default-686574f97c-phvtc\" (UID: \"8508d596-d89b-4f67-9a57-f6ba4890ce5e\") " pod="openshift-ingress/e2e-explicit-c4be41-openshift-default-686574f97c-phvtc"
Oct 31 02:00:10.590171 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:00:10.586490    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8508d596-d89b-4f67-9a57-f6ba4890ce5e-workload-certs\") pod \"e2e-explicit-c4be41-openshift-default-686574f97c-phvtc\" (UID: \"8508d596-d89b-4f67-9a57-f6ba4890ce5e\") " pod="openshift-ingress/e2e-explicit-c4be41-openshift-default-686574f97c-phvtc"
Oct 31 02:00:10.590441 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:00:10.590298    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8508d596-d89b-4f67-9a57-f6ba4890ce5e-istiod-ca-cert\") pod \"e2e-explicit-c4be41-openshift-default-686574f97c-phvtc\" (UID: \"8508d596-d89b-4f67-9a57-f6ba4890ce5e\") " pod="openshift-ingress/e2e-explicit-c4be41-openshift-default-686574f97c-phvtc"
Oct 31 02:00:10.595138 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:00:10.592072    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8508d596-d89b-4f67-9a57-f6ba4890ce5e-istio-podinfo\") pod \"e2e-explicit-c4be41-openshift-default-686574f97c-phvtc\" (UID: \"8508d596-d89b-4f67-9a57-f6ba4890ce5e\") " pod="openshift-ingress/e2e-explicit-c4be41-openshift-default-686574f97c-phvtc"
Oct 31 02:00:10.616524 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:00:10.616487    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8508d596-d89b-4f67-9a57-f6ba4890ce5e-istio-envoy\") pod \"e2e-explicit-c4be41-openshift-default-686574f97c-phvtc\" (UID: \"8508d596-d89b-4f67-9a57-f6ba4890ce5e\") " pod="openshift-ingress/e2e-explicit-c4be41-openshift-default-686574f97c-phvtc"
Oct 31 02:00:10.630297 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:00:10.630264    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 02:00:10.641865 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:00:10.641834    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hzdn9\" (UniqueName: \"kubernetes.io/projected/8508d596-d89b-4f67-9a57-f6ba4890ce5e-kube-api-access-hzdn9\") pod \"e2e-explicit-c4be41-openshift-default-686574f97c-phvtc\" (UID: \"8508d596-d89b-4f67-9a57-f6ba4890ce5e\") " pod="openshift-ingress/e2e-explicit-c4be41-openshift-default-686574f97c-phvtc"
Oct 31 02:00:10.644128 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:00:10.644099    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8508d596-d89b-4f67-9a57-f6ba4890ce5e-istio-token\") pod \"e2e-explicit-c4be41-openshift-default-686574f97c-phvtc\" (UID: \"8508d596-d89b-4f67-9a57-f6ba4890ce5e\") " pod="openshift-ingress/e2e-explicit-c4be41-openshift-default-686574f97c-phvtc"
Oct 31 02:00:10.882026 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:00:10.881941    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-c4be41-openshift-default-686574f97c-phvtc"
Oct 31 02:00:11.241499 ip-10-0-133-197 kubenswrapper[4964]: E1031 02:00:11.241461    4964 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q56x5" containerName="extractor"
Oct 31 02:00:11.303745 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:00:11.303713    4964 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-n5bfg"]
Oct 31 02:00:11.310289 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:00:11.310266    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-n5bfg"
Oct 31 02:00:11.325805 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:00:11.325356    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-explicit-c4be41-openshift-default-686574f97c-phvtc"]
Oct 31 02:00:11.325805 ip-10-0-133-197 kubenswrapper[4964]: W1031 02:00:11.325393    4964 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8508d596_d89b_4f67_9a57_f6ba4890ce5e.slice/crio-dadbaff2de553e33f64c8e633c50718783b85bcafe9f0019cc91627e5804d959 WatchSource:0}: Error finding container dadbaff2de553e33f64c8e633c50718783b85bcafe9f0019cc91627e5804d959: Status 404 returned error can't find the container with id dadbaff2de553e33f64c8e633c50718783b85bcafe9f0019cc91627e5804d959
Oct 31 02:00:11.328173 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:00:11.328135    4964 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 02:00:11.328285 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:00:11.328219    4964 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 02:00:11.328285 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:00:11.328258    4964 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 02:00:11.362213 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:00:11.362190    4964 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-8f6b5b-openshift-default-dockercfg-6t9gt\""
Oct 31 02:00:11.400172 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:00:11.400146    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/597e819f-88c8-4061-8c6a-9adf9a909d30-istiod-ca-cert\") pod \"e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-n5bfg\" (UID: \"597e819f-88c8-4061-8c6a-9adf9a909d30\") " pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-n5bfg"
Oct 31 02:00:11.400293 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:00:11.400227    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/597e819f-88c8-4061-8c6a-9adf9a909d30-credential-socket\") pod \"e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-n5bfg\" (UID: \"597e819f-88c8-4061-8c6a-9adf9a909d30\") " pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-n5bfg"
Oct 31 02:00:11.400451 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:00:11.400426    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/597e819f-88c8-4061-8c6a-9adf9a909d30-istio-token\") pod \"e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-n5bfg\" (UID: \"597e819f-88c8-4061-8c6a-9adf9a909d30\") " pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-n5bfg"
Oct 31 02:00:11.400518 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:00:11.400484    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/597e819f-88c8-4061-8c6a-9adf9a909d30-istio-data\") pod \"e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-n5bfg\" (UID: \"597e819f-88c8-4061-8c6a-9adf9a909d30\") " pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-n5bfg"
Oct 31 02:00:11.400590 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:00:11.400571    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wlcrh\" (UniqueName: \"kubernetes.io/projected/597e819f-88c8-4061-8c6a-9adf9a909d30-kube-api-access-wlcrh\") pod \"e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-n5bfg\" (UID: \"597e819f-88c8-4061-8c6a-9adf9a909d30\") " pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-n5bfg"
Oct 31 02:00:11.400662 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:00:11.400646    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/597e819f-88c8-4061-8c6a-9adf9a909d30-istio-envoy\") pod \"e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-n5bfg\" (UID: \"597e819f-88c8-4061-8c6a-9adf9a909d30\") " pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-n5bfg"
Oct 31 02:00:11.400786 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:00:11.400768    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/597e819f-88c8-4061-8c6a-9adf9a909d30-workload-certs\") pod \"e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-n5bfg\" (UID: \"597e819f-88c8-4061-8c6a-9adf9a909d30\") " pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-n5bfg"
Oct 31 02:00:11.400855 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:00:11.400818    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/597e819f-88c8-4061-8c6a-9adf9a909d30-workload-socket\") pod \"e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-n5bfg\" (UID: \"597e819f-88c8-4061-8c6a-9adf9a909d30\") " pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-n5bfg"
Oct 31 02:00:11.400966 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:00:11.400941    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/597e819f-88c8-4061-8c6a-9adf9a909d30-istio-podinfo\") pod \"e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-n5bfg\" (UID: \"597e819f-88c8-4061-8c6a-9adf9a909d30\") " pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-n5bfg"
Oct 31 02:00:11.435043 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:00:11.422524    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-n5bfg"]
Oct 31 02:00:11.502424 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:00:11.502310    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/597e819f-88c8-4061-8c6a-9adf9a909d30-istiod-ca-cert\") pod \"e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-n5bfg\" (UID: \"597e819f-88c8-4061-8c6a-9adf9a909d30\") " pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-n5bfg"
Oct 31 02:00:11.502732 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:00:11.502709    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/597e819f-88c8-4061-8c6a-9adf9a909d30-credential-socket\") pod \"e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-n5bfg\" (UID: \"597e819f-88c8-4061-8c6a-9adf9a909d30\") " pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-n5bfg"
Oct 31 02:00:11.502985 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:00:11.502967    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/597e819f-88c8-4061-8c6a-9adf9a909d30-istio-token\") pod \"e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-n5bfg\" (UID: \"597e819f-88c8-4061-8c6a-9adf9a909d30\") " pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-n5bfg"
Oct 31 02:00:11.503148 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:00:11.503132    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/597e819f-88c8-4061-8c6a-9adf9a909d30-istio-data\") pod \"e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-n5bfg\" (UID: \"597e819f-88c8-4061-8c6a-9adf9a909d30\") " pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-n5bfg"
Oct 31 02:00:11.503308 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:00:11.503289    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wlcrh\" (UniqueName: \"kubernetes.io/projected/597e819f-88c8-4061-8c6a-9adf9a909d30-kube-api-access-wlcrh\") pod \"e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-n5bfg\" (UID: \"597e819f-88c8-4061-8c6a-9adf9a909d30\") " pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-n5bfg"
Oct 31 02:00:11.503514 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:00:11.503481    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/597e819f-88c8-4061-8c6a-9adf9a909d30-istio-envoy\") pod \"e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-n5bfg\" (UID: \"597e819f-88c8-4061-8c6a-9adf9a909d30\") " pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-n5bfg"
Oct 31 02:00:11.503741 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:00:11.503725    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/597e819f-88c8-4061-8c6a-9adf9a909d30-workload-certs\") pod \"e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-n5bfg\" (UID: \"597e819f-88c8-4061-8c6a-9adf9a909d30\") " pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-n5bfg"
Oct 31 02:00:11.503904 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:00:11.503888    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/597e819f-88c8-4061-8c6a-9adf9a909d30-workload-socket\") pod \"e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-n5bfg\" (UID: \"597e819f-88c8-4061-8c6a-9adf9a909d30\") " pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-n5bfg"
Oct 31 02:00:11.504092 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:00:11.504062    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/597e819f-88c8-4061-8c6a-9adf9a909d30-istio-podinfo\") pod \"e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-n5bfg\" (UID: \"597e819f-88c8-4061-8c6a-9adf9a909d30\") " pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-n5bfg"
Oct 31 02:00:11.507586 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:00:11.507553    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/597e819f-88c8-4061-8c6a-9adf9a909d30-istio-data\") pod \"e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-n5bfg\" (UID: \"597e819f-88c8-4061-8c6a-9adf9a909d30\") " pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-n5bfg"
Oct 31 02:00:11.508583 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:00:11.508549    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/597e819f-88c8-4061-8c6a-9adf9a909d30-istiod-ca-cert\") pod \"e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-n5bfg\" (UID: \"597e819f-88c8-4061-8c6a-9adf9a909d30\") " pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-n5bfg"
Oct 31 02:00:11.509075 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:00:11.509054    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/597e819f-88c8-4061-8c6a-9adf9a909d30-credential-socket\") pod \"e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-n5bfg\" (UID: \"597e819f-88c8-4061-8c6a-9adf9a909d30\") " pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-n5bfg"
Oct 31 02:00:11.510868 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:00:11.510845    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/597e819f-88c8-4061-8c6a-9adf9a909d30-istio-podinfo\") pod \"e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-n5bfg\" (UID: \"597e819f-88c8-4061-8c6a-9adf9a909d30\") " pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-n5bfg"
Oct 31 02:00:11.511423 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:00:11.511387    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/597e819f-88c8-4061-8c6a-9adf9a909d30-workload-certs\") pod \"e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-n5bfg\" (UID: \"597e819f-88c8-4061-8c6a-9adf9a909d30\") " pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-n5bfg"
Oct 31 02:00:11.511624 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:00:11.511602    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/597e819f-88c8-4061-8c6a-9adf9a909d30-workload-socket\") pod \"e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-n5bfg\" (UID: \"597e819f-88c8-4061-8c6a-9adf9a909d30\") " pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-n5bfg"
Oct 31 02:00:11.512862 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:00:11.512843    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/597e819f-88c8-4061-8c6a-9adf9a909d30-istio-envoy\") pod \"e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-n5bfg\" (UID: \"597e819f-88c8-4061-8c6a-9adf9a909d30\") " pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-n5bfg"
Oct 31 02:00:11.539838 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:00:11.539809    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-c4be41-openshift-default-686574f97c-phvtc" event={"ID":"8508d596-d89b-4f67-9a57-f6ba4890ce5e","Type":"ContainerStarted","Data":"5ff20856d937574be979e19ea8bc362cb93a09d20aa7cca886dd99d959280830"}
Oct 31 02:00:11.539974 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:00:11.539871    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-c4be41-openshift-default-686574f97c-phvtc" event={"ID":"8508d596-d89b-4f67-9a57-f6ba4890ce5e","Type":"ContainerStarted","Data":"dadbaff2de553e33f64c8e633c50718783b85bcafe9f0019cc91627e5804d959"}
Oct 31 02:00:11.567368 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:00:11.566540    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/597e819f-88c8-4061-8c6a-9adf9a909d30-istio-token\") pod \"e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-n5bfg\" (UID: \"597e819f-88c8-4061-8c6a-9adf9a909d30\") " pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-n5bfg"
Oct 31 02:00:11.567573 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:00:11.567487    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wlcrh\" (UniqueName: \"kubernetes.io/projected/597e819f-88c8-4061-8c6a-9adf9a909d30-kube-api-access-wlcrh\") pod \"e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-n5bfg\" (UID: \"597e819f-88c8-4061-8c6a-9adf9a909d30\") " pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-n5bfg"
Oct 31 02:00:11.628653 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:00:11.628606    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-n5bfg"
Oct 31 02:00:11.665695 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:00:11.664865    4964 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-explicit-c4be41-openshift-default-686574f97c-phvtc" podStartSLOduration=1.664845298 podStartE2EDuration="1.664845298s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 02:00:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 02:00:11.660023819 +0000 UTC m=+3380.136590199" watchObservedRunningTime="2026-10-31 02:00:11.664845298 +0000 UTC m=+3380.141411680"
Oct 31 02:00:11.882454 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:00:11.882362    4964 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-explicit-c4be41-openshift-default-686574f97c-phvtc"
Oct 31 02:00:12.022921 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:00:12.022898    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-n5bfg"]
Oct 31 02:00:12.025543 ip-10-0-133-197 kubenswrapper[4964]: W1031 02:00:12.025513    4964 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod597e819f_88c8_4061_8c6a_9adf9a909d30.slice/crio-7382473ed4ecbf14e6f1ff122eab7e8c69f584f816ea88c1cd39a4ebdfafd20a WatchSource:0}: Error finding container 7382473ed4ecbf14e6f1ff122eab7e8c69f584f816ea88c1cd39a4ebdfafd20a: Status 404 returned error can't find the container with id 7382473ed4ecbf14e6f1ff122eab7e8c69f584f816ea88c1cd39a4ebdfafd20a
Oct 31 02:00:12.028711 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:00:12.028676    4964 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 02:00:12.028811 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:00:12.028758    4964 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 02:00:12.028811 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:00:12.028798    4964 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 02:00:12.551832 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:00:12.551783    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-n5bfg" event={"ID":"597e819f-88c8-4061-8c6a-9adf9a909d30","Type":"ContainerStarted","Data":"845653a94de03bc46729c960dabb67faf546b7e333c275cc86e818a380da79ef"}
Oct 31 02:00:12.552051 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:00:12.551863    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-n5bfg" event={"ID":"597e819f-88c8-4061-8c6a-9adf9a909d30","Type":"ContainerStarted","Data":"7382473ed4ecbf14e6f1ff122eab7e8c69f584f816ea88c1cd39a4ebdfafd20a"}
Oct 31 02:00:12.629830 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:00:12.629800    4964 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-n5bfg"
Oct 31 02:00:12.674840 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:00:12.674779    4964 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-n5bfg" podStartSLOduration=1.6747618119999998 podStartE2EDuration="1.674761812s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 02:00:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 02:00:12.670846717 +0000 UTC m=+3381.147413099" watchObservedRunningTime="2026-10-31 02:00:12.674761812 +0000 UTC m=+3381.151328193"
Oct 31 02:00:12.888623 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:00:12.888551    4964 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-explicit-c4be41-openshift-default-686574f97c-phvtc"
Oct 31 02:00:13.119343 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:00:13.111043    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 02:00:13.559128 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:00:13.559083    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-explicit-c4be41-openshift-default-686574f97c-phvtc"
Oct 31 02:00:13.560148 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:00:13.560130    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-explicit-c4be41-openshift-default-686574f97c-phvtc"
Oct 31 02:00:13.631661 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:00:13.631626    4964 patch_prober.go:29] interesting pod/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-n5bfg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.186:15021/healthz/ready\": dial tcp 10.132.0.186:15021: connect: connection refused" start-of-body=
Oct 31 02:00:13.631845 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:00:13.631684    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-n5bfg" podUID="597e819f-88c8-4061-8c6a-9adf9a909d30" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.186:15021/healthz/ready\": dial tcp 10.132.0.186:15021: connect: connection refused"
Oct 31 02:00:14.630788 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:00:14.630747    4964 patch_prober.go:29] interesting pod/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-n5bfg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.186:15021/healthz/ready\": dial tcp 10.132.0.186:15021: connect: connection refused" start-of-body=
Oct 31 02:00:14.631390 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:00:14.630809    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-n5bfg" podUID="597e819f-88c8-4061-8c6a-9adf9a909d30" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.186:15021/healthz/ready\": dial tcp 10.132.0.186:15021: connect: connection refused"
Oct 31 02:00:14.680467 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:00:14.680434    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 02:00:14.922190 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:00:14.922097    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 02:00:15.629594 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:00:15.629557    4964 patch_prober.go:29] interesting pod/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-n5bfg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.186:15021/healthz/ready\": dial tcp 10.132.0.186:15021: connect: connection refused" start-of-body=
Oct 31 02:00:15.629827 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:00:15.629621    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-n5bfg" podUID="597e819f-88c8-4061-8c6a-9adf9a909d30" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.186:15021/healthz/ready\": dial tcp 10.132.0.186:15021: connect: connection refused"
Oct 31 02:00:16.629472 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:00:16.629441    4964 patch_prober.go:29] interesting pod/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-n5bfg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.186:15021/healthz/ready\": dial tcp 10.132.0.186:15021: connect: connection refused" start-of-body=
Oct 31 02:00:16.629993 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:00:16.629494    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-n5bfg" podUID="597e819f-88c8-4061-8c6a-9adf9a909d30" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.186:15021/healthz/ready\": dial tcp 10.132.0.186:15021: connect: connection refused"
Oct 31 02:00:17.019670 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:00:17.019617    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 02:00:17.629377 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:00:17.629349    4964 patch_prober.go:29] interesting pod/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-n5bfg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.186:15021/healthz/ready\": dial tcp 10.132.0.186:15021: connect: connection refused" start-of-body=
Oct 31 02:00:17.629560 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:00:17.629410    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-n5bfg" podUID="597e819f-88c8-4061-8c6a-9adf9a909d30" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.186:15021/healthz/ready\": dial tcp 10.132.0.186:15021: connect: connection refused"
Oct 31 02:00:18.317367 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:00:18.314851    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 02:00:18.523083 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:00:18.523042    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 02:00:18.674410 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:00:18.672452    4964 patch_prober.go:29] interesting pod/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-n5bfg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.186:15021/healthz/ready\": dial tcp 10.132.0.186:15021: connect: connection refused" start-of-body=
Oct 31 02:00:18.674410 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:00:18.672520    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-n5bfg" podUID="597e819f-88c8-4061-8c6a-9adf9a909d30" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.186:15021/healthz/ready\": dial tcp 10.132.0.186:15021: connect: connection refused"
Oct 31 02:00:18.746241 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:00:18.746201    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 02:00:19.912834 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:00:19.912800    4964 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-n5bfg"
Oct 31 02:00:19.913263 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:00:19.913227    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-n5bfg"
Oct 31 02:00:19.914352 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:00:19.914307    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-n5bfg"
Oct 31 02:00:23.116155 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:00:23.116123    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 02:00:27.880166 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:00:27.880125    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 02:00:32.641798 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:00:32.641761    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 02:00:37.417463 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:00:37.417413    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 02:00:37.825476 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:00:37.825429    4964 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-pxz92"]
Oct 31 02:00:37.825699 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:00:37.825672    4964 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-pxz92" podUID="4b2d61c3-25da-4fdb-a53b-5989ff15122d" containerName="istio-proxy" containerID="cri-o://c8b224ecbd33a5d60ae48b4414a7eda333e5652dd9455a8912cdc60c08baf7d7" gracePeriod=30
Oct 31 02:00:37.846534 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:00:37.846504    4964 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-4xmfp"]
Oct 31 02:00:37.846772 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:00:37.846746    4964 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-4xmfp" podUID="bb5fa688-49e8-4e29-9643-bab6160bec86" containerName="istio-proxy" containerID="cri-o://863e1be1786bcb96ef7cab29a270c0e0db116019f9970b424006c0b037cae7e3" gracePeriod=30
Oct 31 02:00:41.238338 ip-10-0-133-197 kubenswrapper[4964]: E1031 02:00:41.238284    4964 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q56x5" containerName="extractor"
Oct 31 02:00:41.709900 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:00:41.709823    4964 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-psgmw"]
Oct 31 02:00:41.710070 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:00:41.710044    4964 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-psgmw" podUID="a331bf16-e6db-4827-a796-b6a3cca34ead" containerName="istio-proxy" containerID="cri-o://178462a4a89e2e8f0d7dd44478445cf52c24cbce514be5af9c369e8eecb4f763" gracePeriod=30
Oct 31 02:00:41.761591 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:00:41.761561    4964 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-g5qrb"]
Oct 31 02:00:41.761784 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:00:41.761749    4964 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-g5qrb" podUID="4fbf3cd6-80b7-4f80-b086-f6596fd2c011" containerName="istio-proxy" containerID="cri-o://7f988f23592cf5bba8883479cb6cd27449d15df86ed32d13f015e5662e2c1ed5" gracePeriod=30
Oct 31 02:00:41.766838 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:00:41.766813    4964 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-bz562"]
Oct 31 02:00:41.766988 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:00:41.766970    4964 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-bz562" podUID="6c374822-4758-4cc7-b71f-716c77fd5157" containerName="istio-proxy" containerID="cri-o://be9bea42ca15ca82dfc94ffa747bea66fe2ae5ff558cba1a069c2602b0099f19" gracePeriod=30
Oct 31 02:00:42.202725 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:00:42.202687    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 02:00:43.138773 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:00:43.138742    4964 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-4xmfp"
Oct 31 02:00:43.164546 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:00:43.164525    4964 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-pxz92"
Oct 31 02:00:43.214020 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:00:43.213994    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bb5fa688-49e8-4e29-9643-bab6160bec86-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb5fa688-49e8-4e29-9643-bab6160bec86-workload-socket\") pod \"bb5fa688-49e8-4e29-9643-bab6160bec86\" (UID: \"bb5fa688-49e8-4e29-9643-bab6160bec86\") "
Oct 31 02:00:43.214164 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:00:43.214045    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bb5fa688-49e8-4e29-9643-bab6160bec86-kube-api-access-r8bbj\" (UniqueName: \"kubernetes.io/projected/bb5fa688-49e8-4e29-9643-bab6160bec86-kube-api-access-r8bbj\") pod \"bb5fa688-49e8-4e29-9643-bab6160bec86\" (UID: \"bb5fa688-49e8-4e29-9643-bab6160bec86\") "
Oct 31 02:00:43.214164 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:00:43.214068    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4b2d61c3-25da-4fdb-a53b-5989ff15122d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b2d61c3-25da-4fdb-a53b-5989ff15122d-workload-socket\") pod \"4b2d61c3-25da-4fdb-a53b-5989ff15122d\" (UID: \"4b2d61c3-25da-4fdb-a53b-5989ff15122d\") "
Oct 31 02:00:43.214164 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:00:43.214123    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bb5fa688-49e8-4e29-9643-bab6160bec86-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bb5fa688-49e8-4e29-9643-bab6160bec86-istio-data\") pod \"bb5fa688-49e8-4e29-9643-bab6160bec86\" (UID: \"bb5fa688-49e8-4e29-9643-bab6160bec86\") "
Oct 31 02:00:43.214164 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:00:43.214157    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4b2d61c3-25da-4fdb-a53b-5989ff15122d-istio-token\" (UniqueName: \"kubernetes.io/projected/4b2d61c3-25da-4fdb-a53b-5989ff15122d-istio-token\") pod \"4b2d61c3-25da-4fdb-a53b-5989ff15122d\" (UID: \"4b2d61c3-25da-4fdb-a53b-5989ff15122d\") "
Oct 31 02:00:43.214415 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:00:43.214173    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4b2d61c3-25da-4fdb-a53b-5989ff15122d-kube-api-access-2hrp8\" (UniqueName: \"kubernetes.io/projected/4b2d61c3-25da-4fdb-a53b-5989ff15122d-kube-api-access-2hrp8\") pod \"4b2d61c3-25da-4fdb-a53b-5989ff15122d\" (UID: \"4b2d61c3-25da-4fdb-a53b-5989ff15122d\") "
Oct 31 02:00:43.214415 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:00:43.214239    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4b2d61c3-25da-4fdb-a53b-5989ff15122d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4b2d61c3-25da-4fdb-a53b-5989ff15122d-workload-certs\") pod \"4b2d61c3-25da-4fdb-a53b-5989ff15122d\" (UID: \"4b2d61c3-25da-4fdb-a53b-5989ff15122d\") "
Oct 31 02:00:43.214415 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:00:43.214237    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb5fa688-49e8-4e29-9643-bab6160bec86-workload-socket" pod "bb5fa688-49e8-4e29-9643-bab6160bec86" (UID: "bb5fa688-49e8-4e29-9643-bab6160bec86"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:00:43.214415 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:00:43.214290    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4b2d61c3-25da-4fdb-a53b-5989ff15122d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4b2d61c3-25da-4fdb-a53b-5989ff15122d-istio-envoy\") pod \"4b2d61c3-25da-4fdb-a53b-5989ff15122d\" (UID: \"4b2d61c3-25da-4fdb-a53b-5989ff15122d\") "
Oct 31 02:00:43.214415 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:00:43.214340    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4b2d61c3-25da-4fdb-a53b-5989ff15122d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4b2d61c3-25da-4fdb-a53b-5989ff15122d-istio-data\") pod \"4b2d61c3-25da-4fdb-a53b-5989ff15122d\" (UID: \"4b2d61c3-25da-4fdb-a53b-5989ff15122d\") "
Oct 31 02:00:43.214415 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:00:43.214362    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b2d61c3-25da-4fdb-a53b-5989ff15122d-workload-socket" pod "4b2d61c3-25da-4fdb-a53b-5989ff15122d" (UID: "4b2d61c3-25da-4fdb-a53b-5989ff15122d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:00:43.214415 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:00:43.214371    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4b2d61c3-25da-4fdb-a53b-5989ff15122d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4b2d61c3-25da-4fdb-a53b-5989ff15122d-istio-podinfo\") pod \"4b2d61c3-25da-4fdb-a53b-5989ff15122d\" (UID: \"4b2d61c3-25da-4fdb-a53b-5989ff15122d\") "
Oct 31 02:00:43.214767 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:00:43.214456    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb5fa688-49e8-4e29-9643-bab6160bec86-istio-data" pod "bb5fa688-49e8-4e29-9643-bab6160bec86" (UID: "bb5fa688-49e8-4e29-9643-bab6160bec86"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:00:43.214767 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:00:43.214499    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bb5fa688-49e8-4e29-9643-bab6160bec86-istio-token\" (UniqueName: \"kubernetes.io/projected/bb5fa688-49e8-4e29-9643-bab6160bec86-istio-token\") pod \"bb5fa688-49e8-4e29-9643-bab6160bec86\" (UID: \"bb5fa688-49e8-4e29-9643-bab6160bec86\") "
Oct 31 02:00:43.214767 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:00:43.214538    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4b2d61c3-25da-4fdb-a53b-5989ff15122d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b2d61c3-25da-4fdb-a53b-5989ff15122d-credential-socket\") pod \"4b2d61c3-25da-4fdb-a53b-5989ff15122d\" (UID: \"4b2d61c3-25da-4fdb-a53b-5989ff15122d\") "
Oct 31 02:00:43.214767 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:00:43.214573    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/bb5fa688-49e8-4e29-9643-bab6160bec86-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bb5fa688-49e8-4e29-9643-bab6160bec86-istio-podinfo\") pod \"bb5fa688-49e8-4e29-9643-bab6160bec86\" (UID: \"bb5fa688-49e8-4e29-9643-bab6160bec86\") "
Oct 31 02:00:43.214767 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:00:43.214653    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bb5fa688-49e8-4e29-9643-bab6160bec86-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb5fa688-49e8-4e29-9643-bab6160bec86-credential-socket\") pod \"bb5fa688-49e8-4e29-9643-bab6160bec86\" (UID: \"bb5fa688-49e8-4e29-9643-bab6160bec86\") "
Oct 31 02:00:43.214767 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:00:43.214684    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4b2d61c3-25da-4fdb-a53b-5989ff15122d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4b2d61c3-25da-4fdb-a53b-5989ff15122d-istiod-ca-cert\") pod \"4b2d61c3-25da-4fdb-a53b-5989ff15122d\" (UID: \"4b2d61c3-25da-4fdb-a53b-5989ff15122d\") "
Oct 31 02:00:43.214767 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:00:43.214719    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bb5fa688-49e8-4e29-9643-bab6160bec86-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bb5fa688-49e8-4e29-9643-bab6160bec86-workload-certs\") pod \"bb5fa688-49e8-4e29-9643-bab6160bec86\" (UID: \"bb5fa688-49e8-4e29-9643-bab6160bec86\") "
Oct 31 02:00:43.215113 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:00:43.214785    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bb5fa688-49e8-4e29-9643-bab6160bec86-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bb5fa688-49e8-4e29-9643-bab6160bec86-istiod-ca-cert\") pod \"bb5fa688-49e8-4e29-9643-bab6160bec86\" (UID: \"bb5fa688-49e8-4e29-9643-bab6160bec86\") "
Oct 31 02:00:43.215113 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:00:43.214974    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bb5fa688-49e8-4e29-9643-bab6160bec86-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bb5fa688-49e8-4e29-9643-bab6160bec86-istio-envoy\") pod \"bb5fa688-49e8-4e29-9643-bab6160bec86\" (UID: \"bb5fa688-49e8-4e29-9643-bab6160bec86\") "
Oct 31 02:00:43.215217 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:00:43.215151    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb5fa688-49e8-4e29-9643-bab6160bec86-credential-socket" pod "bb5fa688-49e8-4e29-9643-bab6160bec86" (UID: "bb5fa688-49e8-4e29-9643-bab6160bec86"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:00:43.215275 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:00:43.215211    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b2d61c3-25da-4fdb-a53b-5989ff15122d-workload-certs" pod "4b2d61c3-25da-4fdb-a53b-5989ff15122d" (UID: "4b2d61c3-25da-4fdb-a53b-5989ff15122d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:00:43.215735 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:00:43.215708    4964 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bb5fa688-49e8-4e29-9643-bab6160bec86-istio-data\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:00:43.215866 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:00:43.215741    4964 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4b2d61c3-25da-4fdb-a53b-5989ff15122d-workload-certs\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:00:43.215866 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:00:43.215743    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b2d61c3-25da-4fdb-a53b-5989ff15122d-istio-data" pod "4b2d61c3-25da-4fdb-a53b-5989ff15122d" (UID: "4b2d61c3-25da-4fdb-a53b-5989ff15122d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:00:43.215866 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:00:43.215757    4964 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb5fa688-49e8-4e29-9643-bab6160bec86-credential-socket\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:00:43.215866 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:00:43.215771    4964 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb5fa688-49e8-4e29-9643-bab6160bec86-workload-socket\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:00:43.215866 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:00:43.215786    4964 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b2d61c3-25da-4fdb-a53b-5989ff15122d-workload-socket\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:00:43.216139 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:00:43.215999    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb5fa688-49e8-4e29-9643-bab6160bec86-workload-certs" pod "bb5fa688-49e8-4e29-9643-bab6160bec86" (UID: "bb5fa688-49e8-4e29-9643-bab6160bec86"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:00:43.216228 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:00:43.216205    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b2d61c3-25da-4fdb-a53b-5989ff15122d-credential-socket" pod "4b2d61c3-25da-4fdb-a53b-5989ff15122d" (UID: "4b2d61c3-25da-4fdb-a53b-5989ff15122d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:00:43.216371 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:00:43.216346    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4b2d61c3-25da-4fdb-a53b-5989ff15122d-istiod-ca-cert" pod "4b2d61c3-25da-4fdb-a53b-5989ff15122d" (UID: "4b2d61c3-25da-4fdb-a53b-5989ff15122d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 02:00:43.216662 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:00:43.216638    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bb5fa688-49e8-4e29-9643-bab6160bec86-istiod-ca-cert" pod "bb5fa688-49e8-4e29-9643-bab6160bec86" (UID: "bb5fa688-49e8-4e29-9643-bab6160bec86"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 02:00:43.217785 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:00:43.217759    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb5fa688-49e8-4e29-9643-bab6160bec86-kube-api-access-r8bbj" pod "bb5fa688-49e8-4e29-9643-bab6160bec86" (UID: "bb5fa688-49e8-4e29-9643-bab6160bec86"). InnerVolumeSpecName "kube-api-access-r8bbj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:00:43.217934 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:00:43.217914    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb5fa688-49e8-4e29-9643-bab6160bec86-istio-envoy" pod "bb5fa688-49e8-4e29-9643-bab6160bec86" (UID: "bb5fa688-49e8-4e29-9643-bab6160bec86"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:00:43.218050 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:00:43.217931    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b2d61c3-25da-4fdb-a53b-5989ff15122d-kube-api-access-2hrp8" pod "4b2d61c3-25da-4fdb-a53b-5989ff15122d" (UID: "4b2d61c3-25da-4fdb-a53b-5989ff15122d"). InnerVolumeSpecName "kube-api-access-2hrp8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:00:43.218393 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:00:43.218371    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b2d61c3-25da-4fdb-a53b-5989ff15122d-istio-token" pod "4b2d61c3-25da-4fdb-a53b-5989ff15122d" (UID: "4b2d61c3-25da-4fdb-a53b-5989ff15122d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:00:43.218570 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:00:43.218552    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb5fa688-49e8-4e29-9643-bab6160bec86-istio-token" pod "bb5fa688-49e8-4e29-9643-bab6160bec86" (UID: "bb5fa688-49e8-4e29-9643-bab6160bec86"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:00:43.218994 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:00:43.218972    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b2d61c3-25da-4fdb-a53b-5989ff15122d-istio-envoy" pod "4b2d61c3-25da-4fdb-a53b-5989ff15122d" (UID: "4b2d61c3-25da-4fdb-a53b-5989ff15122d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:00:43.219118 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:00:43.219100    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4b2d61c3-25da-4fdb-a53b-5989ff15122d-istio-podinfo" pod "4b2d61c3-25da-4fdb-a53b-5989ff15122d" (UID: "4b2d61c3-25da-4fdb-a53b-5989ff15122d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 02:00:43.219615 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:00:43.219593    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bb5fa688-49e8-4e29-9643-bab6160bec86-istio-podinfo" pod "bb5fa688-49e8-4e29-9643-bab6160bec86" (UID: "bb5fa688-49e8-4e29-9643-bab6160bec86"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 02:00:43.316758 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:00:43.316731    4964 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bb5fa688-49e8-4e29-9643-bab6160bec86-istio-podinfo\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:00:43.316758 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:00:43.316758    4964 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4b2d61c3-25da-4fdb-a53b-5989ff15122d-istiod-ca-cert\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:00:43.316984 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:00:43.316771    4964 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bb5fa688-49e8-4e29-9643-bab6160bec86-workload-certs\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:00:43.316984 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:00:43.316786    4964 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bb5fa688-49e8-4e29-9643-bab6160bec86-istiod-ca-cert\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:00:43.316984 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:00:43.316799    4964 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bb5fa688-49e8-4e29-9643-bab6160bec86-istio-envoy\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:00:43.316984 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:00:43.316813    4964 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r8bbj\" (UniqueName: \"kubernetes.io/projected/bb5fa688-49e8-4e29-9643-bab6160bec86-kube-api-access-r8bbj\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:00:43.316984 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:00:43.316832    4964 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4b2d61c3-25da-4fdb-a53b-5989ff15122d-istio-token\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:00:43.316984 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:00:43.316846    4964 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2hrp8\" (UniqueName: \"kubernetes.io/projected/4b2d61c3-25da-4fdb-a53b-5989ff15122d-kube-api-access-2hrp8\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:00:43.316984 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:00:43.316860    4964 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4b2d61c3-25da-4fdb-a53b-5989ff15122d-istio-envoy\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:00:43.316984 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:00:43.316872    4964 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4b2d61c3-25da-4fdb-a53b-5989ff15122d-istio-data\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:00:43.316984 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:00:43.316881    4964 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4b2d61c3-25da-4fdb-a53b-5989ff15122d-istio-podinfo\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:00:43.316984 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:00:43.316892    4964 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bb5fa688-49e8-4e29-9643-bab6160bec86-istio-token\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:00:43.316984 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:00:43.316905    4964 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b2d61c3-25da-4fdb-a53b-5989ff15122d-credential-socket\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:00:43.802887 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:00:43.802853    4964 generic.go:356] "Generic (PLEG): container finished" podID="4b2d61c3-25da-4fdb-a53b-5989ff15122d" containerID="c8b224ecbd33a5d60ae48b4414a7eda333e5652dd9455a8912cdc60c08baf7d7" exitCode=0
Oct 31 02:00:43.803023 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:00:43.802905    4964 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-pxz92"
Oct 31 02:00:43.803023 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:00:43.802921    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-pxz92" event={"ID":"4b2d61c3-25da-4fdb-a53b-5989ff15122d","Type":"ContainerDied","Data":"c8b224ecbd33a5d60ae48b4414a7eda333e5652dd9455a8912cdc60c08baf7d7"}
Oct 31 02:00:43.803023 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:00:43.802998    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-pxz92" event={"ID":"4b2d61c3-25da-4fdb-a53b-5989ff15122d","Type":"ContainerDied","Data":"ba32c7005c5deca98b3ff9f291c60eaf3655f647c84818a40ef6f90ae0ca3674"}
Oct 31 02:00:43.803176 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:00:43.803028    4964 scope.go:122] "RemoveContainer" containerID="c8b224ecbd33a5d60ae48b4414a7eda333e5652dd9455a8912cdc60c08baf7d7"
Oct 31 02:00:43.804524 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:00:43.804499    4964 generic.go:356] "Generic (PLEG): container finished" podID="bb5fa688-49e8-4e29-9643-bab6160bec86" containerID="863e1be1786bcb96ef7cab29a270c0e0db116019f9970b424006c0b037cae7e3" exitCode=0
Oct 31 02:00:43.804644 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:00:43.804546    4964 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-4xmfp"
Oct 31 02:00:43.804644 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:00:43.804581    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-4xmfp" event={"ID":"bb5fa688-49e8-4e29-9643-bab6160bec86","Type":"ContainerDied","Data":"863e1be1786bcb96ef7cab29a270c0e0db116019f9970b424006c0b037cae7e3"}
Oct 31 02:00:43.804644 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:00:43.804620    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-4xmfp" event={"ID":"bb5fa688-49e8-4e29-9643-bab6160bec86","Type":"ContainerDied","Data":"736e21b8ae149c9b746cc99c5f38e0611e0c4ca05ff35fd291f97de311341ead"}
Oct 31 02:00:43.814529 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:00:43.814509    4964 scope.go:122] "RemoveContainer" containerID="c8b224ecbd33a5d60ae48b4414a7eda333e5652dd9455a8912cdc60c08baf7d7"
Oct 31 02:00:43.814800 ip-10-0-133-197 kubenswrapper[4964]: E1031 02:00:43.814779    4964 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c8b224ecbd33a5d60ae48b4414a7eda333e5652dd9455a8912cdc60c08baf7d7\": container with ID starting with c8b224ecbd33a5d60ae48b4414a7eda333e5652dd9455a8912cdc60c08baf7d7 not found: ID does not exist" containerID="c8b224ecbd33a5d60ae48b4414a7eda333e5652dd9455a8912cdc60c08baf7d7"
Oct 31 02:00:43.814869 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:00:43.814806    4964 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c8b224ecbd33a5d60ae48b4414a7eda333e5652dd9455a8912cdc60c08baf7d7"} err="failed to get container status \"c8b224ecbd33a5d60ae48b4414a7eda333e5652dd9455a8912cdc60c08baf7d7\": rpc error: code = NotFound desc = could not find container \"c8b224ecbd33a5d60ae48b4414a7eda333e5652dd9455a8912cdc60c08baf7d7\": container with ID starting with c8b224ecbd33a5d60ae48b4414a7eda333e5652dd9455a8912cdc60c08baf7d7 not found: ID does not exist"
Oct 31 02:00:43.814869 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:00:43.814821    4964 scope.go:122] "RemoveContainer" containerID="863e1be1786bcb96ef7cab29a270c0e0db116019f9970b424006c0b037cae7e3"
Oct 31 02:00:43.824238 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:00:43.824218    4964 scope.go:122] "RemoveContainer" containerID="863e1be1786bcb96ef7cab29a270c0e0db116019f9970b424006c0b037cae7e3"
Oct 31 02:00:43.824628 ip-10-0-133-197 kubenswrapper[4964]: E1031 02:00:43.824601    4964 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"863e1be1786bcb96ef7cab29a270c0e0db116019f9970b424006c0b037cae7e3\": container with ID starting with 863e1be1786bcb96ef7cab29a270c0e0db116019f9970b424006c0b037cae7e3 not found: ID does not exist" containerID="863e1be1786bcb96ef7cab29a270c0e0db116019f9970b424006c0b037cae7e3"
Oct 31 02:00:43.824715 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:00:43.824639    4964 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"863e1be1786bcb96ef7cab29a270c0e0db116019f9970b424006c0b037cae7e3"} err="failed to get container status \"863e1be1786bcb96ef7cab29a270c0e0db116019f9970b424006c0b037cae7e3\": rpc error: code = NotFound desc = could not find container \"863e1be1786bcb96ef7cab29a270c0e0db116019f9970b424006c0b037cae7e3\": container with ID starting with 863e1be1786bcb96ef7cab29a270c0e0db116019f9970b424006c0b037cae7e3 not found: ID does not exist"
Oct 31 02:00:43.967049 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:00:43.966974    4964 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-4xmfp"]
Oct 31 02:00:44.017928 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:00:44.017898    4964 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-4xmfp"]
Oct 31 02:00:44.130921 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:00:44.130880    4964 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-pxz92"]
Oct 31 02:00:44.134213 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:00:44.134188    4964 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-pxz92"]
Oct 31 02:00:45.777751 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:00:45.777722    4964 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4b2d61c3-25da-4fdb-a53b-5989ff15122d" path="/var/lib/kubelet/pods/4b2d61c3-25da-4fdb-a53b-5989ff15122d/volumes"
Oct 31 02:00:45.778185 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:00:45.778171    4964 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bb5fa688-49e8-4e29-9643-bab6160bec86" path="/var/lib/kubelet/pods/bb5fa688-49e8-4e29-9643-bab6160bec86/volumes"
Oct 31 02:00:46.829332 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:00:46.829283    4964 generic.go:356] "Generic (PLEG): container finished" podID="a331bf16-e6db-4827-a796-b6a3cca34ead" containerID="178462a4a89e2e8f0d7dd44478445cf52c24cbce514be5af9c369e8eecb4f763" exitCode=0
Oct 31 02:00:46.829687 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:00:46.829358    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-psgmw" event={"ID":"a331bf16-e6db-4827-a796-b6a3cca34ead","Type":"ContainerDied","Data":"178462a4a89e2e8f0d7dd44478445cf52c24cbce514be5af9c369e8eecb4f763"}
Oct 31 02:00:47.033688 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:00:47.033418    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 02:00:47.085909 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:00:47.085589    4964 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-bz562"
Oct 31 02:00:47.126544 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:00:47.126519    4964 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-psgmw"
Oct 31 02:00:47.141339 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:00:47.141295    4964 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-g5qrb"
Oct 31 02:00:47.162116 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:00:47.162088    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6c374822-4758-4cc7-b71f-716c77fd5157-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6c374822-4758-4cc7-b71f-716c77fd5157-workload-certs\") pod \"6c374822-4758-4cc7-b71f-716c77fd5157\" (UID: \"6c374822-4758-4cc7-b71f-716c77fd5157\") "
Oct 31 02:00:47.162230 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:00:47.162131    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6c374822-4758-4cc7-b71f-716c77fd5157-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6c374822-4758-4cc7-b71f-716c77fd5157-istiod-ca-cert\") pod \"6c374822-4758-4cc7-b71f-716c77fd5157\" (UID: \"6c374822-4758-4cc7-b71f-716c77fd5157\") "
Oct 31 02:00:47.162287 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:00:47.162263    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6c374822-4758-4cc7-b71f-716c77fd5157-istio-token\" (UniqueName: \"kubernetes.io/projected/6c374822-4758-4cc7-b71f-716c77fd5157-istio-token\") pod \"6c374822-4758-4cc7-b71f-716c77fd5157\" (UID: \"6c374822-4758-4cc7-b71f-716c77fd5157\") "
Oct 31 02:00:47.162370 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:00:47.162328    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6c374822-4758-4cc7-b71f-716c77fd5157-kube-api-access-fn64x\" (UniqueName: \"kubernetes.io/projected/6c374822-4758-4cc7-b71f-716c77fd5157-kube-api-access-fn64x\") pod \"6c374822-4758-4cc7-b71f-716c77fd5157\" (UID: \"6c374822-4758-4cc7-b71f-716c77fd5157\") "
Oct 31 02:00:47.162414 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:00:47.162394    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c374822-4758-4cc7-b71f-716c77fd5157-workload-certs" pod "6c374822-4758-4cc7-b71f-716c77fd5157" (UID: "6c374822-4758-4cc7-b71f-716c77fd5157"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:00:47.162462 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:00:47.162431    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6c374822-4758-4cc7-b71f-716c77fd5157-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c374822-4758-4cc7-b71f-716c77fd5157-credential-socket\") pod \"6c374822-4758-4cc7-b71f-716c77fd5157\" (UID: \"6c374822-4758-4cc7-b71f-716c77fd5157\") "
Oct 31 02:00:47.162539 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:00:47.162521    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6c374822-4758-4cc7-b71f-716c77fd5157-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6c374822-4758-4cc7-b71f-716c77fd5157-istio-envoy\") pod \"6c374822-4758-4cc7-b71f-716c77fd5157\" (UID: \"6c374822-4758-4cc7-b71f-716c77fd5157\") "
Oct 31 02:00:47.162632 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:00:47.162615    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6c374822-4758-4cc7-b71f-716c77fd5157-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c374822-4758-4cc7-b71f-716c77fd5157-workload-socket\") pod \"6c374822-4758-4cc7-b71f-716c77fd5157\" (UID: \"6c374822-4758-4cc7-b71f-716c77fd5157\") "
Oct 31 02:00:47.162699 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:00:47.162678    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6c374822-4758-4cc7-b71f-716c77fd5157-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6c374822-4758-4cc7-b71f-716c77fd5157-istio-data\") pod \"6c374822-4758-4cc7-b71f-716c77fd5157\" (UID: \"6c374822-4758-4cc7-b71f-716c77fd5157\") "
Oct 31 02:00:47.162761 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:00:47.162718    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c374822-4758-4cc7-b71f-716c77fd5157-credential-socket" pod "6c374822-4758-4cc7-b71f-716c77fd5157" (UID: "6c374822-4758-4cc7-b71f-716c77fd5157"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:00:47.162761 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:00:47.162728    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6c374822-4758-4cc7-b71f-716c77fd5157-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6c374822-4758-4cc7-b71f-716c77fd5157-istio-podinfo\") pod \"6c374822-4758-4cc7-b71f-716c77fd5157\" (UID: \"6c374822-4758-4cc7-b71f-716c77fd5157\") "
Oct 31 02:00:47.162934 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:00:47.162908    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c374822-4758-4cc7-b71f-716c77fd5157-workload-socket" pod "6c374822-4758-4cc7-b71f-716c77fd5157" (UID: "6c374822-4758-4cc7-b71f-716c77fd5157"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:00:47.163180 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:00:47.163159    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6c374822-4758-4cc7-b71f-716c77fd5157-istiod-ca-cert" pod "6c374822-4758-4cc7-b71f-716c77fd5157" (UID: "6c374822-4758-4cc7-b71f-716c77fd5157"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 02:00:47.163426 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:00:47.163403    4964 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c374822-4758-4cc7-b71f-716c77fd5157-workload-socket\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:00:47.163539 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:00:47.163432    4964 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6c374822-4758-4cc7-b71f-716c77fd5157-workload-certs\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:00:47.163539 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:00:47.163448    4964 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6c374822-4758-4cc7-b71f-716c77fd5157-istiod-ca-cert\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:00:47.163539 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:00:47.163463    4964 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c374822-4758-4cc7-b71f-716c77fd5157-credential-socket\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:00:47.163539 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:00:47.163407    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c374822-4758-4cc7-b71f-716c77fd5157-istio-data" pod "6c374822-4758-4cc7-b71f-716c77fd5157" (UID: "6c374822-4758-4cc7-b71f-716c77fd5157"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:00:47.164986 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:00:47.164964    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6c374822-4758-4cc7-b71f-716c77fd5157-istio-podinfo" pod "6c374822-4758-4cc7-b71f-716c77fd5157" (UID: "6c374822-4758-4cc7-b71f-716c77fd5157"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 02:00:47.165177 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:00:47.165152    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c374822-4758-4cc7-b71f-716c77fd5157-istio-token" pod "6c374822-4758-4cc7-b71f-716c77fd5157" (UID: "6c374822-4758-4cc7-b71f-716c77fd5157"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:00:47.165273 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:00:47.165221    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c374822-4758-4cc7-b71f-716c77fd5157-istio-envoy" pod "6c374822-4758-4cc7-b71f-716c77fd5157" (UID: "6c374822-4758-4cc7-b71f-716c77fd5157"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:00:47.165502 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:00:47.165486    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c374822-4758-4cc7-b71f-716c77fd5157-kube-api-access-fn64x" pod "6c374822-4758-4cc7-b71f-716c77fd5157" (UID: "6c374822-4758-4cc7-b71f-716c77fd5157"). InnerVolumeSpecName "kube-api-access-fn64x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:00:47.264736 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:00:47.264704    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4fbf3cd6-80b7-4f80-b086-f6596fd2c011-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4fbf3cd6-80b7-4f80-b086-f6596fd2c011-istio-envoy\") pod \"4fbf3cd6-80b7-4f80-b086-f6596fd2c011\" (UID: \"4fbf3cd6-80b7-4f80-b086-f6596fd2c011\") "
Oct 31 02:00:47.264908 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:00:47.264747    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a331bf16-e6db-4827-a796-b6a3cca34ead-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a331bf16-e6db-4827-a796-b6a3cca34ead-istio-podinfo\") pod \"a331bf16-e6db-4827-a796-b6a3cca34ead\" (UID: \"a331bf16-e6db-4827-a796-b6a3cca34ead\") "
Oct 31 02:00:47.264908 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:00:47.264818    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a331bf16-e6db-4827-a796-b6a3cca34ead-kube-api-access-wq5qz\" (UniqueName: \"kubernetes.io/projected/a331bf16-e6db-4827-a796-b6a3cca34ead-kube-api-access-wq5qz\") pod \"a331bf16-e6db-4827-a796-b6a3cca34ead\" (UID: \"a331bf16-e6db-4827-a796-b6a3cca34ead\") "
Oct 31 02:00:47.264908 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:00:47.264860    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4fbf3cd6-80b7-4f80-b086-f6596fd2c011-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4fbf3cd6-80b7-4f80-b086-f6596fd2c011-workload-socket\") pod \"4fbf3cd6-80b7-4f80-b086-f6596fd2c011\" (UID: \"4fbf3cd6-80b7-4f80-b086-f6596fd2c011\") "
Oct 31 02:00:47.265105 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:00:47.264925    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4fbf3cd6-80b7-4f80-b086-f6596fd2c011-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4fbf3cd6-80b7-4f80-b086-f6596fd2c011-credential-socket\") pod \"4fbf3cd6-80b7-4f80-b086-f6596fd2c011\" (UID: \"4fbf3cd6-80b7-4f80-b086-f6596fd2c011\") "
Oct 31 02:00:47.265105 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:00:47.264951    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4fbf3cd6-80b7-4f80-b086-f6596fd2c011-istio-token\" (UniqueName: \"kubernetes.io/projected/4fbf3cd6-80b7-4f80-b086-f6596fd2c011-istio-token\") pod \"4fbf3cd6-80b7-4f80-b086-f6596fd2c011\" (UID: \"4fbf3cd6-80b7-4f80-b086-f6596fd2c011\") "
Oct 31 02:00:47.265105 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:00:47.265009    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a331bf16-e6db-4827-a796-b6a3cca34ead-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a331bf16-e6db-4827-a796-b6a3cca34ead-istio-data\") pod \"a331bf16-e6db-4827-a796-b6a3cca34ead\" (UID: \"a331bf16-e6db-4827-a796-b6a3cca34ead\") "
Oct 31 02:00:47.265105 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:00:47.265056    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4fbf3cd6-80b7-4f80-b086-f6596fd2c011-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4fbf3cd6-80b7-4f80-b086-f6596fd2c011-workload-certs\") pod \"4fbf3cd6-80b7-4f80-b086-f6596fd2c011\" (UID: \"4fbf3cd6-80b7-4f80-b086-f6596fd2c011\") "
Oct 31 02:00:47.265105 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:00:47.265090    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a331bf16-e6db-4827-a796-b6a3cca34ead-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a331bf16-e6db-4827-a796-b6a3cca34ead-workload-certs\") pod \"a331bf16-e6db-4827-a796-b6a3cca34ead\" (UID: \"a331bf16-e6db-4827-a796-b6a3cca34ead\") "
Oct 31 02:00:47.265473 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:00:47.265435    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a331bf16-e6db-4827-a796-b6a3cca34ead-workload-certs" pod "a331bf16-e6db-4827-a796-b6a3cca34ead" (UID: "a331bf16-e6db-4827-a796-b6a3cca34ead"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:00:47.265738 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:00:47.265707    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a331bf16-e6db-4827-a796-b6a3cca34ead-istio-data" pod "a331bf16-e6db-4827-a796-b6a3cca34ead" (UID: "a331bf16-e6db-4827-a796-b6a3cca34ead"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:00:47.265943 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:00:47.265917    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4fbf3cd6-80b7-4f80-b086-f6596fd2c011-workload-certs" pod "4fbf3cd6-80b7-4f80-b086-f6596fd2c011" (UID: "4fbf3cd6-80b7-4f80-b086-f6596fd2c011"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:00:47.266194 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:00:47.266060    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4fbf3cd6-80b7-4f80-b086-f6596fd2c011-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4fbf3cd6-80b7-4f80-b086-f6596fd2c011-istio-podinfo\") pod \"4fbf3cd6-80b7-4f80-b086-f6596fd2c011\" (UID: \"4fbf3cd6-80b7-4f80-b086-f6596fd2c011\") "
Oct 31 02:00:47.266194 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:00:47.266149    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a331bf16-e6db-4827-a796-b6a3cca34ead-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a331bf16-e6db-4827-a796-b6a3cca34ead-workload-socket\") pod \"a331bf16-e6db-4827-a796-b6a3cca34ead\" (UID: \"a331bf16-e6db-4827-a796-b6a3cca34ead\") "
Oct 31 02:00:47.266369 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:00:47.266235    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a331bf16-e6db-4827-a796-b6a3cca34ead-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a331bf16-e6db-4827-a796-b6a3cca34ead-istiod-ca-cert\") pod \"a331bf16-e6db-4827-a796-b6a3cca34ead\" (UID: \"a331bf16-e6db-4827-a796-b6a3cca34ead\") "
Oct 31 02:00:47.266369 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:00:47.266288    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4fbf3cd6-80b7-4f80-b086-f6596fd2c011-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4fbf3cd6-80b7-4f80-b086-f6596fd2c011-istio-data\") pod \"4fbf3cd6-80b7-4f80-b086-f6596fd2c011\" (UID: \"4fbf3cd6-80b7-4f80-b086-f6596fd2c011\") "
Oct 31 02:00:47.266369 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:00:47.266355    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a331bf16-e6db-4827-a796-b6a3cca34ead-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a331bf16-e6db-4827-a796-b6a3cca34ead-credential-socket\") pod \"a331bf16-e6db-4827-a796-b6a3cca34ead\" (UID: \"a331bf16-e6db-4827-a796-b6a3cca34ead\") "
Oct 31 02:00:47.266556 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:00:47.266419    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4fbf3cd6-80b7-4f80-b086-f6596fd2c011-kube-api-access-zcfjz\" (UniqueName: \"kubernetes.io/projected/4fbf3cd6-80b7-4f80-b086-f6596fd2c011-kube-api-access-zcfjz\") pod \"4fbf3cd6-80b7-4f80-b086-f6596fd2c011\" (UID: \"4fbf3cd6-80b7-4f80-b086-f6596fd2c011\") "
Oct 31 02:00:47.266556 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:00:47.266476    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4fbf3cd6-80b7-4f80-b086-f6596fd2c011-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4fbf3cd6-80b7-4f80-b086-f6596fd2c011-istiod-ca-cert\") pod \"4fbf3cd6-80b7-4f80-b086-f6596fd2c011\" (UID: \"4fbf3cd6-80b7-4f80-b086-f6596fd2c011\") "
Oct 31 02:00:47.266556 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:00:47.266524    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a331bf16-e6db-4827-a796-b6a3cca34ead-istio-token\" (UniqueName: \"kubernetes.io/projected/a331bf16-e6db-4827-a796-b6a3cca34ead-istio-token\") pod \"a331bf16-e6db-4827-a796-b6a3cca34ead\" (UID: \"a331bf16-e6db-4827-a796-b6a3cca34ead\") "
Oct 31 02:00:47.266728 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:00:47.266565    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a331bf16-e6db-4827-a796-b6a3cca34ead-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a331bf16-e6db-4827-a796-b6a3cca34ead-istio-envoy\") pod \"a331bf16-e6db-4827-a796-b6a3cca34ead\" (UID: \"a331bf16-e6db-4827-a796-b6a3cca34ead\") "
Oct 31 02:00:47.266950 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:00:47.266926    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4fbf3cd6-80b7-4f80-b086-f6596fd2c011-workload-socket" pod "4fbf3cd6-80b7-4f80-b086-f6596fd2c011" (UID: "4fbf3cd6-80b7-4f80-b086-f6596fd2c011"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:00:47.267294 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:00:47.267272    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a331bf16-e6db-4827-a796-b6a3cca34ead-credential-socket" pod "a331bf16-e6db-4827-a796-b6a3cca34ead" (UID: "a331bf16-e6db-4827-a796-b6a3cca34ead"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:00:47.267437 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:00:47.267406    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4fbf3cd6-80b7-4f80-b086-f6596fd2c011-credential-socket" pod "4fbf3cd6-80b7-4f80-b086-f6596fd2c011" (UID: "4fbf3cd6-80b7-4f80-b086-f6596fd2c011"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:00:47.267699 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:00:47.267676    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4fbf3cd6-80b7-4f80-b086-f6596fd2c011-istio-data" pod "4fbf3cd6-80b7-4f80-b086-f6596fd2c011" (UID: "4fbf3cd6-80b7-4f80-b086-f6596fd2c011"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:00:47.267792 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:00:47.267720    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a331bf16-e6db-4827-a796-b6a3cca34ead-istiod-ca-cert" pod "a331bf16-e6db-4827-a796-b6a3cca34ead" (UID: "a331bf16-e6db-4827-a796-b6a3cca34ead"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 02:00:47.267947 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:00:47.267929    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4fbf3cd6-80b7-4f80-b086-f6596fd2c011-istiod-ca-cert" pod "4fbf3cd6-80b7-4f80-b086-f6596fd2c011" (UID: "4fbf3cd6-80b7-4f80-b086-f6596fd2c011"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 02:00:47.268023 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:00:47.267979    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a331bf16-e6db-4827-a796-b6a3cca34ead-workload-socket" pod "a331bf16-e6db-4827-a796-b6a3cca34ead" (UID: "a331bf16-e6db-4827-a796-b6a3cca34ead"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:00:47.268963 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:00:47.268936    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4fbf3cd6-80b7-4f80-b086-f6596fd2c011-istio-token" pod "4fbf3cd6-80b7-4f80-b086-f6596fd2c011" (UID: "4fbf3cd6-80b7-4f80-b086-f6596fd2c011"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:00:47.269627 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:00:47.269604    4964 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a331bf16-e6db-4827-a796-b6a3cca34ead-istio-data\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:00:47.269731 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:00:47.269633    4964 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4fbf3cd6-80b7-4f80-b086-f6596fd2c011-workload-certs\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:00:47.269731 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:00:47.269649    4964 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6c374822-4758-4cc7-b71f-716c77fd5157-istio-envoy\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:00:47.269731 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:00:47.269664    4964 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a331bf16-e6db-4827-a796-b6a3cca34ead-workload-certs\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:00:47.269731 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:00:47.269692    4964 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a331bf16-e6db-4827-a796-b6a3cca34ead-workload-socket\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:00:47.269731 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:00:47.269707    4964 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6c374822-4758-4cc7-b71f-716c77fd5157-istio-data\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:00:47.269731 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:00:47.269724    4964 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6c374822-4758-4cc7-b71f-716c77fd5157-istio-podinfo\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:00:47.270080 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:00:47.269738    4964 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a331bf16-e6db-4827-a796-b6a3cca34ead-istiod-ca-cert\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:00:47.270080 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:00:47.269753    4964 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4fbf3cd6-80b7-4f80-b086-f6596fd2c011-istio-data\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:00:47.270080 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:00:47.269767    4964 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a331bf16-e6db-4827-a796-b6a3cca34ead-credential-socket\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:00:47.270080 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:00:47.269781    4964 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4fbf3cd6-80b7-4f80-b086-f6596fd2c011-istiod-ca-cert\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:00:47.270080 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:00:47.269796    4964 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6c374822-4758-4cc7-b71f-716c77fd5157-istio-token\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:00:47.270080 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:00:47.269811    4964 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4fbf3cd6-80b7-4f80-b086-f6596fd2c011-workload-socket\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:00:47.270080 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:00:47.269825    4964 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fn64x\" (UniqueName: \"kubernetes.io/projected/6c374822-4758-4cc7-b71f-716c77fd5157-kube-api-access-fn64x\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:00:47.270080 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:00:47.269840    4964 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4fbf3cd6-80b7-4f80-b086-f6596fd2c011-credential-socket\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:00:47.270080 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:00:47.269853    4964 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4fbf3cd6-80b7-4f80-b086-f6596fd2c011-istio-token\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:00:47.270618 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:00:47.270603    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4fbf3cd6-80b7-4f80-b086-f6596fd2c011-istio-podinfo" pod "4fbf3cd6-80b7-4f80-b086-f6596fd2c011" (UID: "4fbf3cd6-80b7-4f80-b086-f6596fd2c011"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 02:00:47.270685 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:00:47.270676    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a331bf16-e6db-4827-a796-b6a3cca34ead-istio-token" pod "a331bf16-e6db-4827-a796-b6a3cca34ead" (UID: "a331bf16-e6db-4827-a796-b6a3cca34ead"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:00:47.271197 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:00:47.271177    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a331bf16-e6db-4827-a796-b6a3cca34ead-istio-envoy" pod "a331bf16-e6db-4827-a796-b6a3cca34ead" (UID: "a331bf16-e6db-4827-a796-b6a3cca34ead"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:00:47.271490 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:00:47.271299    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a331bf16-e6db-4827-a796-b6a3cca34ead-kube-api-access-wq5qz" pod "a331bf16-e6db-4827-a796-b6a3cca34ead" (UID: "a331bf16-e6db-4827-a796-b6a3cca34ead"). InnerVolumeSpecName "kube-api-access-wq5qz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:00:47.271490 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:00:47.271446    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4fbf3cd6-80b7-4f80-b086-f6596fd2c011-kube-api-access-zcfjz" pod "4fbf3cd6-80b7-4f80-b086-f6596fd2c011" (UID: "4fbf3cd6-80b7-4f80-b086-f6596fd2c011"). InnerVolumeSpecName "kube-api-access-zcfjz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:00:47.272678 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:00:47.272658    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a331bf16-e6db-4827-a796-b6a3cca34ead-istio-podinfo" pod "a331bf16-e6db-4827-a796-b6a3cca34ead" (UID: "a331bf16-e6db-4827-a796-b6a3cca34ead"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 02:00:47.274297 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:00:47.274272    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4fbf3cd6-80b7-4f80-b086-f6596fd2c011-istio-envoy" pod "4fbf3cd6-80b7-4f80-b086-f6596fd2c011" (UID: "4fbf3cd6-80b7-4f80-b086-f6596fd2c011"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:00:47.371055 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:00:47.370963    4964 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a331bf16-e6db-4827-a796-b6a3cca34ead-istio-envoy\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:00:47.371055 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:00:47.371000    4964 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4fbf3cd6-80b7-4f80-b086-f6596fd2c011-istio-envoy\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:00:47.371055 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:00:47.371017    4964 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a331bf16-e6db-4827-a796-b6a3cca34ead-istio-podinfo\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:00:47.371055 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:00:47.371031    4964 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wq5qz\" (UniqueName: \"kubernetes.io/projected/a331bf16-e6db-4827-a796-b6a3cca34ead-kube-api-access-wq5qz\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:00:47.371055 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:00:47.371049    4964 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4fbf3cd6-80b7-4f80-b086-f6596fd2c011-istio-podinfo\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:00:47.371501 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:00:47.371063    4964 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zcfjz\" (UniqueName: \"kubernetes.io/projected/4fbf3cd6-80b7-4f80-b086-f6596fd2c011-kube-api-access-zcfjz\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:00:47.371501 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:00:47.371076    4964 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a331bf16-e6db-4827-a796-b6a3cca34ead-istio-token\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:00:47.590261 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:00:47.590222    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 02:00:47.840467 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:00:47.840433    4964 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-psgmw"
Oct 31 02:00:47.840952 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:00:47.840438    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-psgmw" event={"ID":"a331bf16-e6db-4827-a796-b6a3cca34ead","Type":"ContainerDied","Data":"e5d7320266914d9f00f9ca55430d4b693551a1c47d87d2ab81af1426b1ef9d23"}
Oct 31 02:00:47.840952 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:00:47.840579    4964 scope.go:122] "RemoveContainer" containerID="178462a4a89e2e8f0d7dd44478445cf52c24cbce514be5af9c369e8eecb4f763"
Oct 31 02:00:47.842117 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:00:47.842096    4964 generic.go:356] "Generic (PLEG): container finished" podID="6c374822-4758-4cc7-b71f-716c77fd5157" containerID="be9bea42ca15ca82dfc94ffa747bea66fe2ae5ff558cba1a069c2602b0099f19" exitCode=0
Oct 31 02:00:47.842211 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:00:47.842180    4964 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-bz562"
Oct 31 02:00:47.842411 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:00:47.842234    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-bz562" event={"ID":"6c374822-4758-4cc7-b71f-716c77fd5157","Type":"ContainerDied","Data":"be9bea42ca15ca82dfc94ffa747bea66fe2ae5ff558cba1a069c2602b0099f19"}
Oct 31 02:00:47.842494 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:00:47.842436    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-bz562" event={"ID":"6c374822-4758-4cc7-b71f-716c77fd5157","Type":"ContainerDied","Data":"f257878a53ce5eb07637535afebec3bdde2e8419d2643465006d0f02ec560e3e"}
Oct 31 02:00:47.844064 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:00:47.844031    4964 generic.go:356] "Generic (PLEG): container finished" podID="4fbf3cd6-80b7-4f80-b086-f6596fd2c011" containerID="7f988f23592cf5bba8883479cb6cd27449d15df86ed32d13f015e5662e2c1ed5" exitCode=0
Oct 31 02:00:47.844172 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:00:47.844078    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-g5qrb" event={"ID":"4fbf3cd6-80b7-4f80-b086-f6596fd2c011","Type":"ContainerDied","Data":"7f988f23592cf5bba8883479cb6cd27449d15df86ed32d13f015e5662e2c1ed5"}
Oct 31 02:00:47.844172 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:00:47.844117    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-g5qrb" event={"ID":"4fbf3cd6-80b7-4f80-b086-f6596fd2c011","Type":"ContainerDied","Data":"48505fa47db066eb720946ad14644693f4a2314db8e14a1617c4437031867f03"}
Oct 31 02:00:47.844293 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:00:47.844170    4964 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-g5qrb"
Oct 31 02:00:47.851781 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:00:47.851763    4964 scope.go:122] "RemoveContainer" containerID="be9bea42ca15ca82dfc94ffa747bea66fe2ae5ff558cba1a069c2602b0099f19"
Oct 31 02:00:47.860928 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:00:47.860911    4964 scope.go:122] "RemoveContainer" containerID="be9bea42ca15ca82dfc94ffa747bea66fe2ae5ff558cba1a069c2602b0099f19"
Oct 31 02:00:47.861229 ip-10-0-133-197 kubenswrapper[4964]: E1031 02:00:47.861205    4964 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"be9bea42ca15ca82dfc94ffa747bea66fe2ae5ff558cba1a069c2602b0099f19\": container with ID starting with be9bea42ca15ca82dfc94ffa747bea66fe2ae5ff558cba1a069c2602b0099f19 not found: ID does not exist" containerID="be9bea42ca15ca82dfc94ffa747bea66fe2ae5ff558cba1a069c2602b0099f19"
Oct 31 02:00:47.861281 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:00:47.861237    4964 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"be9bea42ca15ca82dfc94ffa747bea66fe2ae5ff558cba1a069c2602b0099f19"} err="failed to get container status \"be9bea42ca15ca82dfc94ffa747bea66fe2ae5ff558cba1a069c2602b0099f19\": rpc error: code = NotFound desc = could not find container \"be9bea42ca15ca82dfc94ffa747bea66fe2ae5ff558cba1a069c2602b0099f19\": container with ID starting with be9bea42ca15ca82dfc94ffa747bea66fe2ae5ff558cba1a069c2602b0099f19 not found: ID does not exist"
Oct 31 02:00:47.861281 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:00:47.861253    4964 scope.go:122] "RemoveContainer" containerID="7f988f23592cf5bba8883479cb6cd27449d15df86ed32d13f015e5662e2c1ed5"
Oct 31 02:00:47.869994 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:00:47.869976    4964 scope.go:122] "RemoveContainer" containerID="7f988f23592cf5bba8883479cb6cd27449d15df86ed32d13f015e5662e2c1ed5"
Oct 31 02:00:47.870253 ip-10-0-133-197 kubenswrapper[4964]: E1031 02:00:47.870234    4964 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7f988f23592cf5bba8883479cb6cd27449d15df86ed32d13f015e5662e2c1ed5\": container with ID starting with 7f988f23592cf5bba8883479cb6cd27449d15df86ed32d13f015e5662e2c1ed5 not found: ID does not exist" containerID="7f988f23592cf5bba8883479cb6cd27449d15df86ed32d13f015e5662e2c1ed5"
Oct 31 02:00:47.870487 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:00:47.870258    4964 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7f988f23592cf5bba8883479cb6cd27449d15df86ed32d13f015e5662e2c1ed5"} err="failed to get container status \"7f988f23592cf5bba8883479cb6cd27449d15df86ed32d13f015e5662e2c1ed5\": rpc error: code = NotFound desc = could not find container \"7f988f23592cf5bba8883479cb6cd27449d15df86ed32d13f015e5662e2c1ed5\": container with ID starting with 7f988f23592cf5bba8883479cb6cd27449d15df86ed32d13f015e5662e2c1ed5 not found: ID does not exist"
Oct 31 02:00:48.006790 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:00:48.006708    4964 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-g5qrb"]
Oct 31 02:00:48.054231 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:00:48.054199    4964 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-g5qrb"]
Oct 31 02:00:48.162682 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:00:48.162649    4964 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-psgmw"]
Oct 31 02:00:48.166433 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:00:48.166413    4964 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-psgmw"]
Oct 31 02:00:48.274484 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:00:48.274411    4964 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-bz562"]
Oct 31 02:00:48.280188 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:00:48.280159    4964 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-bz562"]
Oct 31 02:00:49.778497 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:00:49.778467    4964 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4fbf3cd6-80b7-4f80-b086-f6596fd2c011" path="/var/lib/kubelet/pods/4fbf3cd6-80b7-4f80-b086-f6596fd2c011/volumes"
Oct 31 02:00:49.778996 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:00:49.778970    4964 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6c374822-4758-4cc7-b71f-716c77fd5157" path="/var/lib/kubelet/pods/6c374822-4758-4cc7-b71f-716c77fd5157/volumes"
Oct 31 02:00:49.779488 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:00:49.779471    4964 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a331bf16-e6db-4827-a796-b6a3cca34ead" path="/var/lib/kubelet/pods/a331bf16-e6db-4827-a796-b6a3cca34ead/volumes"
Oct 31 02:00:51.570889 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:00:51.570838    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 02:00:51.776108 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:00:51.776072    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 02:00:52.769311 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:00:52.769278    4964 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-051514/auth-test-model-051514-kserve-777964c64f-cz8wj"]
Oct 31 02:00:52.769966 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:00:52.769949    4964 cpu_manager.go:414] "containerMap: removing container" podUID="4b2d61c3-25da-4fdb-a53b-5989ff15122d" containerName="istio-proxy"
Oct 31 02:00:52.770012 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:00:52.769968    4964 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4b2d61c3-25da-4fdb-a53b-5989ff15122d" containerName="istio-proxy"
Oct 31 02:00:52.770012 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:00:52.769984    4964 cpu_manager.go:414] "containerMap: removing container" podUID="6c374822-4758-4cc7-b71f-716c77fd5157" containerName="istio-proxy"
Oct 31 02:00:52.770012 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:00:52.769989    4964 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6c374822-4758-4cc7-b71f-716c77fd5157" containerName="istio-proxy"
Oct 31 02:00:52.770012 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:00:52.770003    4964 cpu_manager.go:414] "containerMap: removing container" podUID="a331bf16-e6db-4827-a796-b6a3cca34ead" containerName="istio-proxy"
Oct 31 02:00:52.770012 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:00:52.770010    4964 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a331bf16-e6db-4827-a796-b6a3cca34ead" containerName="istio-proxy"
Oct 31 02:00:52.770170 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:00:52.770025    4964 cpu_manager.go:414] "containerMap: removing container" podUID="4fbf3cd6-80b7-4f80-b086-f6596fd2c011" containerName="istio-proxy"
Oct 31 02:00:52.770170 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:00:52.770029    4964 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4fbf3cd6-80b7-4f80-b086-f6596fd2c011" containerName="istio-proxy"
Oct 31 02:00:52.770170 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:00:52.770036    4964 cpu_manager.go:414] "containerMap: removing container" podUID="bb5fa688-49e8-4e29-9643-bab6160bec86" containerName="istio-proxy"
Oct 31 02:00:52.770170 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:00:52.770041    4964 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bb5fa688-49e8-4e29-9643-bab6160bec86" containerName="istio-proxy"
Oct 31 02:00:52.770170 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:00:52.770135    4964 memory_manager.go:365] "RemoveStaleState removing state" podUID="4b2d61c3-25da-4fdb-a53b-5989ff15122d" containerName="istio-proxy"
Oct 31 02:00:52.770170 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:00:52.770143    4964 memory_manager.go:365] "RemoveStaleState removing state" podUID="6c374822-4758-4cc7-b71f-716c77fd5157" containerName="istio-proxy"
Oct 31 02:00:52.770170 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:00:52.770151    4964 memory_manager.go:365] "RemoveStaleState removing state" podUID="a331bf16-e6db-4827-a796-b6a3cca34ead" containerName="istio-proxy"
Oct 31 02:00:52.770170 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:00:52.770160    4964 memory_manager.go:365] "RemoveStaleState removing state" podUID="4fbf3cd6-80b7-4f80-b086-f6596fd2c011" containerName="istio-proxy"
Oct 31 02:00:52.770170 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:00:52.770165    4964 memory_manager.go:365] "RemoveStaleState removing state" podUID="bb5fa688-49e8-4e29-9643-bab6160bec86" containerName="istio-proxy"
Oct 31 02:00:52.773672 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:00:52.773652    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-051514/auth-test-model-051514-kserve-777964c64f-cz8wj"
Oct 31 02:00:52.875930 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:00:52.875895    4964 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-051514\"/\"openshift-service-ca.crt\""
Oct 31 02:00:52.875930 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:00:52.875928    4964 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-051514\"/\"auth-test-model-051514-kserve-self-signed-certs\""
Oct 31 02:00:52.876166 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:00:52.876002    4964 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-051514\"/\"default-dockercfg-bw89r\""
Oct 31 02:00:52.876166 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:00:52.876049    4964 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-051514\"/\"kube-root-ca.crt\""
Oct 31 02:00:52.879732 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:00:52.879706    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-051514/auth-test-model-051514-kserve-777964c64f-cz8wj"]
Oct 31 02:00:52.899026 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:00:52.898984    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 02:00:52.940192 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:00:52.940162    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3d4d5c24-0d5f-4183-acab-5e2ae32050f5-tls-certs\") pod \"auth-test-model-051514-kserve-777964c64f-cz8wj\" (UID: \"3d4d5c24-0d5f-4183-acab-5e2ae32050f5\") " pod="ai-tenant-e2e-shared-a-w3-051514/auth-test-model-051514-kserve-777964c64f-cz8wj"
Oct 31 02:00:52.940397 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:00:52.940200    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gv4dn\" (UniqueName: \"kubernetes.io/projected/3d4d5c24-0d5f-4183-acab-5e2ae32050f5-kube-api-access-gv4dn\") pod \"auth-test-model-051514-kserve-777964c64f-cz8wj\" (UID: \"3d4d5c24-0d5f-4183-acab-5e2ae32050f5\") " pod="ai-tenant-e2e-shared-a-w3-051514/auth-test-model-051514-kserve-777964c64f-cz8wj"
Oct 31 02:00:52.940473 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:00:52.940450    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3d4d5c24-0d5f-4183-acab-5e2ae32050f5-dshm\") pod \"auth-test-model-051514-kserve-777964c64f-cz8wj\" (UID: \"3d4d5c24-0d5f-4183-acab-5e2ae32050f5\") " pod="ai-tenant-e2e-shared-a-w3-051514/auth-test-model-051514-kserve-777964c64f-cz8wj"
Oct 31 02:00:52.940517 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:00:52.940491    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3d4d5c24-0d5f-4183-acab-5e2ae32050f5-home\") pod \"auth-test-model-051514-kserve-777964c64f-cz8wj\" (UID: \"3d4d5c24-0d5f-4183-acab-5e2ae32050f5\") " pod="ai-tenant-e2e-shared-a-w3-051514/auth-test-model-051514-kserve-777964c64f-cz8wj"
Oct 31 02:00:52.940732 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:00:52.940707    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3d4d5c24-0d5f-4183-acab-5e2ae32050f5-model-cache\") pod \"auth-test-model-051514-kserve-777964c64f-cz8wj\" (UID: \"3d4d5c24-0d5f-4183-acab-5e2ae32050f5\") " pod="ai-tenant-e2e-shared-a-w3-051514/auth-test-model-051514-kserve-777964c64f-cz8wj"
Oct 31 02:00:52.940859 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:00:52.940780    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3d4d5c24-0d5f-4183-acab-5e2ae32050f5-tmp-dir\") pod \"auth-test-model-051514-kserve-777964c64f-cz8wj\" (UID: \"3d4d5c24-0d5f-4183-acab-5e2ae32050f5\") " pod="ai-tenant-e2e-shared-a-w3-051514/auth-test-model-051514-kserve-777964c64f-cz8wj"
Oct 31 02:00:53.037347 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:00:53.037247    4964 scope.go:122] "RemoveContainer" containerID="910e66802e5f0d58fde77ba312626cc5b6b776d4112bae6c68076da98709d1fc"
Oct 31 02:00:53.041976 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:00:53.041946    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3d4d5c24-0d5f-4183-acab-5e2ae32050f5-model-cache\") pod \"auth-test-model-051514-kserve-777964c64f-cz8wj\" (UID: \"3d4d5c24-0d5f-4183-acab-5e2ae32050f5\") " pod="ai-tenant-e2e-shared-a-w3-051514/auth-test-model-051514-kserve-777964c64f-cz8wj"
Oct 31 02:00:53.042153 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:00:53.042003    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3d4d5c24-0d5f-4183-acab-5e2ae32050f5-tmp-dir\") pod \"auth-test-model-051514-kserve-777964c64f-cz8wj\" (UID: \"3d4d5c24-0d5f-4183-acab-5e2ae32050f5\") " pod="ai-tenant-e2e-shared-a-w3-051514/auth-test-model-051514-kserve-777964c64f-cz8wj"
Oct 31 02:00:53.042153 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:00:53.042054    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3d4d5c24-0d5f-4183-acab-5e2ae32050f5-tls-certs\") pod \"auth-test-model-051514-kserve-777964c64f-cz8wj\" (UID: \"3d4d5c24-0d5f-4183-acab-5e2ae32050f5\") " pod="ai-tenant-e2e-shared-a-w3-051514/auth-test-model-051514-kserve-777964c64f-cz8wj"
Oct 31 02:00:53.042153 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:00:53.042134    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gv4dn\" (UniqueName: \"kubernetes.io/projected/3d4d5c24-0d5f-4183-acab-5e2ae32050f5-kube-api-access-gv4dn\") pod \"auth-test-model-051514-kserve-777964c64f-cz8wj\" (UID: \"3d4d5c24-0d5f-4183-acab-5e2ae32050f5\") " pod="ai-tenant-e2e-shared-a-w3-051514/auth-test-model-051514-kserve-777964c64f-cz8wj"
Oct 31 02:00:53.042423 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:00:53.042403    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3d4d5c24-0d5f-4183-acab-5e2ae32050f5-model-cache\") pod \"auth-test-model-051514-kserve-777964c64f-cz8wj\" (UID: \"3d4d5c24-0d5f-4183-acab-5e2ae32050f5\") " pod="ai-tenant-e2e-shared-a-w3-051514/auth-test-model-051514-kserve-777964c64f-cz8wj"
Oct 31 02:00:53.042423 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:00:53.042414    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3d4d5c24-0d5f-4183-acab-5e2ae32050f5-tmp-dir\") pod \"auth-test-model-051514-kserve-777964c64f-cz8wj\" (UID: \"3d4d5c24-0d5f-4183-acab-5e2ae32050f5\") " pod="ai-tenant-e2e-shared-a-w3-051514/auth-test-model-051514-kserve-777964c64f-cz8wj"
Oct 31 02:00:53.042733 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:00:53.042707    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3d4d5c24-0d5f-4183-acab-5e2ae32050f5-dshm\") pod \"auth-test-model-051514-kserve-777964c64f-cz8wj\" (UID: \"3d4d5c24-0d5f-4183-acab-5e2ae32050f5\") " pod="ai-tenant-e2e-shared-a-w3-051514/auth-test-model-051514-kserve-777964c64f-cz8wj"
Oct 31 02:00:53.042860 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:00:53.042748    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3d4d5c24-0d5f-4183-acab-5e2ae32050f5-home\") pod \"auth-test-model-051514-kserve-777964c64f-cz8wj\" (UID: \"3d4d5c24-0d5f-4183-acab-5e2ae32050f5\") " pod="ai-tenant-e2e-shared-a-w3-051514/auth-test-model-051514-kserve-777964c64f-cz8wj"
Oct 31 02:00:53.043123 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:00:53.043100    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3d4d5c24-0d5f-4183-acab-5e2ae32050f5-home\") pod \"auth-test-model-051514-kserve-777964c64f-cz8wj\" (UID: \"3d4d5c24-0d5f-4183-acab-5e2ae32050f5\") " pod="ai-tenant-e2e-shared-a-w3-051514/auth-test-model-051514-kserve-777964c64f-cz8wj"
Oct 31 02:00:53.045072 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:00:53.045045    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3d4d5c24-0d5f-4183-acab-5e2ae32050f5-tls-certs\") pod \"auth-test-model-051514-kserve-777964c64f-cz8wj\" (UID: \"3d4d5c24-0d5f-4183-acab-5e2ae32050f5\") " pod="ai-tenant-e2e-shared-a-w3-051514/auth-test-model-051514-kserve-777964c64f-cz8wj"
Oct 31 02:00:53.045154 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:00:53.045056    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3d4d5c24-0d5f-4183-acab-5e2ae32050f5-dshm\") pod \"auth-test-model-051514-kserve-777964c64f-cz8wj\" (UID: \"3d4d5c24-0d5f-4183-acab-5e2ae32050f5\") " pod="ai-tenant-e2e-shared-a-w3-051514/auth-test-model-051514-kserve-777964c64f-cz8wj"
Oct 31 02:00:53.098407 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:00:53.098383    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gv4dn\" (UniqueName: \"kubernetes.io/projected/3d4d5c24-0d5f-4183-acab-5e2ae32050f5-kube-api-access-gv4dn\") pod \"auth-test-model-051514-kserve-777964c64f-cz8wj\" (UID: \"3d4d5c24-0d5f-4183-acab-5e2ae32050f5\") " pod="ai-tenant-e2e-shared-a-w3-051514/auth-test-model-051514-kserve-777964c64f-cz8wj"
Oct 31 02:00:53.386640 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:00:53.386558    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-051514/auth-test-model-051514-kserve-777964c64f-cz8wj"
Oct 31 02:00:53.771607 ip-10-0-133-197 kubenswrapper[4964]: W1031 02:00:53.771575    4964 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3d4d5c24_0d5f_4183_acab_5e2ae32050f5.slice/crio-e8207e6079e7d3c104afb0f05eab88e4b692781053f2ee1d0341c87020fbde33 WatchSource:0}: Error finding container e8207e6079e7d3c104afb0f05eab88e4b692781053f2ee1d0341c87020fbde33: Status 404 returned error can't find the container with id e8207e6079e7d3c104afb0f05eab88e4b692781053f2ee1d0341c87020fbde33
Oct 31 02:00:53.780100 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:00:53.780077    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-051514/auth-test-model-051514-kserve-777964c64f-cz8wj"]
Oct 31 02:00:53.893952 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:00:53.893905    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-051514/auth-test-model-051514-kserve-777964c64f-cz8wj" event={"ID":"3d4d5c24-0d5f-4183-acab-5e2ae32050f5","Type":"ContainerStarted","Data":"cb78ac7fb5c6a92cbf617e167bc0e1c613165e5c7ed0f6a5353c274dc9e43355"}
Oct 31 02:00:53.894135 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:00:53.893974    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-051514/auth-test-model-051514-kserve-777964c64f-cz8wj" event={"ID":"3d4d5c24-0d5f-4183-acab-5e2ae32050f5","Type":"ContainerStarted","Data":"e8207e6079e7d3c104afb0f05eab88e4b692781053f2ee1d0341c87020fbde33"}
Oct 31 02:00:54.007376 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:00:54.007250    4964 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-051514/auth-test-model-051514-kserve-777964c64f-cz8wj" podStartSLOduration=2.007233484 podStartE2EDuration="2.007233484s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 02:00:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 02:00:54.006991899 +0000 UTC m=+3422.483558276" watchObservedRunningTime="2026-10-31 02:00:54.007233484 +0000 UTC m=+3422.483799866"
Oct 31 02:00:54.386950 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:00:54.386915    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-051514/auth-test-model-051514-kserve-777964c64f-cz8wj"
Oct 31 02:00:54.595161 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:00:54.595120    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 02:00:55.378107 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:00:55.378066    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 02:00:56.496983 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:00:56.496948    4964 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-5qwk9"]
Oct 31 02:00:56.501191 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:00:56.501171    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-5qwk9"
Oct 31 02:00:56.544811 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:00:56.544659    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 02:00:56.603479 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:00:56.603447    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-5qwk9"]
Oct 31 02:00:56.689759 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:00:56.689726    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/383b0139-d5c3-4492-8245-8faac4af0a10-istio-podinfo\") pod \"e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-5qwk9\" (UID: \"383b0139-d5c3-4492-8245-8faac4af0a10\") " pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-5qwk9"
Oct 31 02:00:56.689920 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:00:56.689797    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/383b0139-d5c3-4492-8245-8faac4af0a10-istio-token\") pod \"e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-5qwk9\" (UID: \"383b0139-d5c3-4492-8245-8faac4af0a10\") " pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-5qwk9"
Oct 31 02:00:56.689995 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:00:56.689937    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/383b0139-d5c3-4492-8245-8faac4af0a10-workload-certs\") pod \"e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-5qwk9\" (UID: \"383b0139-d5c3-4492-8245-8faac4af0a10\") " pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-5qwk9"
Oct 31 02:00:56.690040 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:00:56.689997    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/383b0139-d5c3-4492-8245-8faac4af0a10-istiod-ca-cert\") pod \"e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-5qwk9\" (UID: \"383b0139-d5c3-4492-8245-8faac4af0a10\") " pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-5qwk9"
Oct 31 02:00:56.690113 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:00:56.690089    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/383b0139-d5c3-4492-8245-8faac4af0a10-workload-socket\") pod \"e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-5qwk9\" (UID: \"383b0139-d5c3-4492-8245-8faac4af0a10\") " pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-5qwk9"
Oct 31 02:00:56.690212 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:00:56.690200    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/383b0139-d5c3-4492-8245-8faac4af0a10-credential-socket\") pod \"e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-5qwk9\" (UID: \"383b0139-d5c3-4492-8245-8faac4af0a10\") " pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-5qwk9"
Oct 31 02:00:56.690248 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:00:56.690226    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wh8sq\" (UniqueName: \"kubernetes.io/projected/383b0139-d5c3-4492-8245-8faac4af0a10-kube-api-access-wh8sq\") pod \"e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-5qwk9\" (UID: \"383b0139-d5c3-4492-8245-8faac4af0a10\") " pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-5qwk9"
Oct 31 02:00:56.690283 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:00:56.690272    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/383b0139-d5c3-4492-8245-8faac4af0a10-istio-envoy\") pod \"e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-5qwk9\" (UID: \"383b0139-d5c3-4492-8245-8faac4af0a10\") " pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-5qwk9"
Oct 31 02:00:56.690334 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:00:56.690301    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/383b0139-d5c3-4492-8245-8faac4af0a10-istio-data\") pod \"e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-5qwk9\" (UID: \"383b0139-d5c3-4492-8245-8faac4af0a10\") " pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-5qwk9"
Oct 31 02:00:56.791223 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:00:56.791127    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/383b0139-d5c3-4492-8245-8faac4af0a10-credential-socket\") pod \"e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-5qwk9\" (UID: \"383b0139-d5c3-4492-8245-8faac4af0a10\") " pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-5qwk9"
Oct 31 02:00:56.791223 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:00:56.791174    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wh8sq\" (UniqueName: \"kubernetes.io/projected/383b0139-d5c3-4492-8245-8faac4af0a10-kube-api-access-wh8sq\") pod \"e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-5qwk9\" (UID: \"383b0139-d5c3-4492-8245-8faac4af0a10\") " pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-5qwk9"
Oct 31 02:00:56.791223 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:00:56.791213    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/383b0139-d5c3-4492-8245-8faac4af0a10-istio-envoy\") pod \"e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-5qwk9\" (UID: \"383b0139-d5c3-4492-8245-8faac4af0a10\") " pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-5qwk9"
Oct 31 02:00:56.791623 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:00:56.791255    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/383b0139-d5c3-4492-8245-8faac4af0a10-istio-data\") pod \"e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-5qwk9\" (UID: \"383b0139-d5c3-4492-8245-8faac4af0a10\") " pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-5qwk9"
Oct 31 02:00:56.791623 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:00:56.791279    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/383b0139-d5c3-4492-8245-8faac4af0a10-istio-podinfo\") pod \"e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-5qwk9\" (UID: \"383b0139-d5c3-4492-8245-8faac4af0a10\") " pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-5qwk9"
Oct 31 02:00:56.791623 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:00:56.791389    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/383b0139-d5c3-4492-8245-8faac4af0a10-istio-token\") pod \"e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-5qwk9\" (UID: \"383b0139-d5c3-4492-8245-8faac4af0a10\") " pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-5qwk9"
Oct 31 02:00:56.791623 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:00:56.791473    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/383b0139-d5c3-4492-8245-8faac4af0a10-workload-certs\") pod \"e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-5qwk9\" (UID: \"383b0139-d5c3-4492-8245-8faac4af0a10\") " pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-5qwk9"
Oct 31 02:00:56.791623 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:00:56.791608    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/383b0139-d5c3-4492-8245-8faac4af0a10-istiod-ca-cert\") pod \"e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-5qwk9\" (UID: \"383b0139-d5c3-4492-8245-8faac4af0a10\") " pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-5qwk9"
Oct 31 02:00:56.791945 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:00:56.791641    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/383b0139-d5c3-4492-8245-8faac4af0a10-workload-socket\") pod \"e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-5qwk9\" (UID: \"383b0139-d5c3-4492-8245-8faac4af0a10\") " pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-5qwk9"
Oct 31 02:00:56.791945 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:00:56.791662    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/383b0139-d5c3-4492-8245-8faac4af0a10-istio-data\") pod \"e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-5qwk9\" (UID: \"383b0139-d5c3-4492-8245-8faac4af0a10\") " pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-5qwk9"
Oct 31 02:00:56.791945 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:00:56.791663    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/383b0139-d5c3-4492-8245-8faac4af0a10-credential-socket\") pod \"e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-5qwk9\" (UID: \"383b0139-d5c3-4492-8245-8faac4af0a10\") " pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-5qwk9"
Oct 31 02:00:56.791945 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:00:56.791904    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/383b0139-d5c3-4492-8245-8faac4af0a10-workload-certs\") pod \"e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-5qwk9\" (UID: \"383b0139-d5c3-4492-8245-8faac4af0a10\") " pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-5qwk9"
Oct 31 02:00:56.792158 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:00:56.792012    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/383b0139-d5c3-4492-8245-8faac4af0a10-workload-socket\") pod \"e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-5qwk9\" (UID: \"383b0139-d5c3-4492-8245-8faac4af0a10\") " pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-5qwk9"
Oct 31 02:00:56.792423 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:00:56.792400    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/383b0139-d5c3-4492-8245-8faac4af0a10-istiod-ca-cert\") pod \"e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-5qwk9\" (UID: \"383b0139-d5c3-4492-8245-8faac4af0a10\") " pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-5qwk9"
Oct 31 02:00:56.793846 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:00:56.793826    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/383b0139-d5c3-4492-8245-8faac4af0a10-istio-envoy\") pod \"e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-5qwk9\" (UID: \"383b0139-d5c3-4492-8245-8faac4af0a10\") " pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-5qwk9"
Oct 31 02:00:56.794030 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:00:56.794012    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/383b0139-d5c3-4492-8245-8faac4af0a10-istio-podinfo\") pod \"e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-5qwk9\" (UID: \"383b0139-d5c3-4492-8245-8faac4af0a10\") " pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-5qwk9"
Oct 31 02:00:56.847844 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:00:56.847805    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/383b0139-d5c3-4492-8245-8faac4af0a10-istio-token\") pod \"e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-5qwk9\" (UID: \"383b0139-d5c3-4492-8245-8faac4af0a10\") " pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-5qwk9"
Oct 31 02:00:56.847996 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:00:56.847913    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wh8sq\" (UniqueName: \"kubernetes.io/projected/383b0139-d5c3-4492-8245-8faac4af0a10-kube-api-access-wh8sq\") pod \"e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-5qwk9\" (UID: \"383b0139-d5c3-4492-8245-8faac4af0a10\") " pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-5qwk9"
Oct 31 02:00:57.115915 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:00:57.115823    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-5qwk9"
Oct 31 02:00:57.552903 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:00:57.552874    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-5qwk9"]
Oct 31 02:00:57.553847 ip-10-0-133-197 kubenswrapper[4964]: W1031 02:00:57.553826    4964 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod383b0139_d5c3_4492_8245_8faac4af0a10.slice/crio-92efe09a8d8052009c17db6dec62f57aacf3ea954ab389436ee961956857e6de WatchSource:0}: Error finding container 92efe09a8d8052009c17db6dec62f57aacf3ea954ab389436ee961956857e6de: Status 404 returned error can't find the container with id 92efe09a8d8052009c17db6dec62f57aacf3ea954ab389436ee961956857e6de
Oct 31 02:00:57.556537 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:00:57.556506    4964 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 02:00:57.556627 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:00:57.556571    4964 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 02:00:57.556627 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:00:57.556601    4964 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 02:00:57.924609 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:00:57.924517    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-5qwk9" event={"ID":"383b0139-d5c3-4492-8245-8faac4af0a10","Type":"ContainerStarted","Data":"b47abc86de11e36471640ebf86f6793403f1d7dfaef6cabac1363f3e801cb706"}
Oct 31 02:00:57.924609 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:00:57.924569    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-5qwk9" event={"ID":"383b0139-d5c3-4492-8245-8faac4af0a10","Type":"ContainerStarted","Data":"92efe09a8d8052009c17db6dec62f57aacf3ea954ab389436ee961956857e6de"}
Oct 31 02:00:58.042848 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:00:58.042791    4964 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-5qwk9" podStartSLOduration=2.04277682 podStartE2EDuration="2.04277682s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 02:00:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 02:00:58.041653827 +0000 UTC m=+3426.518220210" watchObservedRunningTime="2026-10-31 02:00:58.04277682 +0000 UTC m=+3426.519343251"
Oct 31 02:00:58.116651 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:00:58.116604    4964 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-5qwk9"
Oct 31 02:00:58.118764 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:00:58.118726    4964 patch_prober.go:29] interesting pod/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-5qwk9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused" start-of-body=
Oct 31 02:00:58.118946 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:00:58.118772    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-5qwk9" podUID="383b0139-d5c3-4492-8245-8faac4af0a10" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused"
Oct 31 02:00:59.116303 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:00:59.116252    4964 patch_prober.go:29] interesting pod/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-5qwk9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused" start-of-body=
Oct 31 02:00:59.116850 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:00:59.116370    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-5qwk9" podUID="383b0139-d5c3-4492-8245-8faac4af0a10" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused"
Oct 31 02:00:59.187229 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:00:59.187201    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 02:01:00.116718 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:01:00.116670    4964 patch_prober.go:29] interesting pod/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-5qwk9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused" start-of-body=
Oct 31 02:01:00.117231 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:01:00.116739    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-5qwk9" podUID="383b0139-d5c3-4492-8245-8faac4af0a10" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused"
Oct 31 02:01:01.116119 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:01:01.116085    4964 patch_prober.go:29] interesting pod/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-5qwk9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused" start-of-body=
Oct 31 02:01:01.116285 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:01:01.116135    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-5qwk9" podUID="383b0139-d5c3-4492-8245-8faac4af0a10" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused"
Oct 31 02:01:01.289603 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:01:01.289566    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 02:01:02.116864 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:01:02.116828    4964 patch_prober.go:29] interesting pod/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-5qwk9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused" start-of-body=
Oct 31 02:01:02.117178 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:01:02.117139    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-5qwk9" podUID="383b0139-d5c3-4492-8245-8faac4af0a10" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused"
Oct 31 02:01:03.117351 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:01:03.117001    4964 patch_prober.go:29] interesting pod/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-5qwk9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused" start-of-body=
Oct 31 02:01:03.117351 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:01:03.117059    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-5qwk9" podUID="383b0139-d5c3-4492-8245-8faac4af0a10" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused"
Oct 31 02:01:03.387805 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:01:03.387722    4964 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-051514/auth-test-model-051514-kserve-777964c64f-cz8wj"
Oct 31 02:01:03.444969 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:01:03.444940    4964 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-051514/auth-test-model-051514-kserve-777964c64f-cz8wj"
Oct 31 02:01:03.983562 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:01:03.983529    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-051514/auth-test-model-051514-kserve-777964c64f-cz8wj"
Oct 31 02:01:04.116298 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:01:04.116257    4964 patch_prober.go:29] interesting pod/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-5qwk9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused" start-of-body=
Oct 31 02:01:04.116298 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:01:04.116337    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-5qwk9" podUID="383b0139-d5c3-4492-8245-8faac4af0a10" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused"
Oct 31 02:01:05.122359 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:01:05.122284    4964 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-5qwk9"
Oct 31 02:01:05.122834 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:01:05.122818    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-5qwk9"
Oct 31 02:01:05.123675 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:01:05.123657    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-5qwk9"
Oct 31 02:01:05.910635 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:01:05.910599    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 02:01:06.128160 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:01:06.128122    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 02:01:07.429754 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:01:07.429721    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 02:01:07.632481 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:01:07.632447    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 02:01:10.906123 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:01:10.906084    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 02:01:11.237978 ip-10-0-133-197 kubenswrapper[4964]: E1031 02:01:11.237945    4964 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q56x5" containerName="extractor"
Oct 31 02:01:15.655790 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:01:15.655741    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 02:01:20.219842 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:01:20.219809    4964 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-c4be41-6c8c8d4b57-7czj4"]
Oct 31 02:01:20.225135 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:01:20.225113    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-c4be41-6c8c8d4b57-7czj4"
Oct 31 02:01:20.281099 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:01:20.281060    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-c4be41-6c8c8d4b57-7czj4"]
Oct 31 02:01:20.326244 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:01:20.326203    4964 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-explicit-c4be41-serving-cert\""
Oct 31 02:01:20.394899 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:01:20.394857    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wg6hm\" (UniqueName: \"kubernetes.io/projected/1036dce3-2ee4-4dc2-a8e9-094d35d989ef-kube-api-access-wg6hm\") pod \"maas-api-e2e-explicit-c4be41-6c8c8d4b57-7czj4\" (UID: \"1036dce3-2ee4-4dc2-a8e9-094d35d989ef\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-c4be41-6c8c8d4b57-7czj4"
Oct 31 02:01:20.395097 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:01:20.394975    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1036dce3-2ee4-4dc2-a8e9-094d35d989ef-metrics-certs\") pod \"maas-api-e2e-explicit-c4be41-6c8c8d4b57-7czj4\" (UID: \"1036dce3-2ee4-4dc2-a8e9-094d35d989ef\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-c4be41-6c8c8d4b57-7czj4"
Oct 31 02:01:20.395097 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:01:20.395032    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1036dce3-2ee4-4dc2-a8e9-094d35d989ef-openshift-service-ca\") pod \"maas-api-e2e-explicit-c4be41-6c8c8d4b57-7czj4\" (UID: \"1036dce3-2ee4-4dc2-a8e9-094d35d989ef\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-c4be41-6c8c8d4b57-7czj4"
Oct 31 02:01:20.395196 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:01:20.395176    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1036dce3-2ee4-4dc2-a8e9-094d35d989ef-maas-api-tls\") pod \"maas-api-e2e-explicit-c4be41-6c8c8d4b57-7czj4\" (UID: \"1036dce3-2ee4-4dc2-a8e9-094d35d989ef\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-c4be41-6c8c8d4b57-7czj4"
Oct 31 02:01:20.461752 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:01:20.461707    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 02:01:20.497125 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:01:20.497031    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wg6hm\" (UniqueName: \"kubernetes.io/projected/1036dce3-2ee4-4dc2-a8e9-094d35d989ef-kube-api-access-wg6hm\") pod \"maas-api-e2e-explicit-c4be41-6c8c8d4b57-7czj4\" (UID: \"1036dce3-2ee4-4dc2-a8e9-094d35d989ef\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-c4be41-6c8c8d4b57-7czj4"
Oct 31 02:01:20.497125 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:01:20.497082    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1036dce3-2ee4-4dc2-a8e9-094d35d989ef-metrics-certs\") pod \"maas-api-e2e-explicit-c4be41-6c8c8d4b57-7czj4\" (UID: \"1036dce3-2ee4-4dc2-a8e9-094d35d989ef\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-c4be41-6c8c8d4b57-7czj4"
Oct 31 02:01:20.497125 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:01:20.497119    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1036dce3-2ee4-4dc2-a8e9-094d35d989ef-openshift-service-ca\") pod \"maas-api-e2e-explicit-c4be41-6c8c8d4b57-7czj4\" (UID: \"1036dce3-2ee4-4dc2-a8e9-094d35d989ef\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-c4be41-6c8c8d4b57-7czj4"
Oct 31 02:01:20.497487 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:01:20.497186    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1036dce3-2ee4-4dc2-a8e9-094d35d989ef-maas-api-tls\") pod \"maas-api-e2e-explicit-c4be41-6c8c8d4b57-7czj4\" (UID: \"1036dce3-2ee4-4dc2-a8e9-094d35d989ef\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-c4be41-6c8c8d4b57-7czj4"
Oct 31 02:01:20.497487 ip-10-0-133-197 kubenswrapper[4964]: E1031 02:01:20.497395    4964 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-explicit-c4be41-serving-cert: secret "maas-api-e2e-explicit-c4be41-serving-cert" not found
Oct 31 02:01:20.497487 ip-10-0-133-197 kubenswrapper[4964]: E1031 02:01:20.497480    4964 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1036dce3-2ee4-4dc2-a8e9-094d35d989ef-maas-api-tls podName:1036dce3-2ee4-4dc2-a8e9-094d35d989ef nodeName:}" failed. No retries permitted until 2026-10-31 02:01:20.997459191 +0000 UTC m=+3449.474025564 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/1036dce3-2ee4-4dc2-a8e9-094d35d989ef-maas-api-tls") pod "maas-api-e2e-explicit-c4be41-6c8c8d4b57-7czj4" (UID: "1036dce3-2ee4-4dc2-a8e9-094d35d989ef") : secret "maas-api-e2e-explicit-c4be41-serving-cert" not found
Oct 31 02:01:20.497955 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:01:20.497929    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1036dce3-2ee4-4dc2-a8e9-094d35d989ef-openshift-service-ca\") pod \"maas-api-e2e-explicit-c4be41-6c8c8d4b57-7czj4\" (UID: \"1036dce3-2ee4-4dc2-a8e9-094d35d989ef\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-c4be41-6c8c8d4b57-7czj4"
Oct 31 02:01:20.499846 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:01:20.499824    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1036dce3-2ee4-4dc2-a8e9-094d35d989ef-metrics-certs\") pod \"maas-api-e2e-explicit-c4be41-6c8c8d4b57-7czj4\" (UID: \"1036dce3-2ee4-4dc2-a8e9-094d35d989ef\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-c4be41-6c8c8d4b57-7czj4"
Oct 31 02:01:20.555754 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:01:20.555718    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wg6hm\" (UniqueName: \"kubernetes.io/projected/1036dce3-2ee4-4dc2-a8e9-094d35d989ef-kube-api-access-wg6hm\") pod \"maas-api-e2e-explicit-c4be41-6c8c8d4b57-7czj4\" (UID: \"1036dce3-2ee4-4dc2-a8e9-094d35d989ef\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-c4be41-6c8c8d4b57-7czj4"
Oct 31 02:01:20.641416 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:01:20.641379    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 02:01:20.811965 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:01:20.811872    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 02:01:21.004029 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:01:21.003993    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1036dce3-2ee4-4dc2-a8e9-094d35d989ef-maas-api-tls\") pod \"maas-api-e2e-explicit-c4be41-6c8c8d4b57-7czj4\" (UID: \"1036dce3-2ee4-4dc2-a8e9-094d35d989ef\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-c4be41-6c8c8d4b57-7czj4"
Oct 31 02:01:21.006747 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:01:21.006723    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1036dce3-2ee4-4dc2-a8e9-094d35d989ef-maas-api-tls\") pod \"maas-api-e2e-explicit-c4be41-6c8c8d4b57-7czj4\" (UID: \"1036dce3-2ee4-4dc2-a8e9-094d35d989ef\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-c4be41-6c8c8d4b57-7czj4"
Oct 31 02:01:21.140049 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:01:21.139945    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-c4be41-6c8c8d4b57-7czj4"
Oct 31 02:01:21.524064 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:01:21.524033    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-c4be41-6c8c8d4b57-7czj4"]
Oct 31 02:01:21.526005 ip-10-0-133-197 kubenswrapper[4964]: W1031 02:01:21.525977    4964 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1036dce3_2ee4_4dc2_a8e9_094d35d989ef.slice/crio-88c47a0442ce60a6d41ff7fa21e1d0a61b1dfbb2994e3cab4d3d73a87772f325 WatchSource:0}: Error finding container 88c47a0442ce60a6d41ff7fa21e1d0a61b1dfbb2994e3cab4d3d73a87772f325: Status 404 returned error can't find the container with id 88c47a0442ce60a6d41ff7fa21e1d0a61b1dfbb2994e3cab4d3d73a87772f325
Oct 31 02:01:22.111955 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:01:22.111895    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-c4be41-6c8c8d4b57-7czj4" event={"ID":"1036dce3-2ee4-4dc2-a8e9-094d35d989ef","Type":"ContainerStarted","Data":"88c47a0442ce60a6d41ff7fa21e1d0a61b1dfbb2994e3cab4d3d73a87772f325"}
Oct 31 02:01:23.121408 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:01:23.121373    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-c4be41-6c8c8d4b57-7czj4" event={"ID":"1036dce3-2ee4-4dc2-a8e9-094d35d989ef","Type":"ContainerStarted","Data":"a6f72059569fe67768d8510f2317673c84afb0adb3381b017a5eb4b1d9c1e4cb"}
Oct 31 02:01:23.121864 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:01:23.121519    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-c4be41-6c8c8d4b57-7czj4"
Oct 31 02:01:23.238567 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:01:23.238514    4964 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-c4be41-6c8c8d4b57-7czj4" podStartSLOduration=2.294597176 podStartE2EDuration="3.238500699s" totalImagesPullingTime="943.903523ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 02:01:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 02:01:23.234967592 +0000 UTC m=+3451.711533972" watchObservedRunningTime="2026-10-31 02:01:23.238500699 +0000 UTC m=+3451.715067077"
Oct 31 02:01:25.487263 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:01:25.487231    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 02:01:29.135538 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:01:29.135507    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-c4be41-6c8c8d4b57-7czj4"
Oct 31 02:01:30.215946 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:01:30.215869    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 02:01:30.391475 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:01:30.391440    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 02:01:35.096040 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:01:35.095881    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 02:01:38.715987 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:01:38.715956    4964 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w0-98c748/auth-test-model-98c748-kserve-594b4df7b9-jmbxz"]
Oct 31 02:01:38.721427 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:01:38.721408    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-98c748/auth-test-model-98c748-kserve-594b4df7b9-jmbxz"
Oct 31 02:01:38.778883 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:01:38.778850    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w0-98c748/auth-test-model-98c748-kserve-594b4df7b9-jmbxz"]
Oct 31 02:01:38.824209 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:01:38.824172    4964 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w0-98c748\"/\"openshift-service-ca.crt\""
Oct 31 02:01:38.824446 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:01:38.824408    4964 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w0-98c748\"/\"kube-root-ca.crt\""
Oct 31 02:01:38.824527 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:01:38.824444    4964 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w0-98c748\"/\"default-dockercfg-gnws7\""
Oct 31 02:01:38.824527 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:01:38.824481    4964 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w0-98c748\"/\"auth-test-model-98c748-kserve-self-signed-certs\""
Oct 31 02:01:38.839021 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:01:38.838995    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9700e3ca-d2e5-4171-a3b0-3506d01d7a00-model-cache\") pod \"auth-test-model-98c748-kserve-594b4df7b9-jmbxz\" (UID: \"9700e3ca-d2e5-4171-a3b0-3506d01d7a00\") " pod="ai-tenant-e2e-shared-b-w0-98c748/auth-test-model-98c748-kserve-594b4df7b9-jmbxz"
Oct 31 02:01:38.839173 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:01:38.839035    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9700e3ca-d2e5-4171-a3b0-3506d01d7a00-tls-certs\") pod \"auth-test-model-98c748-kserve-594b4df7b9-jmbxz\" (UID: \"9700e3ca-d2e5-4171-a3b0-3506d01d7a00\") " pod="ai-tenant-e2e-shared-b-w0-98c748/auth-test-model-98c748-kserve-594b4df7b9-jmbxz"
Oct 31 02:01:38.839173 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:01:38.839154    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9700e3ca-d2e5-4171-a3b0-3506d01d7a00-home\") pod \"auth-test-model-98c748-kserve-594b4df7b9-jmbxz\" (UID: \"9700e3ca-d2e5-4171-a3b0-3506d01d7a00\") " pod="ai-tenant-e2e-shared-b-w0-98c748/auth-test-model-98c748-kserve-594b4df7b9-jmbxz"
Oct 31 02:01:38.839338 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:01:38.839261    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v85gq\" (UniqueName: \"kubernetes.io/projected/9700e3ca-d2e5-4171-a3b0-3506d01d7a00-kube-api-access-v85gq\") pod \"auth-test-model-98c748-kserve-594b4df7b9-jmbxz\" (UID: \"9700e3ca-d2e5-4171-a3b0-3506d01d7a00\") " pod="ai-tenant-e2e-shared-b-w0-98c748/auth-test-model-98c748-kserve-594b4df7b9-jmbxz"
Oct 31 02:01:38.839338 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:01:38.839303    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9700e3ca-d2e5-4171-a3b0-3506d01d7a00-dshm\") pod \"auth-test-model-98c748-kserve-594b4df7b9-jmbxz\" (UID: \"9700e3ca-d2e5-4171-a3b0-3506d01d7a00\") " pod="ai-tenant-e2e-shared-b-w0-98c748/auth-test-model-98c748-kserve-594b4df7b9-jmbxz"
Oct 31 02:01:38.839505 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:01:38.839338    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9700e3ca-d2e5-4171-a3b0-3506d01d7a00-tmp-dir\") pod \"auth-test-model-98c748-kserve-594b4df7b9-jmbxz\" (UID: \"9700e3ca-d2e5-4171-a3b0-3506d01d7a00\") " pod="ai-tenant-e2e-shared-b-w0-98c748/auth-test-model-98c748-kserve-594b4df7b9-jmbxz"
Oct 31 02:01:38.850769 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:01:38.850742    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 02:01:38.940715 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:01:38.940672    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9700e3ca-d2e5-4171-a3b0-3506d01d7a00-model-cache\") pod \"auth-test-model-98c748-kserve-594b4df7b9-jmbxz\" (UID: \"9700e3ca-d2e5-4171-a3b0-3506d01d7a00\") " pod="ai-tenant-e2e-shared-b-w0-98c748/auth-test-model-98c748-kserve-594b4df7b9-jmbxz"
Oct 31 02:01:38.940926 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:01:38.940724    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9700e3ca-d2e5-4171-a3b0-3506d01d7a00-tls-certs\") pod \"auth-test-model-98c748-kserve-594b4df7b9-jmbxz\" (UID: \"9700e3ca-d2e5-4171-a3b0-3506d01d7a00\") " pod="ai-tenant-e2e-shared-b-w0-98c748/auth-test-model-98c748-kserve-594b4df7b9-jmbxz"
Oct 31 02:01:38.940926 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:01:38.940806    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9700e3ca-d2e5-4171-a3b0-3506d01d7a00-home\") pod \"auth-test-model-98c748-kserve-594b4df7b9-jmbxz\" (UID: \"9700e3ca-d2e5-4171-a3b0-3506d01d7a00\") " pod="ai-tenant-e2e-shared-b-w0-98c748/auth-test-model-98c748-kserve-594b4df7b9-jmbxz"
Oct 31 02:01:38.940926 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:01:38.940915    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v85gq\" (UniqueName: \"kubernetes.io/projected/9700e3ca-d2e5-4171-a3b0-3506d01d7a00-kube-api-access-v85gq\") pod \"auth-test-model-98c748-kserve-594b4df7b9-jmbxz\" (UID: \"9700e3ca-d2e5-4171-a3b0-3506d01d7a00\") " pod="ai-tenant-e2e-shared-b-w0-98c748/auth-test-model-98c748-kserve-594b4df7b9-jmbxz"
Oct 31 02:01:38.941118 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:01:38.940976    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9700e3ca-d2e5-4171-a3b0-3506d01d7a00-dshm\") pod \"auth-test-model-98c748-kserve-594b4df7b9-jmbxz\" (UID: \"9700e3ca-d2e5-4171-a3b0-3506d01d7a00\") " pod="ai-tenant-e2e-shared-b-w0-98c748/auth-test-model-98c748-kserve-594b4df7b9-jmbxz"
Oct 31 02:01:38.941118 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:01:38.941004    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9700e3ca-d2e5-4171-a3b0-3506d01d7a00-tmp-dir\") pod \"auth-test-model-98c748-kserve-594b4df7b9-jmbxz\" (UID: \"9700e3ca-d2e5-4171-a3b0-3506d01d7a00\") " pod="ai-tenant-e2e-shared-b-w0-98c748/auth-test-model-98c748-kserve-594b4df7b9-jmbxz"
Oct 31 02:01:38.941237 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:01:38.941123    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9700e3ca-d2e5-4171-a3b0-3506d01d7a00-model-cache\") pod \"auth-test-model-98c748-kserve-594b4df7b9-jmbxz\" (UID: \"9700e3ca-d2e5-4171-a3b0-3506d01d7a00\") " pod="ai-tenant-e2e-shared-b-w0-98c748/auth-test-model-98c748-kserve-594b4df7b9-jmbxz"
Oct 31 02:01:38.941237 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:01:38.941191    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9700e3ca-d2e5-4171-a3b0-3506d01d7a00-home\") pod \"auth-test-model-98c748-kserve-594b4df7b9-jmbxz\" (UID: \"9700e3ca-d2e5-4171-a3b0-3506d01d7a00\") " pod="ai-tenant-e2e-shared-b-w0-98c748/auth-test-model-98c748-kserve-594b4df7b9-jmbxz"
Oct 31 02:01:38.941499 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:01:38.941478    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9700e3ca-d2e5-4171-a3b0-3506d01d7a00-tmp-dir\") pod \"auth-test-model-98c748-kserve-594b4df7b9-jmbxz\" (UID: \"9700e3ca-d2e5-4171-a3b0-3506d01d7a00\") " pod="ai-tenant-e2e-shared-b-w0-98c748/auth-test-model-98c748-kserve-594b4df7b9-jmbxz"
Oct 31 02:01:38.943536 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:01:38.943515    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9700e3ca-d2e5-4171-a3b0-3506d01d7a00-dshm\") pod \"auth-test-model-98c748-kserve-594b4df7b9-jmbxz\" (UID: \"9700e3ca-d2e5-4171-a3b0-3506d01d7a00\") " pod="ai-tenant-e2e-shared-b-w0-98c748/auth-test-model-98c748-kserve-594b4df7b9-jmbxz"
Oct 31 02:01:38.943629 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:01:38.943554    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9700e3ca-d2e5-4171-a3b0-3506d01d7a00-tls-certs\") pod \"auth-test-model-98c748-kserve-594b4df7b9-jmbxz\" (UID: \"9700e3ca-d2e5-4171-a3b0-3506d01d7a00\") " pod="ai-tenant-e2e-shared-b-w0-98c748/auth-test-model-98c748-kserve-594b4df7b9-jmbxz"
Oct 31 02:01:38.998491 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:01:38.998459    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v85gq\" (UniqueName: \"kubernetes.io/projected/9700e3ca-d2e5-4171-a3b0-3506d01d7a00-kube-api-access-v85gq\") pod \"auth-test-model-98c748-kserve-594b4df7b9-jmbxz\" (UID: \"9700e3ca-d2e5-4171-a3b0-3506d01d7a00\") " pod="ai-tenant-e2e-shared-b-w0-98c748/auth-test-model-98c748-kserve-594b4df7b9-jmbxz"
Oct 31 02:01:39.033943 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:01:39.033904    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-98c748/auth-test-model-98c748-kserve-594b4df7b9-jmbxz"
Oct 31 02:01:39.422056 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:01:39.422032    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w0-98c748/auth-test-model-98c748-kserve-594b4df7b9-jmbxz"]
Oct 31 02:01:39.424984 ip-10-0-133-197 kubenswrapper[4964]: W1031 02:01:39.424950    4964 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9700e3ca_d2e5_4171_a3b0_3506d01d7a00.slice/crio-18c779fcc58d4914fa6b9af75fa61d206bbafc1920a4830b9778bcc663b3d90b WatchSource:0}: Error finding container 18c779fcc58d4914fa6b9af75fa61d206bbafc1920a4830b9778bcc663b3d90b: Status 404 returned error can't find the container with id 18c779fcc58d4914fa6b9af75fa61d206bbafc1920a4830b9778bcc663b3d90b
Oct 31 02:01:39.895668 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:01:39.895621    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 02:01:40.242108 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:01:40.242076    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-98c748/auth-test-model-98c748-kserve-594b4df7b9-jmbxz" event={"ID":"9700e3ca-d2e5-4171-a3b0-3506d01d7a00","Type":"ContainerStarted","Data":"79c0d02bc96386635fbf924e534d4a78cb4af9460b24d9594def177c38dbe2c1"}
Oct 31 02:01:40.242108 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:01:40.242116    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-98c748/auth-test-model-98c748-kserve-594b4df7b9-jmbxz" event={"ID":"9700e3ca-d2e5-4171-a3b0-3506d01d7a00","Type":"ContainerStarted","Data":"18c779fcc58d4914fa6b9af75fa61d206bbafc1920a4830b9778bcc663b3d90b"}
Oct 31 02:01:40.371117 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:01:40.371066    4964 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w0-98c748/auth-test-model-98c748-kserve-594b4df7b9-jmbxz" podStartSLOduration=2.371050141 podStartE2EDuration="2.371050141s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 02:01:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 02:01:40.369883089 +0000 UTC m=+3468.846449473" watchObservedRunningTime="2026-10-31 02:01:40.371050141 +0000 UTC m=+3468.847616520"
Oct 31 02:01:41.034641 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:01:41.034596    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w0-98c748/auth-test-model-98c748-kserve-594b4df7b9-jmbxz"
Oct 31 02:01:41.241107 ip-10-0-133-197 kubenswrapper[4964]: E1031 02:01:41.241073    4964 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q56x5" containerName="extractor"
Oct 31 02:01:44.635325 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:01:44.635269    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 02:01:47.932181 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:01:47.932139    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 02:01:49.034918 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:01:49.034880    4964 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w0-98c748/auth-test-model-98c748-kserve-594b4df7b9-jmbxz"
Oct 31 02:01:49.048334 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:01:49.048294    4964 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w0-98c748/auth-test-model-98c748-kserve-594b4df7b9-jmbxz"
Oct 31 02:01:49.316767 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:01:49.316686    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w0-98c748/auth-test-model-98c748-kserve-594b4df7b9-jmbxz"
Oct 31 02:01:49.415461 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:01:49.415428    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 02:01:51.910119 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:01:51.910083    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 02:01:54.190301 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:01:54.190268    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 02:01:58.748895 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:01:58.748473    4964 patch_prober.go:29] interesting pod/e2e-explicit-c4be41-openshift-default-686574f97c-phvtc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.185:15021/healthz/ready\": read tcp 10.132.0.2:43750->10.132.0.185:15021: read: connection reset by peer" start-of-body=
Oct 31 02:01:58.748895 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:01:58.748548    4964 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-explicit-c4be41-openshift-default-686574f97c-phvtc" podUID="8508d596-d89b-4f67-9a57-f6ba4890ce5e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.185:15021/healthz/ready\": read tcp 10.132.0.2:43750->10.132.0.185:15021: read: connection reset by peer"
Oct 31 02:01:58.954519 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:01:58.954432    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 02:01:59.388900 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:01:59.388868    4964 generic.go:356] "Generic (PLEG): container finished" podID="8508d596-d89b-4f67-9a57-f6ba4890ce5e" containerID="5ff20856d937574be979e19ea8bc362cb93a09d20aa7cca886dd99d959280830" exitCode=0
Oct 31 02:01:59.389087 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:01:59.388937    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-c4be41-openshift-default-686574f97c-phvtc" event={"ID":"8508d596-d89b-4f67-9a57-f6ba4890ce5e","Type":"ContainerDied","Data":"5ff20856d937574be979e19ea8bc362cb93a09d20aa7cca886dd99d959280830"}
Oct 31 02:01:59.389424 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:01:59.389402    4964 scope.go:122] "RemoveContainer" containerID="5ff20856d937574be979e19ea8bc362cb93a09d20aa7cca886dd99d959280830"
Oct 31 02:01:59.391030 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:01:59.391002    4964 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 02:01:59.391112 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:01:59.391074    4964 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 02:01:59.391112 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:01:59.391105    4964 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 02:01:59.534137 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:01:59.534104    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 02:02:00.404785 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:02:00.404749    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-c4be41-openshift-default-686574f97c-phvtc" event={"ID":"8508d596-d89b-4f67-9a57-f6ba4890ce5e","Type":"ContainerStarted","Data":"b19b565315490533c56c2b4b490e0515ae9254b63031049b50c9900772883685"}
Oct 31 02:02:00.882948 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:02:00.882910    4964 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-explicit-c4be41-openshift-default-686574f97c-phvtc"
Oct 31 02:02:00.883174 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:02:00.883114    4964 patch_prober.go:29] interesting pod/e2e-explicit-c4be41-openshift-default-686574f97c-phvtc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused" start-of-body=
Oct 31 02:02:00.883260 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:02:00.883179    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-c4be41-openshift-default-686574f97c-phvtc" podUID="8508d596-d89b-4f67-9a57-f6ba4890ce5e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused"
Oct 31 02:02:01.882955 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:02:01.882922    4964 patch_prober.go:29] interesting pod/e2e-explicit-c4be41-openshift-default-686574f97c-phvtc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused" start-of-body=
Oct 31 02:02:01.883375 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:02:01.882973    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-c4be41-openshift-default-686574f97c-phvtc" podUID="8508d596-d89b-4f67-9a57-f6ba4890ce5e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused"
Oct 31 02:02:02.882836 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:02:02.882795    4964 patch_prober.go:29] interesting pod/e2e-explicit-c4be41-openshift-default-686574f97c-phvtc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused" start-of-body=
Oct 31 02:02:02.883023 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:02:02.882847    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-c4be41-openshift-default-686574f97c-phvtc" podUID="8508d596-d89b-4f67-9a57-f6ba4890ce5e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused"
Oct 31 02:02:03.678802 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:02:03.678755    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 02:02:03.882411 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:02:03.882373    4964 patch_prober.go:29] interesting pod/e2e-explicit-c4be41-openshift-default-686574f97c-phvtc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused" start-of-body=
Oct 31 02:02:03.882619 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:02:03.882426    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-c4be41-openshift-default-686574f97c-phvtc" podUID="8508d596-d89b-4f67-9a57-f6ba4890ce5e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused"
Oct 31 02:02:04.882530 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:02:04.882495    4964 patch_prober.go:29] interesting pod/e2e-explicit-c4be41-openshift-default-686574f97c-phvtc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused" start-of-body=
Oct 31 02:02:04.882985 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:02:04.882562    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-c4be41-openshift-default-686574f97c-phvtc" podUID="8508d596-d89b-4f67-9a57-f6ba4890ce5e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused"
Oct 31 02:02:05.886797 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:02:05.886762    4964 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-explicit-c4be41-openshift-default-686574f97c-phvtc"
Oct 31 02:02:05.887457 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:02:05.887433    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-explicit-c4be41-openshift-default-686574f97c-phvtc"
Oct 31 02:02:05.888140 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:02:05.888119    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-explicit-c4be41-openshift-default-686574f97c-phvtc"
Oct 31 02:02:08.442826 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:02:08.442786    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 02:02:11.240832 ip-10-0-133-197 kubenswrapper[4964]: E1031 02:02:11.240789    4964 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q56x5" containerName="extractor"
Oct 31 02:02:13.312038 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:02:13.311997    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 02:02:18.103564 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:02:18.103524    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 02:02:20.819875 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:02:20.819833    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 02:02:22.904510 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:02:22.904478    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 02:02:27.685252 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:02:27.685211    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 02:02:32.473165 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:02:32.473127    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 02:02:37.298553 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:02:37.298519    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 02:02:41.240382 ip-10-0-133-197 kubenswrapper[4964]: E1031 02:02:41.240344    4964 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q56x5" containerName="extractor"
Oct 31 02:02:42.403493 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:02:42.403447    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 02:02:42.695120 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:02:42.695016    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 02:02:47.167746 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:02:47.167708    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 02:02:52.000720 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:02:52.000674    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 02:02:56.814650 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:02:56.814618    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 02:02:59.870310 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:02:59.870275    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 02:03:01.579291 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:03:01.579254    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 02:03:06.319125 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:03:06.319092    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 02:03:06.495961 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:03:06.495929    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 02:03:11.108006 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:03:11.107958    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 02:03:11.237828 ip-10-0-133-197 kubenswrapper[4964]: E1031 02:03:11.237794    4964 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q56x5" containerName="extractor"
Oct 31 02:03:15.901635 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:03:15.901601    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 02:03:20.698796 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:03:20.698756    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 02:03:25.431265 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:03:25.431225    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 02:03:30.167784 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:03:30.167746    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 02:03:31.777410 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:03:31.777372    4964 prober.go:122] "Probe failed" probeType="Liveness" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-197.us-west-1.compute.internal" podUID="8dea7006d41de627c8e06bf18f518df6" containerName="haproxy" probeResult="failure" output="Get \"https://172.20.0.1:6443/version\": context deadline exceeded"
Oct 31 02:03:33.544187 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:03:33.544149    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 02:03:34.997757 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:03:34.997715    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 02:03:39.737119 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:03:39.737077    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 02:03:41.241661 ip-10-0-133-197 kubenswrapper[4964]: E1031 02:03:41.241608    4964 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q56x5" containerName="extractor"
Oct 31 02:03:42.735484 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:03:42.735435    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 02:03:42.960285 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:03:42.960241    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 02:03:44.500625 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:03:44.500574    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 02:03:47.410800 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:03:47.410721    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 02:03:49.224392 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:03:49.224352    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 02:03:52.332746 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:03:52.332716    4964 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-9257n_f0c0ada7-7688-4dc2-97fc-813e4d0c4ef3/console-operator/2.log"
Oct 31 02:03:52.345542 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:03:52.345521    4964 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-9257n_f0c0ada7-7688-4dc2-97fc-813e4d0c4ef3/console-operator/2.log"
Oct 31 02:03:53.282083 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:03:53.282047    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 02:03:53.991671 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:03:53.991632    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 02:03:58.745803 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:03:58.745763    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 02:04:03.514087 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:04:03.514048    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 02:04:04.626462 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:04:04.626413    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 02:04:05.097843 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:04:05.097808    4964 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-autores-183266-openshift-default-74c6876c79-qqthj"]
Oct 31 02:04:05.102144 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:04:05.102124    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-183266-openshift-default-74c6876c79-qqthj"
Oct 31 02:04:05.223923 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:04:05.223889    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-183266-openshift-default-74c6876c79-qqthj"]
Oct 31 02:04:05.240026 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:04:05.239999    4964 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-autores-183266-openshift-default-dockercfg-744zc\""
Oct 31 02:04:05.259990 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:04:05.259964    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c7eddf8c-a855-4a72-b3ed-eacf3aaed5af-workload-certs\") pod \"e2e-autores-183266-openshift-default-74c6876c79-qqthj\" (UID: \"c7eddf8c-a855-4a72-b3ed-eacf3aaed5af\") " pod="openshift-ingress/e2e-autores-183266-openshift-default-74c6876c79-qqthj"
Oct 31 02:04:05.260143 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:04:05.260033    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c7eddf8c-a855-4a72-b3ed-eacf3aaed5af-istio-podinfo\") pod \"e2e-autores-183266-openshift-default-74c6876c79-qqthj\" (UID: \"c7eddf8c-a855-4a72-b3ed-eacf3aaed5af\") " pod="openshift-ingress/e2e-autores-183266-openshift-default-74c6876c79-qqthj"
Oct 31 02:04:05.260143 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:04:05.260115    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c7eddf8c-a855-4a72-b3ed-eacf3aaed5af-istio-envoy\") pod \"e2e-autores-183266-openshift-default-74c6876c79-qqthj\" (UID: \"c7eddf8c-a855-4a72-b3ed-eacf3aaed5af\") " pod="openshift-ingress/e2e-autores-183266-openshift-default-74c6876c79-qqthj"
Oct 31 02:04:05.260275 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:04:05.260162    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c7eddf8c-a855-4a72-b3ed-eacf3aaed5af-istio-data\") pod \"e2e-autores-183266-openshift-default-74c6876c79-qqthj\" (UID: \"c7eddf8c-a855-4a72-b3ed-eacf3aaed5af\") " pod="openshift-ingress/e2e-autores-183266-openshift-default-74c6876c79-qqthj"
Oct 31 02:04:05.260275 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:04:05.260244    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7eddf8c-a855-4a72-b3ed-eacf3aaed5af-credential-socket\") pod \"e2e-autores-183266-openshift-default-74c6876c79-qqthj\" (UID: \"c7eddf8c-a855-4a72-b3ed-eacf3aaed5af\") " pod="openshift-ingress/e2e-autores-183266-openshift-default-74c6876c79-qqthj"
Oct 31 02:04:05.260437 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:04:05.260274    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zcvmj\" (UniqueName: \"kubernetes.io/projected/c7eddf8c-a855-4a72-b3ed-eacf3aaed5af-kube-api-access-zcvmj\") pod \"e2e-autores-183266-openshift-default-74c6876c79-qqthj\" (UID: \"c7eddf8c-a855-4a72-b3ed-eacf3aaed5af\") " pod="openshift-ingress/e2e-autores-183266-openshift-default-74c6876c79-qqthj"
Oct 31 02:04:05.260437 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:04:05.260413    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c7eddf8c-a855-4a72-b3ed-eacf3aaed5af-istio-token\") pod \"e2e-autores-183266-openshift-default-74c6876c79-qqthj\" (UID: \"c7eddf8c-a855-4a72-b3ed-eacf3aaed5af\") " pod="openshift-ingress/e2e-autores-183266-openshift-default-74c6876c79-qqthj"
Oct 31 02:04:05.260545 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:04:05.260449    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c7eddf8c-a855-4a72-b3ed-eacf3aaed5af-istiod-ca-cert\") pod \"e2e-autores-183266-openshift-default-74c6876c79-qqthj\" (UID: \"c7eddf8c-a855-4a72-b3ed-eacf3aaed5af\") " pod="openshift-ingress/e2e-autores-183266-openshift-default-74c6876c79-qqthj"
Oct 31 02:04:05.260545 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:04:05.260499    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7eddf8c-a855-4a72-b3ed-eacf3aaed5af-workload-socket\") pod \"e2e-autores-183266-openshift-default-74c6876c79-qqthj\" (UID: \"c7eddf8c-a855-4a72-b3ed-eacf3aaed5af\") " pod="openshift-ingress/e2e-autores-183266-openshift-default-74c6876c79-qqthj"
Oct 31 02:04:05.315427 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:04:05.315386    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 02:04:05.362038 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:04:05.361958    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c7eddf8c-a855-4a72-b3ed-eacf3aaed5af-workload-certs\") pod \"e2e-autores-183266-openshift-default-74c6876c79-qqthj\" (UID: \"c7eddf8c-a855-4a72-b3ed-eacf3aaed5af\") " pod="openshift-ingress/e2e-autores-183266-openshift-default-74c6876c79-qqthj"
Oct 31 02:04:05.362038 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:04:05.362028    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c7eddf8c-a855-4a72-b3ed-eacf3aaed5af-istio-podinfo\") pod \"e2e-autores-183266-openshift-default-74c6876c79-qqthj\" (UID: \"c7eddf8c-a855-4a72-b3ed-eacf3aaed5af\") " pod="openshift-ingress/e2e-autores-183266-openshift-default-74c6876c79-qqthj"
Oct 31 02:04:05.362260 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:04:05.362057    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c7eddf8c-a855-4a72-b3ed-eacf3aaed5af-istio-envoy\") pod \"e2e-autores-183266-openshift-default-74c6876c79-qqthj\" (UID: \"c7eddf8c-a855-4a72-b3ed-eacf3aaed5af\") " pod="openshift-ingress/e2e-autores-183266-openshift-default-74c6876c79-qqthj"
Oct 31 02:04:05.362260 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:04:05.362091    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c7eddf8c-a855-4a72-b3ed-eacf3aaed5af-istio-data\") pod \"e2e-autores-183266-openshift-default-74c6876c79-qqthj\" (UID: \"c7eddf8c-a855-4a72-b3ed-eacf3aaed5af\") " pod="openshift-ingress/e2e-autores-183266-openshift-default-74c6876c79-qqthj"
Oct 31 02:04:05.362260 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:04:05.362134    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7eddf8c-a855-4a72-b3ed-eacf3aaed5af-credential-socket\") pod \"e2e-autores-183266-openshift-default-74c6876c79-qqthj\" (UID: \"c7eddf8c-a855-4a72-b3ed-eacf3aaed5af\") " pod="openshift-ingress/e2e-autores-183266-openshift-default-74c6876c79-qqthj"
Oct 31 02:04:05.362260 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:04:05.362185    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zcvmj\" (UniqueName: \"kubernetes.io/projected/c7eddf8c-a855-4a72-b3ed-eacf3aaed5af-kube-api-access-zcvmj\") pod \"e2e-autores-183266-openshift-default-74c6876c79-qqthj\" (UID: \"c7eddf8c-a855-4a72-b3ed-eacf3aaed5af\") " pod="openshift-ingress/e2e-autores-183266-openshift-default-74c6876c79-qqthj"
Oct 31 02:04:05.362260 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:04:05.362255    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c7eddf8c-a855-4a72-b3ed-eacf3aaed5af-istio-token\") pod \"e2e-autores-183266-openshift-default-74c6876c79-qqthj\" (UID: \"c7eddf8c-a855-4a72-b3ed-eacf3aaed5af\") " pod="openshift-ingress/e2e-autores-183266-openshift-default-74c6876c79-qqthj"
Oct 31 02:04:05.362604 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:04:05.362296    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c7eddf8c-a855-4a72-b3ed-eacf3aaed5af-istiod-ca-cert\") pod \"e2e-autores-183266-openshift-default-74c6876c79-qqthj\" (UID: \"c7eddf8c-a855-4a72-b3ed-eacf3aaed5af\") " pod="openshift-ingress/e2e-autores-183266-openshift-default-74c6876c79-qqthj"
Oct 31 02:04:05.362604 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:04:05.362405    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7eddf8c-a855-4a72-b3ed-eacf3aaed5af-workload-socket\") pod \"e2e-autores-183266-openshift-default-74c6876c79-qqthj\" (UID: \"c7eddf8c-a855-4a72-b3ed-eacf3aaed5af\") " pod="openshift-ingress/e2e-autores-183266-openshift-default-74c6876c79-qqthj"
Oct 31 02:04:05.362604 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:04:05.362437    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c7eddf8c-a855-4a72-b3ed-eacf3aaed5af-workload-certs\") pod \"e2e-autores-183266-openshift-default-74c6876c79-qqthj\" (UID: \"c7eddf8c-a855-4a72-b3ed-eacf3aaed5af\") " pod="openshift-ingress/e2e-autores-183266-openshift-default-74c6876c79-qqthj"
Oct 31 02:04:05.362604 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:04:05.362514    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c7eddf8c-a855-4a72-b3ed-eacf3aaed5af-istio-data\") pod \"e2e-autores-183266-openshift-default-74c6876c79-qqthj\" (UID: \"c7eddf8c-a855-4a72-b3ed-eacf3aaed5af\") " pod="openshift-ingress/e2e-autores-183266-openshift-default-74c6876c79-qqthj"
Oct 31 02:04:05.362604 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:04:05.362595    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7eddf8c-a855-4a72-b3ed-eacf3aaed5af-credential-socket\") pod \"e2e-autores-183266-openshift-default-74c6876c79-qqthj\" (UID: \"c7eddf8c-a855-4a72-b3ed-eacf3aaed5af\") " pod="openshift-ingress/e2e-autores-183266-openshift-default-74c6876c79-qqthj"
Oct 31 02:04:05.362809 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:04:05.362752    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7eddf8c-a855-4a72-b3ed-eacf3aaed5af-workload-socket\") pod \"e2e-autores-183266-openshift-default-74c6876c79-qqthj\" (UID: \"c7eddf8c-a855-4a72-b3ed-eacf3aaed5af\") " pod="openshift-ingress/e2e-autores-183266-openshift-default-74c6876c79-qqthj"
Oct 31 02:04:05.363108 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:04:05.363086    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c7eddf8c-a855-4a72-b3ed-eacf3aaed5af-istiod-ca-cert\") pod \"e2e-autores-183266-openshift-default-74c6876c79-qqthj\" (UID: \"c7eddf8c-a855-4a72-b3ed-eacf3aaed5af\") " pod="openshift-ingress/e2e-autores-183266-openshift-default-74c6876c79-qqthj"
Oct 31 02:04:05.364511 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:04:05.364489    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c7eddf8c-a855-4a72-b3ed-eacf3aaed5af-istio-envoy\") pod \"e2e-autores-183266-openshift-default-74c6876c79-qqthj\" (UID: \"c7eddf8c-a855-4a72-b3ed-eacf3aaed5af\") " pod="openshift-ingress/e2e-autores-183266-openshift-default-74c6876c79-qqthj"
Oct 31 02:04:05.364910 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:04:05.364887    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c7eddf8c-a855-4a72-b3ed-eacf3aaed5af-istio-podinfo\") pod \"e2e-autores-183266-openshift-default-74c6876c79-qqthj\" (UID: \"c7eddf8c-a855-4a72-b3ed-eacf3aaed5af\") " pod="openshift-ingress/e2e-autores-183266-openshift-default-74c6876c79-qqthj"
Oct 31 02:04:05.419182 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:04:05.419157    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zcvmj\" (UniqueName: \"kubernetes.io/projected/c7eddf8c-a855-4a72-b3ed-eacf3aaed5af-kube-api-access-zcvmj\") pod \"e2e-autores-183266-openshift-default-74c6876c79-qqthj\" (UID: \"c7eddf8c-a855-4a72-b3ed-eacf3aaed5af\") " pod="openshift-ingress/e2e-autores-183266-openshift-default-74c6876c79-qqthj"
Oct 31 02:04:05.419182 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:04:05.419166    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c7eddf8c-a855-4a72-b3ed-eacf3aaed5af-istio-token\") pod \"e2e-autores-183266-openshift-default-74c6876c79-qqthj\" (UID: \"c7eddf8c-a855-4a72-b3ed-eacf3aaed5af\") " pod="openshift-ingress/e2e-autores-183266-openshift-default-74c6876c79-qqthj"
Oct 31 02:04:05.543290 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:04:05.543257    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 02:04:05.717153 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:04:05.717079    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-183266-openshift-default-74c6876c79-qqthj"
Oct 31 02:04:06.279472 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:04:06.279446    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-183266-openshift-default-74c6876c79-qqthj"]
Oct 31 02:04:06.280570 ip-10-0-133-197 kubenswrapper[4964]: W1031 02:04:06.280545    4964 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc7eddf8c_a855_4a72_b3ed_eacf3aaed5af.slice/crio-5ff71655dae35f955e09f80d95ba1a49fd62c9b4330153d14c244136c9bbe70b WatchSource:0}: Error finding container 5ff71655dae35f955e09f80d95ba1a49fd62c9b4330153d14c244136c9bbe70b: Status 404 returned error can't find the container with id 5ff71655dae35f955e09f80d95ba1a49fd62c9b4330153d14c244136c9bbe70b
Oct 31 02:04:06.283193 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:04:06.283156    4964 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 02:04:06.283310 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:04:06.283241    4964 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 02:04:06.283310 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:04:06.283285    4964 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 02:04:06.345304 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:04:06.345280    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-183266-openshift-default-74c6876c79-qqthj" event={"ID":"c7eddf8c-a855-4a72-b3ed-eacf3aaed5af","Type":"ContainerStarted","Data":"5ff71655dae35f955e09f80d95ba1a49fd62c9b4330153d14c244136c9bbe70b"}
Oct 31 02:04:07.353275 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:04:07.353240    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-183266-openshift-default-74c6876c79-qqthj" event={"ID":"c7eddf8c-a855-4a72-b3ed-eacf3aaed5af","Type":"ContainerStarted","Data":"998678af9b3d5cbb6f2d50be1b9eba437422d6b2be51a90aa02b87d2102e4abe"}
Oct 31 02:04:07.521404 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:04:07.521356    4964 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-autores-183266-openshift-default-74c6876c79-qqthj" podStartSLOduration=2.521306612 podStartE2EDuration="2.521306612s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 02:04:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 02:04:07.518187343 +0000 UTC m=+3615.994753744" watchObservedRunningTime="2026-10-31 02:04:07.521306612 +0000 UTC m=+3615.997872992"
Oct 31 02:04:07.717959 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:04:07.717880    4964 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-autores-183266-openshift-default-74c6876c79-qqthj"
Oct 31 02:04:07.723789 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:04:07.723766    4964 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-autores-183266-openshift-default-74c6876c79-qqthj"
Oct 31 02:04:08.360268 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:04:08.360240    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-autores-183266-openshift-default-74c6876c79-qqthj"
Oct 31 02:04:08.361486 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:04:08.361461    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-autores-183266-openshift-default-74c6876c79-qqthj"
Oct 31 02:04:08.406153 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:04:08.406120    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 02:04:10.202204 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:04:10.202173    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 02:04:10.389050 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:04:10.389016    4964 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w1-71bc98/auth-test-model-71bc98-kserve-77dd77fdfc-qln45"]
Oct 31 02:04:10.393110 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:04:10.393074    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 02:04:10.574155 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:04:10.574119    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 02:04:10.772425 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:04:10.772387    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 02:04:11.238789 ip-10-0-133-197 kubenswrapper[4964]: E1031 02:04:11.238755    4964 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q56x5" containerName="extractor"
Oct 31 02:04:13.481113 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:04:13.481066    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 02:04:15.833859 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:04:15.833816    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 02:04:16.009255 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:04:16.009220    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 02:04:18.298434 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:04:18.298390    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 02:04:23.098494 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:04:23.098453    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 02:04:25.411024 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:04:25.410961    4964 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w1-71bc98/auth-test-model-71bc98-kserve-77dd77fdfc-qln45" podUID="3d0e640a-41cb-48fe-9b7f-8927fd534ca4" containerName="main" containerID="cri-o://6155efa6a78befb0bca4b3f34af9276dbee8545d7973d851807fce8433268cb3" gracePeriod=45
Oct 31 02:04:25.709222 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:04:25.709197    4964 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-71bc98/auth-test-model-71bc98-kserve-77dd77fdfc-qln45"
Oct 31 02:04:25.850678 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:04:25.850643    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3d0e640a-41cb-48fe-9b7f-8927fd534ca4-kube-api-access-cbz52\" (UniqueName: \"kubernetes.io/projected/3d0e640a-41cb-48fe-9b7f-8927fd534ca4-kube-api-access-cbz52\") pod \"3d0e640a-41cb-48fe-9b7f-8927fd534ca4\" (UID: \"3d0e640a-41cb-48fe-9b7f-8927fd534ca4\") "
Oct 31 02:04:25.850898 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:04:25.850766    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3d0e640a-41cb-48fe-9b7f-8927fd534ca4-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3d0e640a-41cb-48fe-9b7f-8927fd534ca4-model-cache\") pod \"3d0e640a-41cb-48fe-9b7f-8927fd534ca4\" (UID: \"3d0e640a-41cb-48fe-9b7f-8927fd534ca4\") "
Oct 31 02:04:25.850898 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:04:25.850809    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3d0e640a-41cb-48fe-9b7f-8927fd534ca4-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3d0e640a-41cb-48fe-9b7f-8927fd534ca4-tmp-dir\") pod \"3d0e640a-41cb-48fe-9b7f-8927fd534ca4\" (UID: \"3d0e640a-41cb-48fe-9b7f-8927fd534ca4\") "
Oct 31 02:04:25.851039 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:04:25.850942    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3d0e640a-41cb-48fe-9b7f-8927fd534ca4-tls-certs\" (UniqueName: \"kubernetes.io/secret/3d0e640a-41cb-48fe-9b7f-8927fd534ca4-tls-certs\") pod \"3d0e640a-41cb-48fe-9b7f-8927fd534ca4\" (UID: \"3d0e640a-41cb-48fe-9b7f-8927fd534ca4\") "
Oct 31 02:04:25.851039 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:04:25.850983    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3d0e640a-41cb-48fe-9b7f-8927fd534ca4-dshm\" (UniqueName: \"kubernetes.io/empty-dir/3d0e640a-41cb-48fe-9b7f-8927fd534ca4-dshm\") pod \"3d0e640a-41cb-48fe-9b7f-8927fd534ca4\" (UID: \"3d0e640a-41cb-48fe-9b7f-8927fd534ca4\") "
Oct 31 02:04:25.851164 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:04:25.851047    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3d0e640a-41cb-48fe-9b7f-8927fd534ca4-home\" (UniqueName: \"kubernetes.io/empty-dir/3d0e640a-41cb-48fe-9b7f-8927fd534ca4-home\") pod \"3d0e640a-41cb-48fe-9b7f-8927fd534ca4\" (UID: \"3d0e640a-41cb-48fe-9b7f-8927fd534ca4\") "
Oct 31 02:04:25.851164 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:04:25.851112    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d0e640a-41cb-48fe-9b7f-8927fd534ca4-tmp-dir" pod "3d0e640a-41cb-48fe-9b7f-8927fd534ca4" (UID: "3d0e640a-41cb-48fe-9b7f-8927fd534ca4"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:04:25.851413 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:04:25.851388    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d0e640a-41cb-48fe-9b7f-8927fd534ca4-model-cache" pod "3d0e640a-41cb-48fe-9b7f-8927fd534ca4" (UID: "3d0e640a-41cb-48fe-9b7f-8927fd534ca4"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:04:25.851656 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:04:25.851628    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d0e640a-41cb-48fe-9b7f-8927fd534ca4-home" pod "3d0e640a-41cb-48fe-9b7f-8927fd534ca4" (UID: "3d0e640a-41cb-48fe-9b7f-8927fd534ca4"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:04:25.851809 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:04:25.851790    4964 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3d0e640a-41cb-48fe-9b7f-8927fd534ca4-home\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:04:25.851891 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:04:25.851812    4964 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3d0e640a-41cb-48fe-9b7f-8927fd534ca4-model-cache\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:04:25.851891 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:04:25.851827    4964 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3d0e640a-41cb-48fe-9b7f-8927fd534ca4-tmp-dir\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:04:25.853591 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:04:25.853563    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d0e640a-41cb-48fe-9b7f-8927fd534ca4-dshm" pod "3d0e640a-41cb-48fe-9b7f-8927fd534ca4" (UID: "3d0e640a-41cb-48fe-9b7f-8927fd534ca4"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:04:25.853815 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:04:25.853785    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3d0e640a-41cb-48fe-9b7f-8927fd534ca4-tls-certs" pod "3d0e640a-41cb-48fe-9b7f-8927fd534ca4" (UID: "3d0e640a-41cb-48fe-9b7f-8927fd534ca4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 02:04:25.853890 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:04:25.853859    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d0e640a-41cb-48fe-9b7f-8927fd534ca4-kube-api-access-cbz52" pod "3d0e640a-41cb-48fe-9b7f-8927fd534ca4" (UID: "3d0e640a-41cb-48fe-9b7f-8927fd534ca4"). InnerVolumeSpecName "kube-api-access-cbz52". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:04:25.953177 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:04:25.953086    4964 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3d0e640a-41cb-48fe-9b7f-8927fd534ca4-tls-certs\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:04:25.953177 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:04:25.953121    4964 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3d0e640a-41cb-48fe-9b7f-8927fd534ca4-dshm\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:04:25.953177 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:04:25.953135    4964 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cbz52\" (UniqueName: \"kubernetes.io/projected/3d0e640a-41cb-48fe-9b7f-8927fd534ca4-kube-api-access-cbz52\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:04:26.488576 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:04:26.488544    4964 generic.go:356] "Generic (PLEG): container finished" podID="3d0e640a-41cb-48fe-9b7f-8927fd534ca4" containerID="6155efa6a78befb0bca4b3f34af9276dbee8545d7973d851807fce8433268cb3" exitCode=0
Oct 31 02:04:26.489072 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:04:26.488602    4964 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-71bc98/auth-test-model-71bc98-kserve-77dd77fdfc-qln45"
Oct 31 02:04:26.489072 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:04:26.488626    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-71bc98/auth-test-model-71bc98-kserve-77dd77fdfc-qln45" event={"ID":"3d0e640a-41cb-48fe-9b7f-8927fd534ca4","Type":"ContainerDied","Data":"6155efa6a78befb0bca4b3f34af9276dbee8545d7973d851807fce8433268cb3"}
Oct 31 02:04:26.489072 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:04:26.488679    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-71bc98/auth-test-model-71bc98-kserve-77dd77fdfc-qln45" event={"ID":"3d0e640a-41cb-48fe-9b7f-8927fd534ca4","Type":"ContainerDied","Data":"7a641b433b44a56cff93fee57e1cc92930be87d2352cf267a13c0d87baaa86a4"}
Oct 31 02:04:26.489072 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:04:26.488698    4964 scope.go:122] "RemoveContainer" containerID="6155efa6a78befb0bca4b3f34af9276dbee8545d7973d851807fce8433268cb3"
Oct 31 02:04:26.501237 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:04:26.501218    4964 scope.go:122] "RemoveContainer" containerID="6155efa6a78befb0bca4b3f34af9276dbee8545d7973d851807fce8433268cb3"
Oct 31 02:04:26.501591 ip-10-0-133-197 kubenswrapper[4964]: E1031 02:04:26.501573    4964 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6155efa6a78befb0bca4b3f34af9276dbee8545d7973d851807fce8433268cb3\": container with ID starting with 6155efa6a78befb0bca4b3f34af9276dbee8545d7973d851807fce8433268cb3 not found: ID does not exist" containerID="6155efa6a78befb0bca4b3f34af9276dbee8545d7973d851807fce8433268cb3"
Oct 31 02:04:26.501675 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:04:26.501603    4964 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6155efa6a78befb0bca4b3f34af9276dbee8545d7973d851807fce8433268cb3"} err="failed to get container status \"6155efa6a78befb0bca4b3f34af9276dbee8545d7973d851807fce8433268cb3\": rpc error: code = NotFound desc = could not find container \"6155efa6a78befb0bca4b3f34af9276dbee8545d7973d851807fce8433268cb3\": container with ID starting with 6155efa6a78befb0bca4b3f34af9276dbee8545d7973d851807fce8433268cb3 not found: ID does not exist"
Oct 31 02:04:26.652594 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:04:26.652555    4964 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w1-71bc98/auth-test-model-71bc98-kserve-77dd77fdfc-qln45"]
Oct 31 02:04:26.703169 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:04:26.703140    4964 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w1-71bc98/auth-test-model-71bc98-kserve-77dd77fdfc-qln45"]
Oct 31 02:04:27.778196 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:04:27.778161    4964 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3d0e640a-41cb-48fe-9b7f-8927fd534ca4" path="/var/lib/kubelet/pods/3d0e640a-41cb-48fe-9b7f-8927fd534ca4/volumes"
Oct 31 02:04:27.871838 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:04:27.871798    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 02:04:32.574528 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:04:32.574452    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 02:04:37.422780 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:04:37.422740    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 02:04:41.239697 ip-10-0-133-197 kubenswrapper[4964]: E1031 02:04:41.239655    4964 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q56x5" containerName="extractor"
Oct 31 02:04:42.594301 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:04:42.594265    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 02:04:47.641882 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:04:47.641852    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 02:04:48.080878 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:04:48.080844    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 02:04:48.967040 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:04:48.967001    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 02:04:49.136422 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:04:49.136380    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 02:04:52.431135 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:04:52.431099    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 02:04:53.616990 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:04:53.616954    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 02:04:56.326421 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:04:56.326384    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 02:04:56.463444 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:04:56.463409    4964 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-c4be41-6c8c8d4b57-7czj4"]
Oct 31 02:04:56.463676 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:04:56.463616    4964 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-c4be41-6c8c8d4b57-7czj4" podUID="1036dce3-2ee4-4dc2-a8e9-094d35d989ef" containerName="maas-api" containerID="cri-o://a6f72059569fe67768d8510f2317673c84afb0adb3381b017a5eb4b1d9c1e4cb" gracePeriod=30
Oct 31 02:04:56.612753 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:04:56.612712    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 02:04:56.729138 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:04:56.729110    4964 generic.go:356] "Generic (PLEG): container finished" podID="1036dce3-2ee4-4dc2-a8e9-094d35d989ef" containerID="a6f72059569fe67768d8510f2317673c84afb0adb3381b017a5eb4b1d9c1e4cb" exitCode=0
Oct 31 02:04:56.729367 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:04:56.729187    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-c4be41-6c8c8d4b57-7czj4" event={"ID":"1036dce3-2ee4-4dc2-a8e9-094d35d989ef","Type":"ContainerDied","Data":"a6f72059569fe67768d8510f2317673c84afb0adb3381b017a5eb4b1d9c1e4cb"}
Oct 31 02:04:56.753738 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:04:56.753716    4964 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-c4be41-6c8c8d4b57-7czj4"
Oct 31 02:04:56.855339 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:04:56.855212    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1036dce3-2ee4-4dc2-a8e9-094d35d989ef-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1036dce3-2ee4-4dc2-a8e9-094d35d989ef-openshift-service-ca\") pod \"1036dce3-2ee4-4dc2-a8e9-094d35d989ef\" (UID: \"1036dce3-2ee4-4dc2-a8e9-094d35d989ef\") "
Oct 31 02:04:56.855531 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:04:56.855362    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1036dce3-2ee4-4dc2-a8e9-094d35d989ef-kube-api-access-wg6hm\" (UniqueName: \"kubernetes.io/projected/1036dce3-2ee4-4dc2-a8e9-094d35d989ef-kube-api-access-wg6hm\") pod \"1036dce3-2ee4-4dc2-a8e9-094d35d989ef\" (UID: \"1036dce3-2ee4-4dc2-a8e9-094d35d989ef\") "
Oct 31 02:04:56.855531 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:04:56.855397    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1036dce3-2ee4-4dc2-a8e9-094d35d989ef-metrics-certs\" (UniqueName: \"kubernetes.io/secret/1036dce3-2ee4-4dc2-a8e9-094d35d989ef-metrics-certs\") pod \"1036dce3-2ee4-4dc2-a8e9-094d35d989ef\" (UID: \"1036dce3-2ee4-4dc2-a8e9-094d35d989ef\") "
Oct 31 02:04:56.855660 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:04:56.855561    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1036dce3-2ee4-4dc2-a8e9-094d35d989ef-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1036dce3-2ee4-4dc2-a8e9-094d35d989ef-maas-api-tls\") pod \"1036dce3-2ee4-4dc2-a8e9-094d35d989ef\" (UID: \"1036dce3-2ee4-4dc2-a8e9-094d35d989ef\") "
Oct 31 02:04:56.858881 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:04:56.858852    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1036dce3-2ee4-4dc2-a8e9-094d35d989ef-metrics-certs" pod "1036dce3-2ee4-4dc2-a8e9-094d35d989ef" (UID: "1036dce3-2ee4-4dc2-a8e9-094d35d989ef"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 02:04:56.859427 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:04:56.859400    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1036dce3-2ee4-4dc2-a8e9-094d35d989ef-openshift-service-ca" pod "1036dce3-2ee4-4dc2-a8e9-094d35d989ef" (UID: "1036dce3-2ee4-4dc2-a8e9-094d35d989ef"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 02:04:56.860233 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:04:56.860197    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1036dce3-2ee4-4dc2-a8e9-094d35d989ef-kube-api-access-wg6hm" pod "1036dce3-2ee4-4dc2-a8e9-094d35d989ef" (UID: "1036dce3-2ee4-4dc2-a8e9-094d35d989ef"). InnerVolumeSpecName "kube-api-access-wg6hm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:04:56.865499 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:04:56.865161    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1036dce3-2ee4-4dc2-a8e9-094d35d989ef-maas-api-tls" pod "1036dce3-2ee4-4dc2-a8e9-094d35d989ef" (UID: "1036dce3-2ee4-4dc2-a8e9-094d35d989ef"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 02:04:56.959903 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:04:56.959870    4964 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1036dce3-2ee4-4dc2-a8e9-094d35d989ef-openshift-service-ca\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:04:56.959903 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:04:56.959899    4964 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wg6hm\" (UniqueName: \"kubernetes.io/projected/1036dce3-2ee4-4dc2-a8e9-094d35d989ef-kube-api-access-wg6hm\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:04:56.959903 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:04:56.959910    4964 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1036dce3-2ee4-4dc2-a8e9-094d35d989ef-metrics-certs\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:04:56.960157 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:04:56.959920    4964 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1036dce3-2ee4-4dc2-a8e9-094d35d989ef-maas-api-tls\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:04:57.738839 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:04:57.738812    4964 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-c4be41-6c8c8d4b57-7czj4"
Oct 31 02:04:57.739368 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:04:57.738806    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-c4be41-6c8c8d4b57-7czj4" event={"ID":"1036dce3-2ee4-4dc2-a8e9-094d35d989ef","Type":"ContainerDied","Data":"88c47a0442ce60a6d41ff7fa21e1d0a61b1dfbb2994e3cab4d3d73a87772f325"}
Oct 31 02:04:57.739368 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:04:57.738957    4964 scope.go:122] "RemoveContainer" containerID="a6f72059569fe67768d8510f2317673c84afb0adb3381b017a5eb4b1d9c1e4cb"
Oct 31 02:04:57.924265 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:04:57.924230    4964 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-c4be41-6c8c8d4b57-7czj4"]
Oct 31 02:04:57.973032 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:04:57.972998    4964 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-c4be41-6c8c8d4b57-7czj4"]
Oct 31 02:04:58.181259 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:04:58.181162    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 02:04:58.399794 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:04:58.399749    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 02:04:58.571330 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:04:58.571279    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 02:04:59.111536 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:04:59.111486    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 02:04:59.786045 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:04:59.785994    4964 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1036dce3-2ee4-4dc2-a8e9-094d35d989ef" path="/var/lib/kubelet/pods/1036dce3-2ee4-4dc2-a8e9-094d35d989ef/volumes"
Oct 31 02:05:02.300470 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:05:02.300431    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 02:05:03.468713 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:05:03.468674    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 02:05:04.727196 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:05:04.727155    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 02:05:04.968422 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:05:04.968385    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 02:05:05.023169 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:05:05.023058    4964 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-explicit-c4be41-openshift-default-686574f97c-phvtc"]
Oct 31 02:05:05.023392 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:05:05.023346    4964 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-explicit-c4be41-openshift-default-686574f97c-phvtc" podUID="8508d596-d89b-4f67-9a57-f6ba4890ce5e" containerName="istio-proxy" containerID="cri-o://b19b565315490533c56c2b4b490e0515ae9254b63031049b50c9900772883685" gracePeriod=30
Oct 31 02:05:05.148013 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:05:05.147973    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 02:05:05.890444 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:05:05.890410    4964 patch_prober.go:29] interesting pod/e2e-explicit-c4be41-openshift-default-686574f97c-phvtc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 02:05:05.890901 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:05:05.890478    4964 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-explicit-c4be41-openshift-default-686574f97c-phvtc" podUID="8508d596-d89b-4f67-9a57-f6ba4890ce5e" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 02:05:08.246911 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:05:08.246875    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 02:05:10.304820 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:05:10.304759    4964 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-c4be41-openshift-default-686574f97c-phvtc"
Oct 31 02:05:10.436774 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:05:10.436741    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8508d596-d89b-4f67-9a57-f6ba4890ce5e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8508d596-d89b-4f67-9a57-f6ba4890ce5e-workload-socket\") pod \"8508d596-d89b-4f67-9a57-f6ba4890ce5e\" (UID: \"8508d596-d89b-4f67-9a57-f6ba4890ce5e\") "
Oct 31 02:05:10.436975 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:05:10.436823    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8508d596-d89b-4f67-9a57-f6ba4890ce5e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8508d596-d89b-4f67-9a57-f6ba4890ce5e-istio-envoy\") pod \"8508d596-d89b-4f67-9a57-f6ba4890ce5e\" (UID: \"8508d596-d89b-4f67-9a57-f6ba4890ce5e\") "
Oct 31 02:05:10.436975 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:05:10.436865    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8508d596-d89b-4f67-9a57-f6ba4890ce5e-istio-token\" (UniqueName: \"kubernetes.io/projected/8508d596-d89b-4f67-9a57-f6ba4890ce5e-istio-token\") pod \"8508d596-d89b-4f67-9a57-f6ba4890ce5e\" (UID: \"8508d596-d89b-4f67-9a57-f6ba4890ce5e\") "
Oct 31 02:05:10.436975 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:05:10.436923    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8508d596-d89b-4f67-9a57-f6ba4890ce5e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8508d596-d89b-4f67-9a57-f6ba4890ce5e-istiod-ca-cert\") pod \"8508d596-d89b-4f67-9a57-f6ba4890ce5e\" (UID: \"8508d596-d89b-4f67-9a57-f6ba4890ce5e\") "
Oct 31 02:05:10.437147 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:05:10.437055    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8508d596-d89b-4f67-9a57-f6ba4890ce5e-workload-socket" pod "8508d596-d89b-4f67-9a57-f6ba4890ce5e" (UID: "8508d596-d89b-4f67-9a57-f6ba4890ce5e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:05:10.437147 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:05:10.437078    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8508d596-d89b-4f67-9a57-f6ba4890ce5e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8508d596-d89b-4f67-9a57-f6ba4890ce5e-istio-podinfo\") pod \"8508d596-d89b-4f67-9a57-f6ba4890ce5e\" (UID: \"8508d596-d89b-4f67-9a57-f6ba4890ce5e\") "
Oct 31 02:05:10.437265 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:05:10.437168    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8508d596-d89b-4f67-9a57-f6ba4890ce5e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8508d596-d89b-4f67-9a57-f6ba4890ce5e-workload-certs\") pod \"8508d596-d89b-4f67-9a57-f6ba4890ce5e\" (UID: \"8508d596-d89b-4f67-9a57-f6ba4890ce5e\") "
Oct 31 02:05:10.437265 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:05:10.437213    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8508d596-d89b-4f67-9a57-f6ba4890ce5e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8508d596-d89b-4f67-9a57-f6ba4890ce5e-istio-data\") pod \"8508d596-d89b-4f67-9a57-f6ba4890ce5e\" (UID: \"8508d596-d89b-4f67-9a57-f6ba4890ce5e\") "
Oct 31 02:05:10.437443 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:05:10.437277    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8508d596-d89b-4f67-9a57-f6ba4890ce5e-kube-api-access-hzdn9\" (UniqueName: \"kubernetes.io/projected/8508d596-d89b-4f67-9a57-f6ba4890ce5e-kube-api-access-hzdn9\") pod \"8508d596-d89b-4f67-9a57-f6ba4890ce5e\" (UID: \"8508d596-d89b-4f67-9a57-f6ba4890ce5e\") "
Oct 31 02:05:10.437498 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:05:10.437440    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8508d596-d89b-4f67-9a57-f6ba4890ce5e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8508d596-d89b-4f67-9a57-f6ba4890ce5e-credential-socket\") pod \"8508d596-d89b-4f67-9a57-f6ba4890ce5e\" (UID: \"8508d596-d89b-4f67-9a57-f6ba4890ce5e\") "
Oct 31 02:05:10.437498 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:05:10.437384    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8508d596-d89b-4f67-9a57-f6ba4890ce5e-istiod-ca-cert" pod "8508d596-d89b-4f67-9a57-f6ba4890ce5e" (UID: "8508d596-d89b-4f67-9a57-f6ba4890ce5e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 02:05:10.437498 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:05:10.437478    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8508d596-d89b-4f67-9a57-f6ba4890ce5e-workload-certs" pod "8508d596-d89b-4f67-9a57-f6ba4890ce5e" (UID: "8508d596-d89b-4f67-9a57-f6ba4890ce5e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:05:10.437786 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:05:10.437683    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8508d596-d89b-4f67-9a57-f6ba4890ce5e-istio-data" pod "8508d596-d89b-4f67-9a57-f6ba4890ce5e" (UID: "8508d596-d89b-4f67-9a57-f6ba4890ce5e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:05:10.438740 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:05:10.438200    4964 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8508d596-d89b-4f67-9a57-f6ba4890ce5e-workload-socket\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:05:10.438740 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:05:10.438226    4964 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8508d596-d89b-4f67-9a57-f6ba4890ce5e-istiod-ca-cert\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:05:10.438740 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:05:10.438240    4964 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8508d596-d89b-4f67-9a57-f6ba4890ce5e-workload-certs\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:05:10.438740 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:05:10.438253    4964 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8508d596-d89b-4f67-9a57-f6ba4890ce5e-istio-data\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:05:10.438740 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:05:10.438699    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8508d596-d89b-4f67-9a57-f6ba4890ce5e-credential-socket" pod "8508d596-d89b-4f67-9a57-f6ba4890ce5e" (UID: "8508d596-d89b-4f67-9a57-f6ba4890ce5e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:05:10.439630 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:05:10.439593    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8508d596-d89b-4f67-9a57-f6ba4890ce5e-istio-podinfo" pod "8508d596-d89b-4f67-9a57-f6ba4890ce5e" (UID: "8508d596-d89b-4f67-9a57-f6ba4890ce5e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 02:05:10.439829 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:05:10.439809    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8508d596-d89b-4f67-9a57-f6ba4890ce5e-istio-token" pod "8508d596-d89b-4f67-9a57-f6ba4890ce5e" (UID: "8508d596-d89b-4f67-9a57-f6ba4890ce5e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:05:10.440515 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:05:10.440480    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8508d596-d89b-4f67-9a57-f6ba4890ce5e-istio-envoy" pod "8508d596-d89b-4f67-9a57-f6ba4890ce5e" (UID: "8508d596-d89b-4f67-9a57-f6ba4890ce5e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:05:10.440515 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:05:10.440492    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8508d596-d89b-4f67-9a57-f6ba4890ce5e-kube-api-access-hzdn9" pod "8508d596-d89b-4f67-9a57-f6ba4890ce5e" (UID: "8508d596-d89b-4f67-9a57-f6ba4890ce5e"). InnerVolumeSpecName "kube-api-access-hzdn9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:05:10.539143 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:05:10.539107    4964 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8508d596-d89b-4f67-9a57-f6ba4890ce5e-credential-socket\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:05:10.539143 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:05:10.539136    4964 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8508d596-d89b-4f67-9a57-f6ba4890ce5e-istio-envoy\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:05:10.539143 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:05:10.539146    4964 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8508d596-d89b-4f67-9a57-f6ba4890ce5e-istio-token\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:05:10.539403 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:05:10.539154    4964 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8508d596-d89b-4f67-9a57-f6ba4890ce5e-istio-podinfo\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:05:10.539403 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:05:10.539163    4964 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hzdn9\" (UniqueName: \"kubernetes.io/projected/8508d596-d89b-4f67-9a57-f6ba4890ce5e-kube-api-access-hzdn9\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:05:10.542145 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:05:10.542118    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 02:05:10.836103 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:05:10.836064    4964 generic.go:356] "Generic (PLEG): container finished" podID="8508d596-d89b-4f67-9a57-f6ba4890ce5e" containerID="b19b565315490533c56c2b4b490e0515ae9254b63031049b50c9900772883685" exitCode=0
Oct 31 02:05:10.836347 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:05:10.836109    4964 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-c4be41-openshift-default-686574f97c-phvtc"
Oct 31 02:05:10.836347 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:05:10.836117    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-c4be41-openshift-default-686574f97c-phvtc" event={"ID":"8508d596-d89b-4f67-9a57-f6ba4890ce5e","Type":"ContainerDied","Data":"b19b565315490533c56c2b4b490e0515ae9254b63031049b50c9900772883685"}
Oct 31 02:05:10.836347 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:05:10.836175    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-c4be41-openshift-default-686574f97c-phvtc" event={"ID":"8508d596-d89b-4f67-9a57-f6ba4890ce5e","Type":"ContainerDied","Data":"dadbaff2de553e33f64c8e633c50718783b85bcafe9f0019cc91627e5804d959"}
Oct 31 02:05:10.836347 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:05:10.836207    4964 scope.go:122] "RemoveContainer" containerID="b19b565315490533c56c2b4b490e0515ae9254b63031049b50c9900772883685"
Oct 31 02:05:10.847819 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:05:10.847803    4964 scope.go:122] "RemoveContainer" containerID="5ff20856d937574be979e19ea8bc362cb93a09d20aa7cca886dd99d959280830"
Oct 31 02:05:10.857933 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:05:10.857917    4964 scope.go:122] "RemoveContainer" containerID="b19b565315490533c56c2b4b490e0515ae9254b63031049b50c9900772883685"
Oct 31 02:05:10.858147 ip-10-0-133-197 kubenswrapper[4964]: E1031 02:05:10.858126    4964 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b19b565315490533c56c2b4b490e0515ae9254b63031049b50c9900772883685\": container with ID starting with b19b565315490533c56c2b4b490e0515ae9254b63031049b50c9900772883685 not found: ID does not exist" containerID="b19b565315490533c56c2b4b490e0515ae9254b63031049b50c9900772883685"
Oct 31 02:05:10.858206 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:05:10.858154    4964 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b19b565315490533c56c2b4b490e0515ae9254b63031049b50c9900772883685"} err="failed to get container status \"b19b565315490533c56c2b4b490e0515ae9254b63031049b50c9900772883685\": rpc error: code = NotFound desc = could not find container \"b19b565315490533c56c2b4b490e0515ae9254b63031049b50c9900772883685\": container with ID starting with b19b565315490533c56c2b4b490e0515ae9254b63031049b50c9900772883685 not found: ID does not exist"
Oct 31 02:05:10.858206 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:05:10.858170    4964 scope.go:122] "RemoveContainer" containerID="5ff20856d937574be979e19ea8bc362cb93a09d20aa7cca886dd99d959280830"
Oct 31 02:05:10.858437 ip-10-0-133-197 kubenswrapper[4964]: E1031 02:05:10.858418    4964 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5ff20856d937574be979e19ea8bc362cb93a09d20aa7cca886dd99d959280830\": container with ID starting with 5ff20856d937574be979e19ea8bc362cb93a09d20aa7cca886dd99d959280830 not found: ID does not exist" containerID="5ff20856d937574be979e19ea8bc362cb93a09d20aa7cca886dd99d959280830"
Oct 31 02:05:10.858502 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:05:10.858441    4964 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5ff20856d937574be979e19ea8bc362cb93a09d20aa7cca886dd99d959280830"} err="failed to get container status \"5ff20856d937574be979e19ea8bc362cb93a09d20aa7cca886dd99d959280830\": rpc error: code = NotFound desc = could not find container \"5ff20856d937574be979e19ea8bc362cb93a09d20aa7cca886dd99d959280830\": container with ID starting with 5ff20856d937574be979e19ea8bc362cb93a09d20aa7cca886dd99d959280830 not found: ID does not exist"
Oct 31 02:05:10.999212 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:05:10.999182    4964 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-explicit-c4be41-openshift-default-686574f97c-phvtc"]
Oct 31 02:05:11.048468 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:05:11.048439    4964 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-explicit-c4be41-openshift-default-686574f97c-phvtc"]
Oct 31 02:05:11.239261 ip-10-0-133-197 kubenswrapper[4964]: E1031 02:05:11.239223    4964 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q56x5" containerName="extractor"
Oct 31 02:05:11.777085 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:05:11.777056    4964 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8508d596-d89b-4f67-9a57-f6ba4890ce5e" path="/var/lib/kubelet/pods/8508d596-d89b-4f67-9a57-f6ba4890ce5e/volumes"
Oct 31 02:05:12.989527 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:05:12.989495    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 02:05:15.687442 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:05:15.687403    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 02:05:16.292569 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:05:16.292533    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 02:05:16.553119 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:05:16.553022    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 02:05:17.742554 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:05:17.742499    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 02:05:18.086753 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:05:18.086659    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 02:05:18.274562 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:05:18.274525    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 02:05:18.456977 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:05:18.456878    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 02:05:22.409564 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:05:22.409524    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 02:05:22.588591 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:05:22.588555    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 02:05:22.765656 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:05:22.765623    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 02:05:27.414778 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:05:27.414739    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 02:05:32.218816 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:05:32.218773    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 02:05:37.004004 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:05:37.003967    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 02:05:40.793132 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:05:40.793100    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 02:05:41.240912 ip-10-0-133-197 kubenswrapper[4964]: E1031 02:05:41.240862    4964 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q56x5" containerName="extractor"
Oct 31 02:05:41.801432 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:05:41.801397    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 02:05:46.624369 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:05:46.622067    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 02:05:51.397722 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:05:51.397679    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 02:05:53.538336 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:05:53.538267    4964 patch_prober.go:29] interesting pod/e2e-autores-183266-openshift-default-74c6876c79-qqthj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.191:15021/healthz/ready\": read tcp 10.132.0.2:49370->10.132.0.191:15021: read: connection reset by peer" start-of-body=
Oct 31 02:05:53.538808 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:05:53.538391    4964 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-autores-183266-openshift-default-74c6876c79-qqthj" podUID="c7eddf8c-a855-4a72-b3ed-eacf3aaed5af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.191:15021/healthz/ready\": read tcp 10.132.0.2:49370->10.132.0.191:15021: read: connection reset by peer"
Oct 31 02:05:54.183618 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:05:54.183583    4964 generic.go:356] "Generic (PLEG): container finished" podID="c7eddf8c-a855-4a72-b3ed-eacf3aaed5af" containerID="998678af9b3d5cbb6f2d50be1b9eba437422d6b2be51a90aa02b87d2102e4abe" exitCode=0
Oct 31 02:05:54.183832 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:05:54.183661    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-183266-openshift-default-74c6876c79-qqthj" event={"ID":"c7eddf8c-a855-4a72-b3ed-eacf3aaed5af","Type":"ContainerDied","Data":"998678af9b3d5cbb6f2d50be1b9eba437422d6b2be51a90aa02b87d2102e4abe"}
Oct 31 02:05:54.184147 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:05:54.184127    4964 scope.go:122] "RemoveContainer" containerID="998678af9b3d5cbb6f2d50be1b9eba437422d6b2be51a90aa02b87d2102e4abe"
Oct 31 02:05:54.185837 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:05:54.185812    4964 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 02:05:54.185907 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:05:54.185876    4964 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 02:05:54.185945 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:05:54.185909    4964 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 02:05:55.223107 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:05:55.223070    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-183266-openshift-default-74c6876c79-qqthj" event={"ID":"c7eddf8c-a855-4a72-b3ed-eacf3aaed5af","Type":"ContainerStarted","Data":"d7ba0c36d9f6cb9f697bc897d8af55562419ea96a5e71aeb5eba1022f6077f67"}
Oct 31 02:05:55.723443 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:05:55.721731    4964 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-autores-183266-openshift-default-74c6876c79-qqthj"
Oct 31 02:05:55.723443 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:05:55.721857    4964 patch_prober.go:29] interesting pod/e2e-autores-183266-openshift-default-74c6876c79-qqthj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.191:15021/healthz/ready\": dial tcp 10.132.0.191:15021: connect: connection refused" start-of-body=
Oct 31 02:05:55.723443 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:05:55.721909    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-183266-openshift-default-74c6876c79-qqthj" podUID="c7eddf8c-a855-4a72-b3ed-eacf3aaed5af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.191:15021/healthz/ready\": dial tcp 10.132.0.191:15021: connect: connection refused"
Oct 31 02:05:56.182147 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:05:56.182051    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 02:05:56.724344 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:05:56.718661    4964 patch_prober.go:29] interesting pod/e2e-autores-183266-openshift-default-74c6876c79-qqthj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.191:15021/healthz/ready\": dial tcp 10.132.0.191:15021: connect: connection refused" start-of-body=
Oct 31 02:05:56.724344 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:05:56.718721    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-183266-openshift-default-74c6876c79-qqthj" podUID="c7eddf8c-a855-4a72-b3ed-eacf3aaed5af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.191:15021/healthz/ready\": dial tcp 10.132.0.191:15021: connect: connection refused"
Oct 31 02:05:57.721423 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:05:57.719621    4964 patch_prober.go:29] interesting pod/e2e-autores-183266-openshift-default-74c6876c79-qqthj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.191:15021/healthz/ready\": dial tcp 10.132.0.191:15021: connect: connection refused" start-of-body=
Oct 31 02:05:57.721423 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:05:57.719680    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-183266-openshift-default-74c6876c79-qqthj" podUID="c7eddf8c-a855-4a72-b3ed-eacf3aaed5af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.191:15021/healthz/ready\": dial tcp 10.132.0.191:15021: connect: connection refused"
Oct 31 02:05:57.922994 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:05:57.922955    4964 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-5qwk9"]
Oct 31 02:05:57.923561 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:05:57.923178    4964 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-5qwk9" podUID="383b0139-d5c3-4492-8245-8faac4af0a10" containerName="istio-proxy" containerID="cri-o://b47abc86de11e36471640ebf86f6793403f1d7dfaef6cabac1363f3e801cb706" gracePeriod=30
Oct 31 02:05:58.619449 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:05:58.619375    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 02:05:58.718348 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:05:58.718294    4964 patch_prober.go:29] interesting pod/e2e-autores-183266-openshift-default-74c6876c79-qqthj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.191:15021/healthz/ready\": dial tcp 10.132.0.191:15021: connect: connection refused" start-of-body=
Oct 31 02:05:58.718640 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:05:58.718610    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-183266-openshift-default-74c6876c79-qqthj" podUID="c7eddf8c-a855-4a72-b3ed-eacf3aaed5af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.191:15021/healthz/ready\": dial tcp 10.132.0.191:15021: connect: connection refused"
Oct 31 02:05:59.717686 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:05:59.717638    4964 patch_prober.go:29] interesting pod/e2e-autores-183266-openshift-default-74c6876c79-qqthj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.191:15021/healthz/ready\": dial tcp 10.132.0.191:15021: connect: connection refused" start-of-body=
Oct 31 02:05:59.718199 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:05:59.717693    4964 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-183266-openshift-default-74c6876c79-qqthj" podUID="c7eddf8c-a855-4a72-b3ed-eacf3aaed5af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.191:15021/healthz/ready\": dial tcp 10.132.0.191:15021: connect: connection refused"
Oct 31 02:06:00.722524 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:06:00.722448    4964 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-autores-183266-openshift-default-74c6876c79-qqthj"
Oct 31 02:06:00.722923 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:06:00.722824    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-autores-183266-openshift-default-74c6876c79-qqthj"
Oct 31 02:06:00.723582 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:06:00.723564    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-autores-183266-openshift-default-74c6876c79-qqthj"
Oct 31 02:06:01.053878 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:06:01.053768    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 02:06:02.624006 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:06:02.623972    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 02:06:03.201862 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:06:03.201806    4964 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-5qwk9"
Oct 31 02:06:03.285748 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:06:03.285713    4964 generic.go:356] "Generic (PLEG): container finished" podID="383b0139-d5c3-4492-8245-8faac4af0a10" containerID="b47abc86de11e36471640ebf86f6793403f1d7dfaef6cabac1363f3e801cb706" exitCode=0
Oct 31 02:06:03.285936 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:06:03.285763    4964 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-5qwk9"
Oct 31 02:06:03.285936 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:06:03.285768    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-5qwk9" event={"ID":"383b0139-d5c3-4492-8245-8faac4af0a10","Type":"ContainerDied","Data":"b47abc86de11e36471640ebf86f6793403f1d7dfaef6cabac1363f3e801cb706"}
Oct 31 02:06:03.285936 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:06:03.285835    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-5qwk9" event={"ID":"383b0139-d5c3-4492-8245-8faac4af0a10","Type":"ContainerDied","Data":"92efe09a8d8052009c17db6dec62f57aacf3ea954ab389436ee961956857e6de"}
Oct 31 02:06:03.285936 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:06:03.285856    4964 scope.go:122] "RemoveContainer" containerID="b47abc86de11e36471640ebf86f6793403f1d7dfaef6cabac1363f3e801cb706"
Oct 31 02:06:03.296293 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:06:03.296272    4964 scope.go:122] "RemoveContainer" containerID="b47abc86de11e36471640ebf86f6793403f1d7dfaef6cabac1363f3e801cb706"
Oct 31 02:06:03.296617 ip-10-0-133-197 kubenswrapper[4964]: E1031 02:06:03.296596    4964 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b47abc86de11e36471640ebf86f6793403f1d7dfaef6cabac1363f3e801cb706\": container with ID starting with b47abc86de11e36471640ebf86f6793403f1d7dfaef6cabac1363f3e801cb706 not found: ID does not exist" containerID="b47abc86de11e36471640ebf86f6793403f1d7dfaef6cabac1363f3e801cb706"
Oct 31 02:06:03.296706 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:06:03.296635    4964 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b47abc86de11e36471640ebf86f6793403f1d7dfaef6cabac1363f3e801cb706"} err="failed to get container status \"b47abc86de11e36471640ebf86f6793403f1d7dfaef6cabac1363f3e801cb706\": rpc error: code = NotFound desc = could not find container \"b47abc86de11e36471640ebf86f6793403f1d7dfaef6cabac1363f3e801cb706\": container with ID starting with b47abc86de11e36471640ebf86f6793403f1d7dfaef6cabac1363f3e801cb706 not found: ID does not exist"
Oct 31 02:06:03.322124 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:06:03.322100    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/383b0139-d5c3-4492-8245-8faac4af0a10-istio-token\" (UniqueName: \"kubernetes.io/projected/383b0139-d5c3-4492-8245-8faac4af0a10-istio-token\") pod \"383b0139-d5c3-4492-8245-8faac4af0a10\" (UID: \"383b0139-d5c3-4492-8245-8faac4af0a10\") "
Oct 31 02:06:03.322230 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:06:03.322184    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/383b0139-d5c3-4492-8245-8faac4af0a10-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/383b0139-d5c3-4492-8245-8faac4af0a10-istiod-ca-cert\") pod \"383b0139-d5c3-4492-8245-8faac4af0a10\" (UID: \"383b0139-d5c3-4492-8245-8faac4af0a10\") "
Oct 31 02:06:03.322230 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:06:03.322223    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/383b0139-d5c3-4492-8245-8faac4af0a10-kube-api-access-wh8sq\" (UniqueName: \"kubernetes.io/projected/383b0139-d5c3-4492-8245-8faac4af0a10-kube-api-access-wh8sq\") pod \"383b0139-d5c3-4492-8245-8faac4af0a10\" (UID: \"383b0139-d5c3-4492-8245-8faac4af0a10\") "
Oct 31 02:06:03.322335 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:06:03.322246    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/383b0139-d5c3-4492-8245-8faac4af0a10-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/383b0139-d5c3-4492-8245-8faac4af0a10-istio-podinfo\") pod \"383b0139-d5c3-4492-8245-8faac4af0a10\" (UID: \"383b0139-d5c3-4492-8245-8faac4af0a10\") "
Oct 31 02:06:03.322335 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:06:03.322273    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/383b0139-d5c3-4492-8245-8faac4af0a10-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/383b0139-d5c3-4492-8245-8faac4af0a10-istio-data\") pod \"383b0139-d5c3-4492-8245-8faac4af0a10\" (UID: \"383b0139-d5c3-4492-8245-8faac4af0a10\") "
Oct 31 02:06:03.322450 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:06:03.322406    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/383b0139-d5c3-4492-8245-8faac4af0a10-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/383b0139-d5c3-4492-8245-8faac4af0a10-credential-socket\") pod \"383b0139-d5c3-4492-8245-8faac4af0a10\" (UID: \"383b0139-d5c3-4492-8245-8faac4af0a10\") "
Oct 31 02:06:03.322536 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:06:03.322512    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/383b0139-d5c3-4492-8245-8faac4af0a10-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/383b0139-d5c3-4492-8245-8faac4af0a10-istio-envoy\") pod \"383b0139-d5c3-4492-8245-8faac4af0a10\" (UID: \"383b0139-d5c3-4492-8245-8faac4af0a10\") "
Oct 31 02:06:03.322604 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:06:03.322557    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/383b0139-d5c3-4492-8245-8faac4af0a10-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/383b0139-d5c3-4492-8245-8faac4af0a10-workload-certs\") pod \"383b0139-d5c3-4492-8245-8faac4af0a10\" (UID: \"383b0139-d5c3-4492-8245-8faac4af0a10\") "
Oct 31 02:06:03.322604 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:06:03.322578    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/383b0139-d5c3-4492-8245-8faac4af0a10-istiod-ca-cert" pod "383b0139-d5c3-4492-8245-8faac4af0a10" (UID: "383b0139-d5c3-4492-8245-8faac4af0a10"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 02:06:03.322728 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:06:03.322663    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/383b0139-d5c3-4492-8245-8faac4af0a10-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/383b0139-d5c3-4492-8245-8faac4af0a10-workload-socket\") pod \"383b0139-d5c3-4492-8245-8faac4af0a10\" (UID: \"383b0139-d5c3-4492-8245-8faac4af0a10\") "
Oct 31 02:06:03.322790 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:06:03.322718    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/383b0139-d5c3-4492-8245-8faac4af0a10-credential-socket" pod "383b0139-d5c3-4492-8245-8faac4af0a10" (UID: "383b0139-d5c3-4492-8245-8faac4af0a10"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:06:03.322915 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:06:03.322885    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/383b0139-d5c3-4492-8245-8faac4af0a10-workload-certs" pod "383b0139-d5c3-4492-8245-8faac4af0a10" (UID: "383b0139-d5c3-4492-8245-8faac4af0a10"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:06:03.322999 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:06:03.322946    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/383b0139-d5c3-4492-8245-8faac4af0a10-istio-data" pod "383b0139-d5c3-4492-8245-8faac4af0a10" (UID: "383b0139-d5c3-4492-8245-8faac4af0a10"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:06:03.323057 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:06:03.323037    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/383b0139-d5c3-4492-8245-8faac4af0a10-workload-socket" pod "383b0139-d5c3-4492-8245-8faac4af0a10" (UID: "383b0139-d5c3-4492-8245-8faac4af0a10"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:06:03.323333 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:06:03.323299    4964 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/383b0139-d5c3-4492-8245-8faac4af0a10-workload-socket\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:06:03.323421 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:06:03.323340    4964 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/383b0139-d5c3-4492-8245-8faac4af0a10-istiod-ca-cert\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:06:03.323421 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:06:03.323355    4964 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/383b0139-d5c3-4492-8245-8faac4af0a10-istio-data\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:06:03.323421 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:06:03.323369    4964 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/383b0139-d5c3-4492-8245-8faac4af0a10-credential-socket\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:06:03.323421 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:06:03.323382    4964 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/383b0139-d5c3-4492-8245-8faac4af0a10-workload-certs\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:06:03.324718 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:06:03.324685    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/383b0139-d5c3-4492-8245-8faac4af0a10-istio-token" pod "383b0139-d5c3-4492-8245-8faac4af0a10" (UID: "383b0139-d5c3-4492-8245-8faac4af0a10"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:06:03.324916 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:06:03.324895    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/383b0139-d5c3-4492-8245-8faac4af0a10-istio-envoy" pod "383b0139-d5c3-4492-8245-8faac4af0a10" (UID: "383b0139-d5c3-4492-8245-8faac4af0a10"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:06:03.325110 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:06:03.325091    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/383b0139-d5c3-4492-8245-8faac4af0a10-istio-podinfo" pod "383b0139-d5c3-4492-8245-8faac4af0a10" (UID: "383b0139-d5c3-4492-8245-8faac4af0a10"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 02:06:03.325220 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:06:03.325206    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/383b0139-d5c3-4492-8245-8faac4af0a10-kube-api-access-wh8sq" pod "383b0139-d5c3-4492-8245-8faac4af0a10" (UID: "383b0139-d5c3-4492-8245-8faac4af0a10"). InnerVolumeSpecName "kube-api-access-wh8sq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:06:03.424289 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:06:03.424256    4964 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/383b0139-d5c3-4492-8245-8faac4af0a10-istio-token\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:06:03.424289 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:06:03.424284    4964 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wh8sq\" (UniqueName: \"kubernetes.io/projected/383b0139-d5c3-4492-8245-8faac4af0a10-kube-api-access-wh8sq\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:06:03.424289 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:06:03.424294    4964 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/383b0139-d5c3-4492-8245-8faac4af0a10-istio-podinfo\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:06:03.424569 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:06:03.424303    4964 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/383b0139-d5c3-4492-8245-8faac4af0a10-istio-envoy\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:06:03.713015 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:06:03.712981    4964 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-5qwk9"]
Oct 31 02:06:03.716134 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:06:03.716106    4964 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-5qwk9"]
Oct 31 02:06:03.777741 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:06:03.777708    4964 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="383b0139-d5c3-4492-8245-8faac4af0a10" path="/var/lib/kubelet/pods/383b0139-d5c3-4492-8245-8faac4af0a10/volumes"
Oct 31 02:06:05.724414 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:06:05.724370    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 02:06:06.779189 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:06:06.779152    4964 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w0-98b70c/auth-test-model-98b70c-kserve-57ddcb7c85-5wdxx"]
Oct 31 02:06:06.819864 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:06:06.819818    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 02:06:10.483877 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:06:10.483817    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 02:06:10.873749 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:06:10.873654    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 02:06:11.238930 ip-10-0-133-197 kubenswrapper[4964]: E1031 02:06:11.238894    4964 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q56x5" containerName="extractor"
Oct 31 02:06:11.682893 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:06:11.682806    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 02:06:15.201908 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:06:15.201863    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 02:06:19.922800 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:06:19.922761    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 02:06:21.800699 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:06:21.800656    4964 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w0-98b70c/auth-test-model-98b70c-kserve-57ddcb7c85-5wdxx" podUID="5b8433e9-df92-402c-9873-83646d9bfee5" containerName="main" containerID="cri-o://c4bdb29cfd0649fdd6b43e6f66b33166f42516fd9a1c5be67cce65aaac84fba2" gracePeriod=45
Oct 31 02:06:22.081137 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:06:22.081113    4964 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-98b70c/auth-test-model-98b70c-kserve-57ddcb7c85-5wdxx"
Oct 31 02:06:22.158068 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:06:22.158035    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5b8433e9-df92-402c-9873-83646d9bfee5-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5b8433e9-df92-402c-9873-83646d9bfee5-model-cache\") pod \"5b8433e9-df92-402c-9873-83646d9bfee5\" (UID: \"5b8433e9-df92-402c-9873-83646d9bfee5\") "
Oct 31 02:06:22.158209 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:06:22.158075    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5b8433e9-df92-402c-9873-83646d9bfee5-dshm\" (UniqueName: \"kubernetes.io/empty-dir/5b8433e9-df92-402c-9873-83646d9bfee5-dshm\") pod \"5b8433e9-df92-402c-9873-83646d9bfee5\" (UID: \"5b8433e9-df92-402c-9873-83646d9bfee5\") "
Oct 31 02:06:22.158209 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:06:22.158124    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5b8433e9-df92-402c-9873-83646d9bfee5-tls-certs\" (UniqueName: \"kubernetes.io/secret/5b8433e9-df92-402c-9873-83646d9bfee5-tls-certs\") pod \"5b8433e9-df92-402c-9873-83646d9bfee5\" (UID: \"5b8433e9-df92-402c-9873-83646d9bfee5\") "
Oct 31 02:06:22.158290 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:06:22.158224    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5b8433e9-df92-402c-9873-83646d9bfee5-kube-api-access-wld8g\" (UniqueName: \"kubernetes.io/projected/5b8433e9-df92-402c-9873-83646d9bfee5-kube-api-access-wld8g\") pod \"5b8433e9-df92-402c-9873-83646d9bfee5\" (UID: \"5b8433e9-df92-402c-9873-83646d9bfee5\") "
Oct 31 02:06:22.158290 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:06:22.158245    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5b8433e9-df92-402c-9873-83646d9bfee5-home\" (UniqueName: \"kubernetes.io/empty-dir/5b8433e9-df92-402c-9873-83646d9bfee5-home\") pod \"5b8433e9-df92-402c-9873-83646d9bfee5\" (UID: \"5b8433e9-df92-402c-9873-83646d9bfee5\") "
Oct 31 02:06:22.158414 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:06:22.158309    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5b8433e9-df92-402c-9873-83646d9bfee5-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5b8433e9-df92-402c-9873-83646d9bfee5-tmp-dir\") pod \"5b8433e9-df92-402c-9873-83646d9bfee5\" (UID: \"5b8433e9-df92-402c-9873-83646d9bfee5\") "
Oct 31 02:06:22.158414 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:06:22.158359    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b8433e9-df92-402c-9873-83646d9bfee5-model-cache" pod "5b8433e9-df92-402c-9873-83646d9bfee5" (UID: "5b8433e9-df92-402c-9873-83646d9bfee5"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:06:22.158578 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:06:22.158557    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b8433e9-df92-402c-9873-83646d9bfee5-home" pod "5b8433e9-df92-402c-9873-83646d9bfee5" (UID: "5b8433e9-df92-402c-9873-83646d9bfee5"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:06:22.158675 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:06:22.158648    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b8433e9-df92-402c-9873-83646d9bfee5-tmp-dir" pod "5b8433e9-df92-402c-9873-83646d9bfee5" (UID: "5b8433e9-df92-402c-9873-83646d9bfee5"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:06:22.158827 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:06:22.158804    4964 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5b8433e9-df92-402c-9873-83646d9bfee5-tmp-dir\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:06:22.158827 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:06:22.158826    4964 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5b8433e9-df92-402c-9873-83646d9bfee5-model-cache\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:06:22.158974 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:06:22.158841    4964 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5b8433e9-df92-402c-9873-83646d9bfee5-home\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:06:22.160666 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:06:22.160639    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5b8433e9-df92-402c-9873-83646d9bfee5-tls-certs" pod "5b8433e9-df92-402c-9873-83646d9bfee5" (UID: "5b8433e9-df92-402c-9873-83646d9bfee5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 02:06:22.160666 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:06:22.160658    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b8433e9-df92-402c-9873-83646d9bfee5-kube-api-access-wld8g" pod "5b8433e9-df92-402c-9873-83646d9bfee5" (UID: "5b8433e9-df92-402c-9873-83646d9bfee5"). InnerVolumeSpecName "kube-api-access-wld8g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:06:22.160792 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:06:22.160665    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b8433e9-df92-402c-9873-83646d9bfee5-dshm" pod "5b8433e9-df92-402c-9873-83646d9bfee5" (UID: "5b8433e9-df92-402c-9873-83646d9bfee5"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:06:22.259732 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:06:22.259699    4964 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5b8433e9-df92-402c-9873-83646d9bfee5-dshm\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:06:22.259732 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:06:22.259726    4964 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5b8433e9-df92-402c-9873-83646d9bfee5-tls-certs\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:06:22.259732 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:06:22.259735    4964 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wld8g\" (UniqueName: \"kubernetes.io/projected/5b8433e9-df92-402c-9873-83646d9bfee5-kube-api-access-wld8g\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:06:22.422495 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:06:22.422402    4964 generic.go:356] "Generic (PLEG): container finished" podID="5b8433e9-df92-402c-9873-83646d9bfee5" containerID="c4bdb29cfd0649fdd6b43e6f66b33166f42516fd9a1c5be67cce65aaac84fba2" exitCode=0
Oct 31 02:06:22.422495 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:06:22.422454    4964 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-98b70c/auth-test-model-98b70c-kserve-57ddcb7c85-5wdxx"
Oct 31 02:06:22.422495 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:06:22.422468    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-98b70c/auth-test-model-98b70c-kserve-57ddcb7c85-5wdxx" event={"ID":"5b8433e9-df92-402c-9873-83646d9bfee5","Type":"ContainerDied","Data":"c4bdb29cfd0649fdd6b43e6f66b33166f42516fd9a1c5be67cce65aaac84fba2"}
Oct 31 02:06:22.422775 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:06:22.422508    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-98b70c/auth-test-model-98b70c-kserve-57ddcb7c85-5wdxx" event={"ID":"5b8433e9-df92-402c-9873-83646d9bfee5","Type":"ContainerDied","Data":"29e76f1963dcc8c136738558208bddf61d11306563db50e9f0fedf4223536d8d"}
Oct 31 02:06:22.422775 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:06:22.422530    4964 scope.go:122] "RemoveContainer" containerID="c4bdb29cfd0649fdd6b43e6f66b33166f42516fd9a1c5be67cce65aaac84fba2"
Oct 31 02:06:22.434847 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:06:22.434827    4964 scope.go:122] "RemoveContainer" containerID="c4bdb29cfd0649fdd6b43e6f66b33166f42516fd9a1c5be67cce65aaac84fba2"
Oct 31 02:06:22.435146 ip-10-0-133-197 kubenswrapper[4964]: E1031 02:06:22.435127    4964 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c4bdb29cfd0649fdd6b43e6f66b33166f42516fd9a1c5be67cce65aaac84fba2\": container with ID starting with c4bdb29cfd0649fdd6b43e6f66b33166f42516fd9a1c5be67cce65aaac84fba2 not found: ID does not exist" containerID="c4bdb29cfd0649fdd6b43e6f66b33166f42516fd9a1c5be67cce65aaac84fba2"
Oct 31 02:06:22.435201 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:06:22.435154    4964 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c4bdb29cfd0649fdd6b43e6f66b33166f42516fd9a1c5be67cce65aaac84fba2"} err="failed to get container status \"c4bdb29cfd0649fdd6b43e6f66b33166f42516fd9a1c5be67cce65aaac84fba2\": rpc error: code = NotFound desc = could not find container \"c4bdb29cfd0649fdd6b43e6f66b33166f42516fd9a1c5be67cce65aaac84fba2\": container with ID starting with c4bdb29cfd0649fdd6b43e6f66b33166f42516fd9a1c5be67cce65aaac84fba2 not found: ID does not exist"
Oct 31 02:06:22.586082 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:06:22.586043    4964 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w0-98b70c/auth-test-model-98b70c-kserve-57ddcb7c85-5wdxx"]
Oct 31 02:06:22.592404 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:06:22.592380    4964 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w0-98b70c/auth-test-model-98b70c-kserve-57ddcb7c85-5wdxx"]
Oct 31 02:06:23.777859 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:06:23.777825    4964 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5b8433e9-df92-402c-9873-83646d9bfee5" path="/var/lib/kubelet/pods/5b8433e9-df92-402c-9873-83646d9bfee5/volumes"
Oct 31 02:06:24.672525 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:06:24.672482    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 02:06:29.409656 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:06:29.409615    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 02:06:31.102358 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:06:31.102312    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 02:06:31.288702 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:06:31.288666    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 02:06:31.608366 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:06:31.608330    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 02:06:31.787211 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:06:31.787177    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 02:06:34.157211 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:06:34.157171    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 02:06:36.782242 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:06:36.782208    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 02:06:37.747175 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:06:37.747134    4964 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w0-98c748/auth-test-model-98c748-kserve-594b4df7b9-jmbxz"]
Oct 31 02:06:37.793680 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:06:37.793647    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 02:06:39.104253 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:06:39.104210    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 02:06:41.237511 ip-10-0-133-197 kubenswrapper[4964]: E1031 02:06:41.237476    4964 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q56x5" containerName="extractor"
Oct 31 02:06:44.204028 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:06:44.203986    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 02:06:47.882355 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:06:47.882298    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 02:06:49.004576 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:06:49.004537    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 02:06:49.264121 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:06:49.264042    4964 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-98b70c-779f8fcd7f-6lb9k"]
Oct 31 02:06:49.663747 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:06:49.663652    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 02:06:49.865254 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:06:49.865214    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 02:06:52.768530 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:06:52.768474    4964 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w0-98c748/auth-test-model-98c748-kserve-594b4df7b9-jmbxz" podUID="9700e3ca-d2e5-4171-a3b0-3506d01d7a00" containerName="main" containerID="cri-o://79c0d02bc96386635fbf924e534d4a78cb4af9460b24d9594def177c38dbe2c1" gracePeriod=45
Oct 31 02:06:53.058874 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:06:53.058848    4964 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-98c748/auth-test-model-98c748-kserve-594b4df7b9-jmbxz"
Oct 31 02:06:53.146564 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:06:53.146534    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9700e3ca-d2e5-4171-a3b0-3506d01d7a00-tls-certs\" (UniqueName: \"kubernetes.io/secret/9700e3ca-d2e5-4171-a3b0-3506d01d7a00-tls-certs\") pod \"9700e3ca-d2e5-4171-a3b0-3506d01d7a00\" (UID: \"9700e3ca-d2e5-4171-a3b0-3506d01d7a00\") "
Oct 31 02:06:53.146721 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:06:53.146604    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9700e3ca-d2e5-4171-a3b0-3506d01d7a00-kube-api-access-v85gq\" (UniqueName: \"kubernetes.io/projected/9700e3ca-d2e5-4171-a3b0-3506d01d7a00-kube-api-access-v85gq\") pod \"9700e3ca-d2e5-4171-a3b0-3506d01d7a00\" (UID: \"9700e3ca-d2e5-4171-a3b0-3506d01d7a00\") "
Oct 31 02:06:53.146763 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:06:53.146741    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9700e3ca-d2e5-4171-a3b0-3506d01d7a00-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9700e3ca-d2e5-4171-a3b0-3506d01d7a00-model-cache\") pod \"9700e3ca-d2e5-4171-a3b0-3506d01d7a00\" (UID: \"9700e3ca-d2e5-4171-a3b0-3506d01d7a00\") "
Oct 31 02:06:53.146800 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:06:53.146791    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9700e3ca-d2e5-4171-a3b0-3506d01d7a00-dshm\" (UniqueName: \"kubernetes.io/empty-dir/9700e3ca-d2e5-4171-a3b0-3506d01d7a00-dshm\") pod \"9700e3ca-d2e5-4171-a3b0-3506d01d7a00\" (UID: \"9700e3ca-d2e5-4171-a3b0-3506d01d7a00\") "
Oct 31 02:06:53.146925 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:06:53.146895    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9700e3ca-d2e5-4171-a3b0-3506d01d7a00-home\" (UniqueName: \"kubernetes.io/empty-dir/9700e3ca-d2e5-4171-a3b0-3506d01d7a00-home\") pod \"9700e3ca-d2e5-4171-a3b0-3506d01d7a00\" (UID: \"9700e3ca-d2e5-4171-a3b0-3506d01d7a00\") "
Oct 31 02:06:53.147009 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:06:53.146977    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9700e3ca-d2e5-4171-a3b0-3506d01d7a00-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9700e3ca-d2e5-4171-a3b0-3506d01d7a00-tmp-dir\") pod \"9700e3ca-d2e5-4171-a3b0-3506d01d7a00\" (UID: \"9700e3ca-d2e5-4171-a3b0-3506d01d7a00\") "
Oct 31 02:06:53.147072 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:06:53.147000    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9700e3ca-d2e5-4171-a3b0-3506d01d7a00-model-cache" pod "9700e3ca-d2e5-4171-a3b0-3506d01d7a00" (UID: "9700e3ca-d2e5-4171-a3b0-3506d01d7a00"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:06:53.147207 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:06:53.147170    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9700e3ca-d2e5-4171-a3b0-3506d01d7a00-home" pod "9700e3ca-d2e5-4171-a3b0-3506d01d7a00" (UID: "9700e3ca-d2e5-4171-a3b0-3506d01d7a00"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:06:53.147408 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:06:53.147386    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9700e3ca-d2e5-4171-a3b0-3506d01d7a00-tmp-dir" pod "9700e3ca-d2e5-4171-a3b0-3506d01d7a00" (UID: "9700e3ca-d2e5-4171-a3b0-3506d01d7a00"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:06:53.147551 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:06:53.147533    4964 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9700e3ca-d2e5-4171-a3b0-3506d01d7a00-model-cache\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:06:53.147604 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:06:53.147557    4964 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9700e3ca-d2e5-4171-a3b0-3506d01d7a00-home\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:06:53.147604 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:06:53.147569    4964 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9700e3ca-d2e5-4171-a3b0-3506d01d7a00-tmp-dir\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:06:53.149180 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:06:53.149158    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9700e3ca-d2e5-4171-a3b0-3506d01d7a00-dshm" pod "9700e3ca-d2e5-4171-a3b0-3506d01d7a00" (UID: "9700e3ca-d2e5-4171-a3b0-3506d01d7a00"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:06:53.149538 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:06:53.149515    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9700e3ca-d2e5-4171-a3b0-3506d01d7a00-tls-certs" pod "9700e3ca-d2e5-4171-a3b0-3506d01d7a00" (UID: "9700e3ca-d2e5-4171-a3b0-3506d01d7a00"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 02:06:53.149538 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:06:53.149526    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9700e3ca-d2e5-4171-a3b0-3506d01d7a00-kube-api-access-v85gq" pod "9700e3ca-d2e5-4171-a3b0-3506d01d7a00" (UID: "9700e3ca-d2e5-4171-a3b0-3506d01d7a00"). InnerVolumeSpecName "kube-api-access-v85gq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:06:53.248606 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:06:53.248576    4964 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9700e3ca-d2e5-4171-a3b0-3506d01d7a00-tls-certs\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:06:53.248606 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:06:53.248607    4964 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v85gq\" (UniqueName: \"kubernetes.io/projected/9700e3ca-d2e5-4171-a3b0-3506d01d7a00-kube-api-access-v85gq\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:06:53.248757 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:06:53.248618    4964 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9700e3ca-d2e5-4171-a3b0-3506d01d7a00-dshm\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:06:53.654591 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:06:53.654560    4964 generic.go:356] "Generic (PLEG): container finished" podID="9700e3ca-d2e5-4171-a3b0-3506d01d7a00" containerID="79c0d02bc96386635fbf924e534d4a78cb4af9460b24d9594def177c38dbe2c1" exitCode=0
Oct 31 02:06:53.654780 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:06:53.654611    4964 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-98c748/auth-test-model-98c748-kserve-594b4df7b9-jmbxz"
Oct 31 02:06:53.654780 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:06:53.654620    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-98c748/auth-test-model-98c748-kserve-594b4df7b9-jmbxz" event={"ID":"9700e3ca-d2e5-4171-a3b0-3506d01d7a00","Type":"ContainerDied","Data":"79c0d02bc96386635fbf924e534d4a78cb4af9460b24d9594def177c38dbe2c1"}
Oct 31 02:06:53.654780 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:06:53.654655    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-98c748/auth-test-model-98c748-kserve-594b4df7b9-jmbxz" event={"ID":"9700e3ca-d2e5-4171-a3b0-3506d01d7a00","Type":"ContainerDied","Data":"18c779fcc58d4914fa6b9af75fa61d206bbafc1920a4830b9778bcc663b3d90b"}
Oct 31 02:06:53.654780 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:06:53.654675    4964 scope.go:122] "RemoveContainer" containerID="79c0d02bc96386635fbf924e534d4a78cb4af9460b24d9594def177c38dbe2c1"
Oct 31 02:06:53.665112 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:06:53.665094    4964 scope.go:122] "RemoveContainer" containerID="79c0d02bc96386635fbf924e534d4a78cb4af9460b24d9594def177c38dbe2c1"
Oct 31 02:06:53.665418 ip-10-0-133-197 kubenswrapper[4964]: E1031 02:06:53.665398    4964 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"79c0d02bc96386635fbf924e534d4a78cb4af9460b24d9594def177c38dbe2c1\": container with ID starting with 79c0d02bc96386635fbf924e534d4a78cb4af9460b24d9594def177c38dbe2c1 not found: ID does not exist" containerID="79c0d02bc96386635fbf924e534d4a78cb4af9460b24d9594def177c38dbe2c1"
Oct 31 02:06:53.665494 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:06:53.665426    4964 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"79c0d02bc96386635fbf924e534d4a78cb4af9460b24d9594def177c38dbe2c1"} err="failed to get container status \"79c0d02bc96386635fbf924e534d4a78cb4af9460b24d9594def177c38dbe2c1\": rpc error: code = NotFound desc = could not find container \"79c0d02bc96386635fbf924e534d4a78cb4af9460b24d9594def177c38dbe2c1\": container with ID starting with 79c0d02bc96386635fbf924e534d4a78cb4af9460b24d9594def177c38dbe2c1 not found: ID does not exist"
Oct 31 02:06:53.790943 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:06:53.790908    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 02:06:53.818948 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:06:53.818915    4964 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w0-98c748/auth-test-model-98c748-kserve-594b4df7b9-jmbxz"]
Oct 31 02:06:53.821927 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:06:53.821904    4964 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w0-98c748/auth-test-model-98c748-kserve-594b4df7b9-jmbxz"]
Oct 31 02:06:54.176451 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:06:54.176417    4964 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-051514/auth-test-model-051514-kserve-777964c64f-cz8wj"]
Oct 31 02:06:54.254558 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:06:54.254527    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 02:06:54.264444 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:06:54.264409    4964 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-98b70c-779f8fcd7f-6lb9k" podUID="380f14cc-28b6-4557-8dd5-7a36f0bbb3be" containerName="payload-processing" containerID="cri-o://8bd9b2f828f29767d01f336da42721f416e078887cf4e62980fe1d5522e7e7fe" gracePeriod=25
Oct 31 02:06:54.555029 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:06:54.555004    4964 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-98b70c-779f8fcd7f-6lb9k"
Oct 31 02:06:54.664010 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:06:54.663978    4964 generic.go:356] "Generic (PLEG): container finished" podID="380f14cc-28b6-4557-8dd5-7a36f0bbb3be" containerID="8bd9b2f828f29767d01f336da42721f416e078887cf4e62980fe1d5522e7e7fe" exitCode=0
Oct 31 02:06:54.664190 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:06:54.664027    4964 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-98b70c-779f8fcd7f-6lb9k"
Oct 31 02:06:54.664190 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:06:54.664054    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-98b70c-779f8fcd7f-6lb9k" event={"ID":"380f14cc-28b6-4557-8dd5-7a36f0bbb3be","Type":"ContainerDied","Data":"8bd9b2f828f29767d01f336da42721f416e078887cf4e62980fe1d5522e7e7fe"}
Oct 31 02:06:54.664190 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:06:54.664111    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-98b70c-779f8fcd7f-6lb9k" event={"ID":"380f14cc-28b6-4557-8dd5-7a36f0bbb3be","Type":"ContainerDied","Data":"6d7e314d1f3fe59bd0eb36216938b5b7e480caa82d32d733c33bad6b74c08e94"}
Oct 31 02:06:54.664190 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:06:54.664135    4964 scope.go:122] "RemoveContainer" containerID="8bd9b2f828f29767d01f336da42721f416e078887cf4e62980fe1d5522e7e7fe"
Oct 31 02:06:54.664890 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:06:54.664871    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/380f14cc-28b6-4557-8dd5-7a36f0bbb3be-kube-api-access-nb2zr\" (UniqueName: \"kubernetes.io/projected/380f14cc-28b6-4557-8dd5-7a36f0bbb3be-kube-api-access-nb2zr\") pod \"380f14cc-28b6-4557-8dd5-7a36f0bbb3be\" (UID: \"380f14cc-28b6-4557-8dd5-7a36f0bbb3be\") "
Oct 31 02:06:54.665144 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:06:54.665129    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/380f14cc-28b6-4557-8dd5-7a36f0bbb3be-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/380f14cc-28b6-4557-8dd5-7a36f0bbb3be-plugins-config-volume\") pod \"380f14cc-28b6-4557-8dd5-7a36f0bbb3be\" (UID: \"380f14cc-28b6-4557-8dd5-7a36f0bbb3be\") "
Oct 31 02:06:54.665625 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:06:54.665592    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/380f14cc-28b6-4557-8dd5-7a36f0bbb3be-plugins-config-volume" pod "380f14cc-28b6-4557-8dd5-7a36f0bbb3be" (UID: "380f14cc-28b6-4557-8dd5-7a36f0bbb3be"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 02:06:54.667494 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:06:54.667466    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/380f14cc-28b6-4557-8dd5-7a36f0bbb3be-kube-api-access-nb2zr" pod "380f14cc-28b6-4557-8dd5-7a36f0bbb3be" (UID: "380f14cc-28b6-4557-8dd5-7a36f0bbb3be"). InnerVolumeSpecName "kube-api-access-nb2zr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:06:54.694663 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:06:54.694638    4964 scope.go:122] "RemoveContainer" containerID="8bd9b2f828f29767d01f336da42721f416e078887cf4e62980fe1d5522e7e7fe"
Oct 31 02:06:54.695116 ip-10-0-133-197 kubenswrapper[4964]: E1031 02:06:54.695088    4964 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8bd9b2f828f29767d01f336da42721f416e078887cf4e62980fe1d5522e7e7fe\": container with ID starting with 8bd9b2f828f29767d01f336da42721f416e078887cf4e62980fe1d5522e7e7fe not found: ID does not exist" containerID="8bd9b2f828f29767d01f336da42721f416e078887cf4e62980fe1d5522e7e7fe"
Oct 31 02:06:54.695238 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:06:54.695123    4964 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8bd9b2f828f29767d01f336da42721f416e078887cf4e62980fe1d5522e7e7fe"} err="failed to get container status \"8bd9b2f828f29767d01f336da42721f416e078887cf4e62980fe1d5522e7e7fe\": rpc error: code = NotFound desc = could not find container \"8bd9b2f828f29767d01f336da42721f416e078887cf4e62980fe1d5522e7e7fe\": container with ID starting with 8bd9b2f828f29767d01f336da42721f416e078887cf4e62980fe1d5522e7e7fe not found: ID does not exist"
Oct 31 02:06:54.770774 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:06:54.768136    4964 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/380f14cc-28b6-4557-8dd5-7a36f0bbb3be-plugins-config-volume\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:06:54.770774 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:06:54.768258    4964 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nb2zr\" (UniqueName: \"kubernetes.io/projected/380f14cc-28b6-4557-8dd5-7a36f0bbb3be-kube-api-access-nb2zr\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:06:55.084198 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:06:55.084163    4964 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-98b70c-779f8fcd7f-6lb9k"]
Oct 31 02:06:55.088551 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:06:55.088507    4964 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-98b70c-779f8fcd7f-6lb9k"]
Oct 31 02:06:55.777386 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:06:55.777355    4964 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="380f14cc-28b6-4557-8dd5-7a36f0bbb3be" path="/var/lib/kubelet/pods/380f14cc-28b6-4557-8dd5-7a36f0bbb3be/volumes"
Oct 31 02:06:55.777822 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:06:55.777807    4964 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9700e3ca-d2e5-4171-a3b0-3506d01d7a00" path="/var/lib/kubelet/pods/9700e3ca-d2e5-4171-a3b0-3506d01d7a00/volumes"
Oct 31 02:06:58.589370 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:06:58.589299    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 02:06:58.969212 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:06:58.969188    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 02:07:02.956498 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:07:02.956460    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 02:07:03.327367 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:07:03.327310    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 02:07:03.491095 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:07:03.491058    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 02:07:08.148866 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:07:08.148826    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 02:07:09.200011 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:07:09.199949    4964 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-051514/auth-test-model-051514-kserve-777964c64f-cz8wj" podUID="3d4d5c24-0d5f-4183-acab-5e2ae32050f5" containerName="main" containerID="cri-o://cb78ac7fb5c6a92cbf617e167bc0e1c613165e5c7ed0f6a5353c274dc9e43355" gracePeriod=45
Oct 31 02:07:09.486604 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:07:09.486580    4964 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-051514/auth-test-model-051514-kserve-777964c64f-cz8wj"
Oct 31 02:07:09.639271 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:07:09.639242    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3d4d5c24-0d5f-4183-acab-5e2ae32050f5-home\" (UniqueName: \"kubernetes.io/empty-dir/3d4d5c24-0d5f-4183-acab-5e2ae32050f5-home\") pod \"3d4d5c24-0d5f-4183-acab-5e2ae32050f5\" (UID: \"3d4d5c24-0d5f-4183-acab-5e2ae32050f5\") "
Oct 31 02:07:09.639494 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:07:09.639360    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3d4d5c24-0d5f-4183-acab-5e2ae32050f5-dshm\" (UniqueName: \"kubernetes.io/empty-dir/3d4d5c24-0d5f-4183-acab-5e2ae32050f5-dshm\") pod \"3d4d5c24-0d5f-4183-acab-5e2ae32050f5\" (UID: \"3d4d5c24-0d5f-4183-acab-5e2ae32050f5\") "
Oct 31 02:07:09.639567 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:07:09.639524    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3d4d5c24-0d5f-4183-acab-5e2ae32050f5-kube-api-access-gv4dn\" (UniqueName: \"kubernetes.io/projected/3d4d5c24-0d5f-4183-acab-5e2ae32050f5-kube-api-access-gv4dn\") pod \"3d4d5c24-0d5f-4183-acab-5e2ae32050f5\" (UID: \"3d4d5c24-0d5f-4183-acab-5e2ae32050f5\") "
Oct 31 02:07:09.639636 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:07:09.639584    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3d4d5c24-0d5f-4183-acab-5e2ae32050f5-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3d4d5c24-0d5f-4183-acab-5e2ae32050f5-tmp-dir\") pod \"3d4d5c24-0d5f-4183-acab-5e2ae32050f5\" (UID: \"3d4d5c24-0d5f-4183-acab-5e2ae32050f5\") "
Oct 31 02:07:09.639691 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:07:09.639656    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3d4d5c24-0d5f-4183-acab-5e2ae32050f5-tls-certs\" (UniqueName: \"kubernetes.io/secret/3d4d5c24-0d5f-4183-acab-5e2ae32050f5-tls-certs\") pod \"3d4d5c24-0d5f-4183-acab-5e2ae32050f5\" (UID: \"3d4d5c24-0d5f-4183-acab-5e2ae32050f5\") "
Oct 31 02:07:09.639744 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:07:09.639699    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3d4d5c24-0d5f-4183-acab-5e2ae32050f5-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3d4d5c24-0d5f-4183-acab-5e2ae32050f5-model-cache\") pod \"3d4d5c24-0d5f-4183-acab-5e2ae32050f5\" (UID: \"3d4d5c24-0d5f-4183-acab-5e2ae32050f5\") "
Oct 31 02:07:09.639990 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:07:09.639963    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d4d5c24-0d5f-4183-acab-5e2ae32050f5-tmp-dir" pod "3d4d5c24-0d5f-4183-acab-5e2ae32050f5" (UID: "3d4d5c24-0d5f-4183-acab-5e2ae32050f5"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:07:09.640146 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:07:09.640128    4964 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3d4d5c24-0d5f-4183-acab-5e2ae32050f5-tmp-dir\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:07:09.640615 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:07:09.640592    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d4d5c24-0d5f-4183-acab-5e2ae32050f5-home" pod "3d4d5c24-0d5f-4183-acab-5e2ae32050f5" (UID: "3d4d5c24-0d5f-4183-acab-5e2ae32050f5"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:07:09.640777 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:07:09.640634    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d4d5c24-0d5f-4183-acab-5e2ae32050f5-model-cache" pod "3d4d5c24-0d5f-4183-acab-5e2ae32050f5" (UID: "3d4d5c24-0d5f-4183-acab-5e2ae32050f5"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:07:09.642458 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:07:09.642431    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d4d5c24-0d5f-4183-acab-5e2ae32050f5-dshm" pod "3d4d5c24-0d5f-4183-acab-5e2ae32050f5" (UID: "3d4d5c24-0d5f-4183-acab-5e2ae32050f5"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:07:09.642780 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:07:09.642762    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3d4d5c24-0d5f-4183-acab-5e2ae32050f5-tls-certs" pod "3d4d5c24-0d5f-4183-acab-5e2ae32050f5" (UID: "3d4d5c24-0d5f-4183-acab-5e2ae32050f5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 02:07:09.642870 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:07:09.642815    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d4d5c24-0d5f-4183-acab-5e2ae32050f5-kube-api-access-gv4dn" pod "3d4d5c24-0d5f-4183-acab-5e2ae32050f5" (UID: "3d4d5c24-0d5f-4183-acab-5e2ae32050f5"). InnerVolumeSpecName "kube-api-access-gv4dn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:07:09.741414 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:07:09.741385    4964 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gv4dn\" (UniqueName: \"kubernetes.io/projected/3d4d5c24-0d5f-4183-acab-5e2ae32050f5-kube-api-access-gv4dn\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:07:09.741414 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:07:09.741411    4964 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3d4d5c24-0d5f-4183-acab-5e2ae32050f5-tls-certs\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:07:09.741618 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:07:09.741424    4964 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3d4d5c24-0d5f-4183-acab-5e2ae32050f5-model-cache\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:07:09.741618 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:07:09.741432    4964 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3d4d5c24-0d5f-4183-acab-5e2ae32050f5-home\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:07:09.741618 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:07:09.741441    4964 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3d4d5c24-0d5f-4183-acab-5e2ae32050f5-dshm\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:07:09.785091 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:07:09.785060    4964 generic.go:356] "Generic (PLEG): container finished" podID="3d4d5c24-0d5f-4183-acab-5e2ae32050f5" containerID="cb78ac7fb5c6a92cbf617e167bc0e1c613165e5c7ed0f6a5353c274dc9e43355" exitCode=0
Oct 31 02:07:09.785282 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:07:09.785202    4964 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-051514/auth-test-model-051514-kserve-777964c64f-cz8wj"
Oct 31 02:07:09.786699 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:07:09.786665    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-051514/auth-test-model-051514-kserve-777964c64f-cz8wj" event={"ID":"3d4d5c24-0d5f-4183-acab-5e2ae32050f5","Type":"ContainerDied","Data":"cb78ac7fb5c6a92cbf617e167bc0e1c613165e5c7ed0f6a5353c274dc9e43355"}
Oct 31 02:07:09.786699 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:07:09.786698    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-051514/auth-test-model-051514-kserve-777964c64f-cz8wj" event={"ID":"3d4d5c24-0d5f-4183-acab-5e2ae32050f5","Type":"ContainerDied","Data":"e8207e6079e7d3c104afb0f05eab88e4b692781053f2ee1d0341c87020fbde33"}
Oct 31 02:07:09.786896 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:07:09.786720    4964 scope.go:122] "RemoveContainer" containerID="cb78ac7fb5c6a92cbf617e167bc0e1c613165e5c7ed0f6a5353c274dc9e43355"
Oct 31 02:07:09.799577 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:07:09.799558    4964 scope.go:122] "RemoveContainer" containerID="cb78ac7fb5c6a92cbf617e167bc0e1c613165e5c7ed0f6a5353c274dc9e43355"
Oct 31 02:07:09.799901 ip-10-0-133-197 kubenswrapper[4964]: E1031 02:07:09.799878    4964 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cb78ac7fb5c6a92cbf617e167bc0e1c613165e5c7ed0f6a5353c274dc9e43355\": container with ID starting with cb78ac7fb5c6a92cbf617e167bc0e1c613165e5c7ed0f6a5353c274dc9e43355 not found: ID does not exist" containerID="cb78ac7fb5c6a92cbf617e167bc0e1c613165e5c7ed0f6a5353c274dc9e43355"
Oct 31 02:07:09.800020 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:07:09.799905    4964 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cb78ac7fb5c6a92cbf617e167bc0e1c613165e5c7ed0f6a5353c274dc9e43355"} err="failed to get container status \"cb78ac7fb5c6a92cbf617e167bc0e1c613165e5c7ed0f6a5353c274dc9e43355\": rpc error: code = NotFound desc = could not find container \"cb78ac7fb5c6a92cbf617e167bc0e1c613165e5c7ed0f6a5353c274dc9e43355\": container with ID starting with cb78ac7fb5c6a92cbf617e167bc0e1c613165e5c7ed0f6a5353c274dc9e43355 not found: ID does not exist"
Oct 31 02:07:09.952071 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:07:09.952037    4964 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-051514/auth-test-model-051514-kserve-777964c64f-cz8wj"]
Oct 31 02:07:09.955598 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:07:09.955576    4964 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-051514/auth-test-model-051514-kserve-777964c64f-cz8wj"]
Oct 31 02:07:11.240177 ip-10-0-133-197 kubenswrapper[4964]: E1031 02:07:11.240133    4964 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q56x5" containerName="extractor"
Oct 31 02:07:11.251379 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:07:11.251349    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 02:07:11.777739 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:07:11.777706    4964 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3d4d5c24-0d5f-4183-acab-5e2ae32050f5" path="/var/lib/kubelet/pods/3d4d5c24-0d5f-4183-acab-5e2ae32050f5/volumes"
Oct 31 02:07:12.163579 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:07:12.163494    4964 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-98b70c-55c5cd675b-ltbhs"]
Oct 31 02:07:12.163779 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:07:12.163750    4964 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-98b70c-55c5cd675b-ltbhs" podUID="5ce8150f-4c48-4d65-8416-442f611c53fc" containerName="maas-api" containerID="cri-o://cad610a9b748cbf4df64bcd9825a3333c00ab79bfdc3e210e5fc40ab22ac17a3" gracePeriod=30
Oct 31 02:07:12.299076 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:07:12.298980    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 02:07:12.448766 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:07:12.448743    4964 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-98b70c-55c5cd675b-ltbhs"
Oct 31 02:07:12.471168 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:07:12.471138    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5ce8150f-4c48-4d65-8416-442f611c53fc-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5ce8150f-4c48-4d65-8416-442f611c53fc-openshift-service-ca\") pod \"5ce8150f-4c48-4d65-8416-442f611c53fc\" (UID: \"5ce8150f-4c48-4d65-8416-442f611c53fc\") "
Oct 31 02:07:12.471345 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:07:12.471195    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5ce8150f-4c48-4d65-8416-442f611c53fc-metrics-certs\" (UniqueName: \"kubernetes.io/secret/5ce8150f-4c48-4d65-8416-442f611c53fc-metrics-certs\") pod \"5ce8150f-4c48-4d65-8416-442f611c53fc\" (UID: \"5ce8150f-4c48-4d65-8416-442f611c53fc\") "
Oct 31 02:07:12.471345 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:07:12.471263    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5ce8150f-4c48-4d65-8416-442f611c53fc-kube-api-access-tdmkx\" (UniqueName: \"kubernetes.io/projected/5ce8150f-4c48-4d65-8416-442f611c53fc-kube-api-access-tdmkx\") pod \"5ce8150f-4c48-4d65-8416-442f611c53fc\" (UID: \"5ce8150f-4c48-4d65-8416-442f611c53fc\") "
Oct 31 02:07:12.471473 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:07:12.471425    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5ce8150f-4c48-4d65-8416-442f611c53fc-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5ce8150f-4c48-4d65-8416-442f611c53fc-maas-api-tls\") pod \"5ce8150f-4c48-4d65-8416-442f611c53fc\" (UID: \"5ce8150f-4c48-4d65-8416-442f611c53fc\") "
Oct 31 02:07:12.471537 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:07:12.471503    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5ce8150f-4c48-4d65-8416-442f611c53fc-openshift-service-ca" pod "5ce8150f-4c48-4d65-8416-442f611c53fc" (UID: "5ce8150f-4c48-4d65-8416-442f611c53fc"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 02:07:12.471802 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:07:12.471785    4964 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5ce8150f-4c48-4d65-8416-442f611c53fc-openshift-service-ca\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:07:12.473488 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:07:12.473461    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ce8150f-4c48-4d65-8416-442f611c53fc-kube-api-access-tdmkx" pod "5ce8150f-4c48-4d65-8416-442f611c53fc" (UID: "5ce8150f-4c48-4d65-8416-442f611c53fc"). InnerVolumeSpecName "kube-api-access-tdmkx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:07:12.473628 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:07:12.473608    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5ce8150f-4c48-4d65-8416-442f611c53fc-maas-api-tls" pod "5ce8150f-4c48-4d65-8416-442f611c53fc" (UID: "5ce8150f-4c48-4d65-8416-442f611c53fc"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 02:07:12.473726 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:07:12.473705    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5ce8150f-4c48-4d65-8416-442f611c53fc-metrics-certs" pod "5ce8150f-4c48-4d65-8416-442f611c53fc" (UID: "5ce8150f-4c48-4d65-8416-442f611c53fc"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 02:07:12.572769 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:07:12.572726    4964 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tdmkx\" (UniqueName: \"kubernetes.io/projected/5ce8150f-4c48-4d65-8416-442f611c53fc-kube-api-access-tdmkx\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:07:12.572769 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:07:12.572764    4964 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5ce8150f-4c48-4d65-8416-442f611c53fc-maas-api-tls\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:07:12.573030 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:07:12.572786    4964 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5ce8150f-4c48-4d65-8416-442f611c53fc-metrics-certs\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:07:12.592593 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:07:12.592559    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 02:07:12.815233 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:07:12.815193    4964 generic.go:356] "Generic (PLEG): container finished" podID="5ce8150f-4c48-4d65-8416-442f611c53fc" containerID="cad610a9b748cbf4df64bcd9825a3333c00ab79bfdc3e210e5fc40ab22ac17a3" exitCode=0
Oct 31 02:07:12.815459 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:07:12.815263    4964 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-98b70c-55c5cd675b-ltbhs"
Oct 31 02:07:12.815459 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:07:12.815290    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-98b70c-55c5cd675b-ltbhs" event={"ID":"5ce8150f-4c48-4d65-8416-442f611c53fc","Type":"ContainerDied","Data":"cad610a9b748cbf4df64bcd9825a3333c00ab79bfdc3e210e5fc40ab22ac17a3"}
Oct 31 02:07:12.815459 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:07:12.815402    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-98b70c-55c5cd675b-ltbhs" event={"ID":"5ce8150f-4c48-4d65-8416-442f611c53fc","Type":"ContainerDied","Data":"23ea9be5e35113119900969b02547ec097eaadc4ec9380df9a4641bfd9a938d2"}
Oct 31 02:07:12.815459 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:07:12.815438    4964 scope.go:122] "RemoveContainer" containerID="cad610a9b748cbf4df64bcd9825a3333c00ab79bfdc3e210e5fc40ab22ac17a3"
Oct 31 02:07:12.828732 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:07:12.828495    4964 scope.go:122] "RemoveContainer" containerID="cad610a9b748cbf4df64bcd9825a3333c00ab79bfdc3e210e5fc40ab22ac17a3"
Oct 31 02:07:12.828820 ip-10-0-133-197 kubenswrapper[4964]: E1031 02:07:12.828781    4964 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cad610a9b748cbf4df64bcd9825a3333c00ab79bfdc3e210e5fc40ab22ac17a3\": container with ID starting with cad610a9b748cbf4df64bcd9825a3333c00ab79bfdc3e210e5fc40ab22ac17a3 not found: ID does not exist" containerID="cad610a9b748cbf4df64bcd9825a3333c00ab79bfdc3e210e5fc40ab22ac17a3"
Oct 31 02:07:12.828820 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:07:12.828805    4964 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cad610a9b748cbf4df64bcd9825a3333c00ab79bfdc3e210e5fc40ab22ac17a3"} err="failed to get container status \"cad610a9b748cbf4df64bcd9825a3333c00ab79bfdc3e210e5fc40ab22ac17a3\": rpc error: code = NotFound desc = could not find container \"cad610a9b748cbf4df64bcd9825a3333c00ab79bfdc3e210e5fc40ab22ac17a3\": container with ID starting with cad610a9b748cbf4df64bcd9825a3333c00ab79bfdc3e210e5fc40ab22ac17a3 not found: ID does not exist"
Oct 31 02:07:12.980184 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:07:12.980153    4964 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-98b70c-55c5cd675b-ltbhs"]
Oct 31 02:07:12.984878 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:07:12.984853    4964 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-98b70c-55c5cd675b-ltbhs"]
Oct 31 02:07:13.778159 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:07:13.778123    4964 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5ce8150f-4c48-4d65-8416-442f611c53fc" path="/var/lib/kubelet/pods/5ce8150f-4c48-4d65-8416-442f611c53fc/volumes"
Oct 31 02:07:13.901790 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:07:13.901749    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 02:07:14.085880 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:07:14.085793    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 02:07:14.260550 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:07:14.260497    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 02:07:14.433769 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:07:14.433653    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 02:07:15.164811 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:07:15.164776    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 02:07:19.425085 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:07:19.425047    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 02:07:21.039272 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:07:21.039238    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 02:07:21.226581 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:07:21.226545    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 02:07:21.344124 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:07:21.344048    4964 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-98b70c-openshift-default-7b4cfc957-9ntnk"]
Oct 31 02:07:21.344287 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:07:21.344262    4964 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w0-98b70c-openshift-default-7b4cfc957-9ntnk" podUID="55ee39ac-8129-4228-b0d7-a9b7841d3969" containerName="istio-proxy" containerID="cri-o://bd4e5278482b509af0d764b7c3d16f8d4a61ede0d0513849adf2ac1664727a5d" gracePeriod=30
Oct 31 02:07:21.417236 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:07:21.417187    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 02:07:24.132733 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:07:24.132693    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 02:07:24.549269 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:07:24.549231    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 02:07:25.797308 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:07:25.797271    4964 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-051514-546778dd4c-vbm49"]
Oct 31 02:07:25.907212 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:07:25.907171    4964 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-051514-bd7d495b7-pkp25"]
Oct 31 02:07:25.955802 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:07:25.955753    4964 patch_prober.go:29] interesting pod/e2e-shared-a-w0-98b70c-openshift-default-7b4cfc957-9ntnk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 02:07:25.956009 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:07:25.955834    4964 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w0-98b70c-openshift-default-7b4cfc957-9ntnk" podUID="55ee39ac-8129-4228-b0d7-a9b7841d3969" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 02:07:26.206545 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:07:26.206448    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 02:07:26.384282 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:07:26.384251    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 02:07:26.628206 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:07:26.628184    4964 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-98b70c-openshift-default-7b4cfc957-9ntnk"
Oct 31 02:07:26.739913 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:07:26.739821    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/55ee39ac-8129-4228-b0d7-a9b7841d3969-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/55ee39ac-8129-4228-b0d7-a9b7841d3969-istio-podinfo\") pod \"55ee39ac-8129-4228-b0d7-a9b7841d3969\" (UID: \"55ee39ac-8129-4228-b0d7-a9b7841d3969\") "
Oct 31 02:07:26.740100 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:07:26.739926    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/55ee39ac-8129-4228-b0d7-a9b7841d3969-kube-api-access-w4rpm\" (UniqueName: \"kubernetes.io/projected/55ee39ac-8129-4228-b0d7-a9b7841d3969-kube-api-access-w4rpm\") pod \"55ee39ac-8129-4228-b0d7-a9b7841d3969\" (UID: \"55ee39ac-8129-4228-b0d7-a9b7841d3969\") "
Oct 31 02:07:26.740100 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:07:26.740038    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/55ee39ac-8129-4228-b0d7-a9b7841d3969-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/55ee39ac-8129-4228-b0d7-a9b7841d3969-credential-socket\") pod \"55ee39ac-8129-4228-b0d7-a9b7841d3969\" (UID: \"55ee39ac-8129-4228-b0d7-a9b7841d3969\") "
Oct 31 02:07:26.740100 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:07:26.740062    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/55ee39ac-8129-4228-b0d7-a9b7841d3969-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/55ee39ac-8129-4228-b0d7-a9b7841d3969-workload-certs\") pod \"55ee39ac-8129-4228-b0d7-a9b7841d3969\" (UID: \"55ee39ac-8129-4228-b0d7-a9b7841d3969\") "
Oct 31 02:07:26.740100 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:07:26.740078    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/55ee39ac-8129-4228-b0d7-a9b7841d3969-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/55ee39ac-8129-4228-b0d7-a9b7841d3969-istio-envoy\") pod \"55ee39ac-8129-4228-b0d7-a9b7841d3969\" (UID: \"55ee39ac-8129-4228-b0d7-a9b7841d3969\") "
Oct 31 02:07:26.740359 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:07:26.740124    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/55ee39ac-8129-4228-b0d7-a9b7841d3969-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/55ee39ac-8129-4228-b0d7-a9b7841d3969-istio-data\") pod \"55ee39ac-8129-4228-b0d7-a9b7841d3969\" (UID: \"55ee39ac-8129-4228-b0d7-a9b7841d3969\") "
Oct 31 02:07:26.740359 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:07:26.740168    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/55ee39ac-8129-4228-b0d7-a9b7841d3969-istio-token\" (UniqueName: \"kubernetes.io/projected/55ee39ac-8129-4228-b0d7-a9b7841d3969-istio-token\") pod \"55ee39ac-8129-4228-b0d7-a9b7841d3969\" (UID: \"55ee39ac-8129-4228-b0d7-a9b7841d3969\") "
Oct 31 02:07:26.740359 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:07:26.740200    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/55ee39ac-8129-4228-b0d7-a9b7841d3969-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/55ee39ac-8129-4228-b0d7-a9b7841d3969-istiod-ca-cert\") pod \"55ee39ac-8129-4228-b0d7-a9b7841d3969\" (UID: \"55ee39ac-8129-4228-b0d7-a9b7841d3969\") "
Oct 31 02:07:26.740359 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:07:26.740240    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/55ee39ac-8129-4228-b0d7-a9b7841d3969-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/55ee39ac-8129-4228-b0d7-a9b7841d3969-workload-socket\") pod \"55ee39ac-8129-4228-b0d7-a9b7841d3969\" (UID: \"55ee39ac-8129-4228-b0d7-a9b7841d3969\") "
Oct 31 02:07:26.740359 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:07:26.740302    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55ee39ac-8129-4228-b0d7-a9b7841d3969-workload-certs" pod "55ee39ac-8129-4228-b0d7-a9b7841d3969" (UID: "55ee39ac-8129-4228-b0d7-a9b7841d3969"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:07:26.740645 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:07:26.740358    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55ee39ac-8129-4228-b0d7-a9b7841d3969-credential-socket" pod "55ee39ac-8129-4228-b0d7-a9b7841d3969" (UID: "55ee39ac-8129-4228-b0d7-a9b7841d3969"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:07:26.740645 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:07:26.740466    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55ee39ac-8129-4228-b0d7-a9b7841d3969-workload-socket" pod "55ee39ac-8129-4228-b0d7-a9b7841d3969" (UID: "55ee39ac-8129-4228-b0d7-a9b7841d3969"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:07:26.740645 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:07:26.740598    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55ee39ac-8129-4228-b0d7-a9b7841d3969-istio-data" pod "55ee39ac-8129-4228-b0d7-a9b7841d3969" (UID: "55ee39ac-8129-4228-b0d7-a9b7841d3969"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:07:26.740835 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:07:26.740760    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/55ee39ac-8129-4228-b0d7-a9b7841d3969-istiod-ca-cert" pod "55ee39ac-8129-4228-b0d7-a9b7841d3969" (UID: "55ee39ac-8129-4228-b0d7-a9b7841d3969"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 02:07:26.741012 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:07:26.740955    4964 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/55ee39ac-8129-4228-b0d7-a9b7841d3969-workload-socket\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:07:26.741012 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:07:26.740981    4964 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/55ee39ac-8129-4228-b0d7-a9b7841d3969-credential-socket\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:07:26.741012 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:07:26.740995    4964 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/55ee39ac-8129-4228-b0d7-a9b7841d3969-workload-certs\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:07:26.741012 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:07:26.741008    4964 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/55ee39ac-8129-4228-b0d7-a9b7841d3969-istio-data\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:07:26.741012 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:07:26.741022    4964 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/55ee39ac-8129-4228-b0d7-a9b7841d3969-istiod-ca-cert\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:07:26.742595 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:07:26.742558    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/55ee39ac-8129-4228-b0d7-a9b7841d3969-kube-api-access-w4rpm" pod "55ee39ac-8129-4228-b0d7-a9b7841d3969" (UID: "55ee39ac-8129-4228-b0d7-a9b7841d3969"). InnerVolumeSpecName "kube-api-access-w4rpm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:07:26.742729 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:07:26.742706    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/55ee39ac-8129-4228-b0d7-a9b7841d3969-istio-token" pod "55ee39ac-8129-4228-b0d7-a9b7841d3969" (UID: "55ee39ac-8129-4228-b0d7-a9b7841d3969"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:07:26.742902 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:07:26.742880    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/55ee39ac-8129-4228-b0d7-a9b7841d3969-istio-podinfo" pod "55ee39ac-8129-4228-b0d7-a9b7841d3969" (UID: "55ee39ac-8129-4228-b0d7-a9b7841d3969"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 02:07:26.742947 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:07:26.742881    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55ee39ac-8129-4228-b0d7-a9b7841d3969-istio-envoy" pod "55ee39ac-8129-4228-b0d7-a9b7841d3969" (UID: "55ee39ac-8129-4228-b0d7-a9b7841d3969"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:07:26.842728 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:07:26.842687    4964 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/55ee39ac-8129-4228-b0d7-a9b7841d3969-istio-envoy\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:07:26.842728 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:07:26.842721    4964 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/55ee39ac-8129-4228-b0d7-a9b7841d3969-istio-token\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:07:26.842728 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:07:26.842731    4964 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/55ee39ac-8129-4228-b0d7-a9b7841d3969-istio-podinfo\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:07:26.843204 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:07:26.842741    4964 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w4rpm\" (UniqueName: \"kubernetes.io/projected/55ee39ac-8129-4228-b0d7-a9b7841d3969-kube-api-access-w4rpm\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:07:26.921032 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:07:26.920998    4964 generic.go:356] "Generic (PLEG): container finished" podID="55ee39ac-8129-4228-b0d7-a9b7841d3969" containerID="bd4e5278482b509af0d764b7c3d16f8d4a61ede0d0513849adf2ac1664727a5d" exitCode=0
Oct 31 02:07:26.921217 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:07:26.921042    4964 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-98b70c-openshift-default-7b4cfc957-9ntnk"
Oct 31 02:07:26.921217 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:07:26.921076    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-98b70c-openshift-default-7b4cfc957-9ntnk" event={"ID":"55ee39ac-8129-4228-b0d7-a9b7841d3969","Type":"ContainerDied","Data":"bd4e5278482b509af0d764b7c3d16f8d4a61ede0d0513849adf2ac1664727a5d"}
Oct 31 02:07:26.921217 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:07:26.921128    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-98b70c-openshift-default-7b4cfc957-9ntnk" event={"ID":"55ee39ac-8129-4228-b0d7-a9b7841d3969","Type":"ContainerDied","Data":"485c294d6340ce038f813985ddde58bc9fa215199bfb32efc9eabcbf8f4489d8"}
Oct 31 02:07:26.921217 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:07:26.921150    4964 scope.go:122] "RemoveContainer" containerID="bd4e5278482b509af0d764b7c3d16f8d4a61ede0d0513849adf2ac1664727a5d"
Oct 31 02:07:26.932572 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:07:26.932553    4964 scope.go:122] "RemoveContainer" containerID="1395b33b38671b1bce5439e4857d67389b27b91047f8ad7debd534f9773edd3f"
Oct 31 02:07:26.942488 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:07:26.942467    4964 scope.go:122] "RemoveContainer" containerID="bd4e5278482b509af0d764b7c3d16f8d4a61ede0d0513849adf2ac1664727a5d"
Oct 31 02:07:26.942759 ip-10-0-133-197 kubenswrapper[4964]: E1031 02:07:26.942736    4964 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bd4e5278482b509af0d764b7c3d16f8d4a61ede0d0513849adf2ac1664727a5d\": container with ID starting with bd4e5278482b509af0d764b7c3d16f8d4a61ede0d0513849adf2ac1664727a5d not found: ID does not exist" containerID="bd4e5278482b509af0d764b7c3d16f8d4a61ede0d0513849adf2ac1664727a5d"
Oct 31 02:07:26.942824 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:07:26.942777    4964 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bd4e5278482b509af0d764b7c3d16f8d4a61ede0d0513849adf2ac1664727a5d"} err="failed to get container status \"bd4e5278482b509af0d764b7c3d16f8d4a61ede0d0513849adf2ac1664727a5d\": rpc error: code = NotFound desc = could not find container \"bd4e5278482b509af0d764b7c3d16f8d4a61ede0d0513849adf2ac1664727a5d\": container with ID starting with bd4e5278482b509af0d764b7c3d16f8d4a61ede0d0513849adf2ac1664727a5d not found: ID does not exist"
Oct 31 02:07:26.942824 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:07:26.942796    4964 scope.go:122] "RemoveContainer" containerID="1395b33b38671b1bce5439e4857d67389b27b91047f8ad7debd534f9773edd3f"
Oct 31 02:07:26.943059 ip-10-0-133-197 kubenswrapper[4964]: E1031 02:07:26.943040    4964 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1395b33b38671b1bce5439e4857d67389b27b91047f8ad7debd534f9773edd3f\": container with ID starting with 1395b33b38671b1bce5439e4857d67389b27b91047f8ad7debd534f9773edd3f not found: ID does not exist" containerID="1395b33b38671b1bce5439e4857d67389b27b91047f8ad7debd534f9773edd3f"
Oct 31 02:07:26.943112 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:07:26.943063    4964 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1395b33b38671b1bce5439e4857d67389b27b91047f8ad7debd534f9773edd3f"} err="failed to get container status \"1395b33b38671b1bce5439e4857d67389b27b91047f8ad7debd534f9773edd3f\": rpc error: code = NotFound desc = could not find container \"1395b33b38671b1bce5439e4857d67389b27b91047f8ad7debd534f9773edd3f\": container with ID starting with 1395b33b38671b1bce5439e4857d67389b27b91047f8ad7debd534f9773edd3f not found: ID does not exist"
Oct 31 02:07:27.085532 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:07:27.085433    4964 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-98b70c-openshift-default-7b4cfc957-9ntnk"]
Oct 31 02:07:27.088736 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:07:27.088701    4964 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-98b70c-openshift-default-7b4cfc957-9ntnk"]
Oct 31 02:07:27.777932 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:07:27.777900    4964 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="55ee39ac-8129-4228-b0d7-a9b7841d3969" path="/var/lib/kubelet/pods/55ee39ac-8129-4228-b0d7-a9b7841d3969/volumes"
Oct 31 02:07:29.255720 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:07:29.255684    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 02:07:29.678268 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:07:29.678170    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 02:07:30.797773 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:07:30.797689    4964 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-051514-546778dd4c-vbm49" podUID="d2790587-1361-49b0-a964-2e7fa86effad" containerName="payload-processing" containerID="cri-o://a611d7f94cf8ff2f2c79886fa950e3434e575d851c56efde595a05cd71a74577" gracePeriod=25
Oct 31 02:07:30.907981 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:07:30.907906    4964 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-051514-bd7d495b7-pkp25" podUID="64532a2f-bc54-4250-a617-95865b1d7e26" containerName="payload-pre-processing" containerID="cri-o://b3d139e9aa9743d9393155d2f71bc9cce78e1912dfb6c132af9fc7eb37f995e8" gracePeriod=25
Oct 31 02:07:30.954690 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:07:30.954661    4964 generic.go:356] "Generic (PLEG): container finished" podID="d2790587-1361-49b0-a964-2e7fa86effad" containerID="a611d7f94cf8ff2f2c79886fa950e3434e575d851c56efde595a05cd71a74577" exitCode=0
Oct 31 02:07:30.954894 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:07:30.954875    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-051514-546778dd4c-vbm49" event={"ID":"d2790587-1361-49b0-a964-2e7fa86effad","Type":"ContainerDied","Data":"a611d7f94cf8ff2f2c79886fa950e3434e575d851c56efde595a05cd71a74577"}
Oct 31 02:07:31.071413 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:07:31.071387    4964 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-051514-546778dd4c-vbm49"
Oct 31 02:07:31.180988 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:07:31.180967    4964 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-051514-bd7d495b7-pkp25"
Oct 31 02:07:31.197772 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:07:31.197745    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d2790587-1361-49b0-a964-2e7fa86effad-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d2790587-1361-49b0-a964-2e7fa86effad-plugins-config-volume\") pod \"d2790587-1361-49b0-a964-2e7fa86effad\" (UID: \"d2790587-1361-49b0-a964-2e7fa86effad\") "
Oct 31 02:07:31.197919 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:07:31.197842    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d2790587-1361-49b0-a964-2e7fa86effad-kube-api-access-2vzgm\" (UniqueName: \"kubernetes.io/projected/d2790587-1361-49b0-a964-2e7fa86effad-kube-api-access-2vzgm\") pod \"d2790587-1361-49b0-a964-2e7fa86effad\" (UID: \"d2790587-1361-49b0-a964-2e7fa86effad\") "
Oct 31 02:07:31.198160 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:07:31.198136    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d2790587-1361-49b0-a964-2e7fa86effad-plugins-config-volume" pod "d2790587-1361-49b0-a964-2e7fa86effad" (UID: "d2790587-1361-49b0-a964-2e7fa86effad"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 02:07:31.198356 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:07:31.198337    4964 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d2790587-1361-49b0-a964-2e7fa86effad-plugins-config-volume\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:07:31.200431 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:07:31.200408    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d2790587-1361-49b0-a964-2e7fa86effad-kube-api-access-2vzgm" pod "d2790587-1361-49b0-a964-2e7fa86effad" (UID: "d2790587-1361-49b0-a964-2e7fa86effad"). InnerVolumeSpecName "kube-api-access-2vzgm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:07:31.299047 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:07:31.298942    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/64532a2f-bc54-4250-a617-95865b1d7e26-kube-api-access-cz9gh\" (UniqueName: \"kubernetes.io/projected/64532a2f-bc54-4250-a617-95865b1d7e26-kube-api-access-cz9gh\") pod \"64532a2f-bc54-4250-a617-95865b1d7e26\" (UID: \"64532a2f-bc54-4250-a617-95865b1d7e26\") "
Oct 31 02:07:31.299047 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:07:31.299020    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/64532a2f-bc54-4250-a617-95865b1d7e26-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/64532a2f-bc54-4250-a617-95865b1d7e26-plugins-config-volume\") pod \"64532a2f-bc54-4250-a617-95865b1d7e26\" (UID: \"64532a2f-bc54-4250-a617-95865b1d7e26\") "
Oct 31 02:07:31.299676 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:07:31.299653    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/64532a2f-bc54-4250-a617-95865b1d7e26-plugins-config-volume" pod "64532a2f-bc54-4250-a617-95865b1d7e26" (UID: "64532a2f-bc54-4250-a617-95865b1d7e26"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 02:07:31.299914 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:07:31.299887    4964 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2vzgm\" (UniqueName: \"kubernetes.io/projected/d2790587-1361-49b0-a964-2e7fa86effad-kube-api-access-2vzgm\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:07:31.300003 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:07:31.299912    4964 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/64532a2f-bc54-4250-a617-95865b1d7e26-plugins-config-volume\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:07:31.302175 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:07:31.302141    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/64532a2f-bc54-4250-a617-95865b1d7e26-kube-api-access-cz9gh" pod "64532a2f-bc54-4250-a617-95865b1d7e26" (UID: "64532a2f-bc54-4250-a617-95865b1d7e26"). InnerVolumeSpecName "kube-api-access-cz9gh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:07:31.400705 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:07:31.400669    4964 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cz9gh\" (UniqueName: \"kubernetes.io/projected/64532a2f-bc54-4250-a617-95865b1d7e26-kube-api-access-cz9gh\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:07:31.962670 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:07:31.962635    4964 generic.go:356] "Generic (PLEG): container finished" podID="64532a2f-bc54-4250-a617-95865b1d7e26" containerID="b3d139e9aa9743d9393155d2f71bc9cce78e1912dfb6c132af9fc7eb37f995e8" exitCode=0
Oct 31 02:07:31.963127 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:07:31.962692    4964 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-051514-bd7d495b7-pkp25"
Oct 31 02:07:31.963127 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:07:31.962717    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-051514-bd7d495b7-pkp25" event={"ID":"64532a2f-bc54-4250-a617-95865b1d7e26","Type":"ContainerDied","Data":"b3d139e9aa9743d9393155d2f71bc9cce78e1912dfb6c132af9fc7eb37f995e8"}
Oct 31 02:07:31.963127 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:07:31.962770    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-051514-bd7d495b7-pkp25" event={"ID":"64532a2f-bc54-4250-a617-95865b1d7e26","Type":"ContainerDied","Data":"408c6920fed8d5a343b7b119c192ad7bf114f2e0db0f651cc0ff734dcdfc4b03"}
Oct 31 02:07:31.963127 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:07:31.962791    4964 scope.go:122] "RemoveContainer" containerID="b3d139e9aa9743d9393155d2f71bc9cce78e1912dfb6c132af9fc7eb37f995e8"
Oct 31 02:07:31.964452 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:07:31.964433    4964 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-051514-546778dd4c-vbm49"
Oct 31 02:07:31.964582 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:07:31.964452    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-051514-546778dd4c-vbm49" event={"ID":"d2790587-1361-49b0-a964-2e7fa86effad","Type":"ContainerDied","Data":"2078284688fc53cc4f6f42ab006f411e317e0b9b9a232f520fc711d180578957"}
Oct 31 02:07:31.972803 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:07:31.972785    4964 scope.go:122] "RemoveContainer" containerID="b3d139e9aa9743d9393155d2f71bc9cce78e1912dfb6c132af9fc7eb37f995e8"
Oct 31 02:07:31.973062 ip-10-0-133-197 kubenswrapper[4964]: E1031 02:07:31.973045    4964 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b3d139e9aa9743d9393155d2f71bc9cce78e1912dfb6c132af9fc7eb37f995e8\": container with ID starting with b3d139e9aa9743d9393155d2f71bc9cce78e1912dfb6c132af9fc7eb37f995e8 not found: ID does not exist" containerID="b3d139e9aa9743d9393155d2f71bc9cce78e1912dfb6c132af9fc7eb37f995e8"
Oct 31 02:07:31.973116 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:07:31.973070    4964 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b3d139e9aa9743d9393155d2f71bc9cce78e1912dfb6c132af9fc7eb37f995e8"} err="failed to get container status \"b3d139e9aa9743d9393155d2f71bc9cce78e1912dfb6c132af9fc7eb37f995e8\": rpc error: code = NotFound desc = could not find container \"b3d139e9aa9743d9393155d2f71bc9cce78e1912dfb6c132af9fc7eb37f995e8\": container with ID starting with b3d139e9aa9743d9393155d2f71bc9cce78e1912dfb6c132af9fc7eb37f995e8 not found: ID does not exist"
Oct 31 02:07:31.973116 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:07:31.973085    4964 scope.go:122] "RemoveContainer" containerID="a611d7f94cf8ff2f2c79886fa950e3434e575d851c56efde595a05cd71a74577"
Oct 31 02:07:32.126601 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:07:32.126517    4964 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-051514-546778dd4c-vbm49"]
Oct 31 02:07:32.129277 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:07:32.129253    4964 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-051514-546778dd4c-vbm49"]
Oct 31 02:07:32.236688 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:07:32.236604    4964 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-051514-bd7d495b7-pkp25"]
Oct 31 02:07:32.240038 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:07:32.240014    4964 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-051514-bd7d495b7-pkp25"]
Oct 31 02:07:33.777824 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:07:33.777791    4964 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="64532a2f-bc54-4250-a617-95865b1d7e26" path="/var/lib/kubelet/pods/64532a2f-bc54-4250-a617-95865b1d7e26/volumes"
Oct 31 02:07:33.778232 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:07:33.778192    4964 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d2790587-1361-49b0-a964-2e7fa86effad" path="/var/lib/kubelet/pods/d2790587-1361-49b0-a964-2e7fa86effad/volumes"
Oct 31 02:07:34.134969 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:07:34.134601    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 02:07:34.360061 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:07:34.360025    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 02:07:34.530497 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:07:34.530455    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 02:07:36.562592 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:07:36.562546    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 02:07:36.742388 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:07:36.742351    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 02:07:38.517390 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:07:38.517353    4964 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-051514-6d5fbc88-4xc4r"]
Oct 31 02:07:38.517821 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:07:38.517575    4964 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-051514-6d5fbc88-4xc4r" podUID="05dff628-00f6-4008-8eee-f7e7dcdf8729" containerName="maas-api" containerID="cri-o://ecb949a45b3da47bb0a0884365e71dff9e7d9b6558c71e228aa5d4882ae96dac" gracePeriod=30
Oct 31 02:07:38.657669 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:07:38.657633    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 02:07:38.799193 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:07:38.799162    4964 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-051514-6d5fbc88-4xc4r"
Oct 31 02:07:38.886876 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:07:38.886827    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/05dff628-00f6-4008-8eee-f7e7dcdf8729-kube-api-access-nrg7c\" (UniqueName: \"kubernetes.io/projected/05dff628-00f6-4008-8eee-f7e7dcdf8729-kube-api-access-nrg7c\") pod \"05dff628-00f6-4008-8eee-f7e7dcdf8729\" (UID: \"05dff628-00f6-4008-8eee-f7e7dcdf8729\") "
Oct 31 02:07:38.887101 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:07:38.886946    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/05dff628-00f6-4008-8eee-f7e7dcdf8729-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/05dff628-00f6-4008-8eee-f7e7dcdf8729-openshift-service-ca\") pod \"05dff628-00f6-4008-8eee-f7e7dcdf8729\" (UID: \"05dff628-00f6-4008-8eee-f7e7dcdf8729\") "
Oct 31 02:07:38.887101 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:07:38.887060    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/05dff628-00f6-4008-8eee-f7e7dcdf8729-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/05dff628-00f6-4008-8eee-f7e7dcdf8729-maas-api-tls\") pod \"05dff628-00f6-4008-8eee-f7e7dcdf8729\" (UID: \"05dff628-00f6-4008-8eee-f7e7dcdf8729\") "
Oct 31 02:07:38.887258 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:07:38.887159    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/05dff628-00f6-4008-8eee-f7e7dcdf8729-metrics-certs\" (UniqueName: \"kubernetes.io/secret/05dff628-00f6-4008-8eee-f7e7dcdf8729-metrics-certs\") pod \"05dff628-00f6-4008-8eee-f7e7dcdf8729\" (UID: \"05dff628-00f6-4008-8eee-f7e7dcdf8729\") "
Oct 31 02:07:38.888226 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:07:38.887678    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/05dff628-00f6-4008-8eee-f7e7dcdf8729-openshift-service-ca" pod "05dff628-00f6-4008-8eee-f7e7dcdf8729" (UID: "05dff628-00f6-4008-8eee-f7e7dcdf8729"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 02:07:38.888226 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:07:38.887781    4964 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/05dff628-00f6-4008-8eee-f7e7dcdf8729-openshift-service-ca\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:07:38.889988 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:07:38.889956    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/05dff628-00f6-4008-8eee-f7e7dcdf8729-kube-api-access-nrg7c" pod "05dff628-00f6-4008-8eee-f7e7dcdf8729" (UID: "05dff628-00f6-4008-8eee-f7e7dcdf8729"). InnerVolumeSpecName "kube-api-access-nrg7c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:07:38.890605 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:07:38.890580    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/05dff628-00f6-4008-8eee-f7e7dcdf8729-metrics-certs" pod "05dff628-00f6-4008-8eee-f7e7dcdf8729" (UID: "05dff628-00f6-4008-8eee-f7e7dcdf8729"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 02:07:38.890931 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:07:38.890890    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/05dff628-00f6-4008-8eee-f7e7dcdf8729-maas-api-tls" pod "05dff628-00f6-4008-8eee-f7e7dcdf8729" (UID: "05dff628-00f6-4008-8eee-f7e7dcdf8729"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 02:07:38.988975 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:07:38.988941    4964 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/05dff628-00f6-4008-8eee-f7e7dcdf8729-maas-api-tls\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:07:38.988975 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:07:38.988973    4964 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/05dff628-00f6-4008-8eee-f7e7dcdf8729-metrics-certs\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:07:38.988975 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:07:38.988983    4964 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nrg7c\" (UniqueName: \"kubernetes.io/projected/05dff628-00f6-4008-8eee-f7e7dcdf8729-kube-api-access-nrg7c\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:07:39.017562 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:07:39.017523    4964 generic.go:356] "Generic (PLEG): container finished" podID="05dff628-00f6-4008-8eee-f7e7dcdf8729" containerID="ecb949a45b3da47bb0a0884365e71dff9e7d9b6558c71e228aa5d4882ae96dac" exitCode=0
Oct 31 02:07:39.017774 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:07:39.017575    4964 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-051514-6d5fbc88-4xc4r"
Oct 31 02:07:39.017774 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:07:39.017607    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-051514-6d5fbc88-4xc4r" event={"ID":"05dff628-00f6-4008-8eee-f7e7dcdf8729","Type":"ContainerDied","Data":"ecb949a45b3da47bb0a0884365e71dff9e7d9b6558c71e228aa5d4882ae96dac"}
Oct 31 02:07:39.017774 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:07:39.017666    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-051514-6d5fbc88-4xc4r" event={"ID":"05dff628-00f6-4008-8eee-f7e7dcdf8729","Type":"ContainerDied","Data":"ca5b2ea2d9f23a747055f01cb214bb8ea5fe31a2809b6be332b05941c21bf3a4"}
Oct 31 02:07:39.017774 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:07:39.017696    4964 scope.go:122] "RemoveContainer" containerID="ecb949a45b3da47bb0a0884365e71dff9e7d9b6558c71e228aa5d4882ae96dac"
Oct 31 02:07:39.028889 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:07:39.028871    4964 scope.go:122] "RemoveContainer" containerID="ecb949a45b3da47bb0a0884365e71dff9e7d9b6558c71e228aa5d4882ae96dac"
Oct 31 02:07:39.029247 ip-10-0-133-197 kubenswrapper[4964]: E1031 02:07:39.029228    4964 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ecb949a45b3da47bb0a0884365e71dff9e7d9b6558c71e228aa5d4882ae96dac\": container with ID starting with ecb949a45b3da47bb0a0884365e71dff9e7d9b6558c71e228aa5d4882ae96dac not found: ID does not exist" containerID="ecb949a45b3da47bb0a0884365e71dff9e7d9b6558c71e228aa5d4882ae96dac"
Oct 31 02:07:39.029354 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:07:39.029256    4964 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ecb949a45b3da47bb0a0884365e71dff9e7d9b6558c71e228aa5d4882ae96dac"} err="failed to get container status \"ecb949a45b3da47bb0a0884365e71dff9e7d9b6558c71e228aa5d4882ae96dac\": rpc error: code = NotFound desc = could not find container \"ecb949a45b3da47bb0a0884365e71dff9e7d9b6558c71e228aa5d4882ae96dac\": container with ID starting with ecb949a45b3da47bb0a0884365e71dff9e7d9b6558c71e228aa5d4882ae96dac not found: ID does not exist"
Oct 31 02:07:39.180937 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:07:39.180901    4964 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-051514-6d5fbc88-4xc4r"]
Oct 31 02:07:39.186346 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:07:39.186301    4964 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-051514-6d5fbc88-4xc4r"]
Oct 31 02:07:39.780122 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:07:39.780087    4964 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="05dff628-00f6-4008-8eee-f7e7dcdf8729" path="/var/lib/kubelet/pods/05dff628-00f6-4008-8eee-f7e7dcdf8729/volumes"
Oct 31 02:07:40.178515 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:07:40.178428    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 02:07:40.411225 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:07:40.411185    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 02:07:40.586917 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:07:40.586875    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 02:07:41.149489 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:07:41.149441    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 02:07:41.239148 ip-10-0-133-197 kubenswrapper[4964]: E1031 02:07:41.239111    4964 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q56x5" containerName="extractor"
Oct 31 02:07:41.788825 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:07:41.788793    4964 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-autores-183266/autores-model-183266-kserve-758f9bc869-tc88v"]
Oct 31 02:07:41.789541 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:07:41.789519    4964 cpu_manager.go:414] "containerMap: removing container" podUID="3d0e640a-41cb-48fe-9b7f-8927fd534ca4" containerName="main"
Oct 31 02:07:41.789541 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:07:41.789540    4964 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3d0e640a-41cb-48fe-9b7f-8927fd534ca4" containerName="main"
Oct 31 02:07:41.789700 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:07:41.789561    4964 cpu_manager.go:414] "containerMap: removing container" podUID="3d4d5c24-0d5f-4183-acab-5e2ae32050f5" containerName="main"
Oct 31 02:07:41.789700 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:07:41.789569    4964 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3d4d5c24-0d5f-4183-acab-5e2ae32050f5" containerName="main"
Oct 31 02:07:41.789700 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:07:41.789594    4964 cpu_manager.go:414] "containerMap: removing container" podUID="380f14cc-28b6-4557-8dd5-7a36f0bbb3be" containerName="payload-processing"
Oct 31 02:07:41.789700 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:07:41.789602    4964 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="380f14cc-28b6-4557-8dd5-7a36f0bbb3be" containerName="payload-processing"
Oct 31 02:07:41.789700 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:07:41.789617    4964 cpu_manager.go:414] "containerMap: removing container" podUID="5ce8150f-4c48-4d65-8416-442f611c53fc" containerName="maas-api"
Oct 31 02:07:41.789700 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:07:41.789624    4964 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5ce8150f-4c48-4d65-8416-442f611c53fc" containerName="maas-api"
Oct 31 02:07:41.789700 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:07:41.789638    4964 cpu_manager.go:414] "containerMap: removing container" podUID="55ee39ac-8129-4228-b0d7-a9b7841d3969" containerName="istio-proxy"
Oct 31 02:07:41.789700 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:07:41.789645    4964 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="55ee39ac-8129-4228-b0d7-a9b7841d3969" containerName="istio-proxy"
Oct 31 02:07:41.789700 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:07:41.789657    4964 cpu_manager.go:414] "containerMap: removing container" podUID="5b8433e9-df92-402c-9873-83646d9bfee5" containerName="main"
Oct 31 02:07:41.789700 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:07:41.789663    4964 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5b8433e9-df92-402c-9873-83646d9bfee5" containerName="main"
Oct 31 02:07:41.789700 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:07:41.789680    4964 cpu_manager.go:414] "containerMap: removing container" podUID="05dff628-00f6-4008-8eee-f7e7dcdf8729" containerName="maas-api"
Oct 31 02:07:41.789700 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:07:41.789686    4964 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="05dff628-00f6-4008-8eee-f7e7dcdf8729" containerName="maas-api"
Oct 31 02:07:41.789700 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:07:41.789698    4964 cpu_manager.go:414] "containerMap: removing container" podUID="55ee39ac-8129-4228-b0d7-a9b7841d3969" containerName="istio-proxy"
Oct 31 02:07:41.789700 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:07:41.789705    4964 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="55ee39ac-8129-4228-b0d7-a9b7841d3969" containerName="istio-proxy"
Oct 31 02:07:41.790122 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:07:41.789717    4964 cpu_manager.go:414] "containerMap: removing container" podUID="d2790587-1361-49b0-a964-2e7fa86effad" containerName="payload-processing"
Oct 31 02:07:41.790122 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:07:41.789724    4964 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d2790587-1361-49b0-a964-2e7fa86effad" containerName="payload-processing"
Oct 31 02:07:41.790122 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:07:41.789733    4964 cpu_manager.go:414] "containerMap: removing container" podUID="8508d596-d89b-4f67-9a57-f6ba4890ce5e" containerName="istio-proxy"
Oct 31 02:07:41.790122 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:07:41.789740    4964 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8508d596-d89b-4f67-9a57-f6ba4890ce5e" containerName="istio-proxy"
Oct 31 02:07:41.790122 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:07:41.789756    4964 cpu_manager.go:414] "containerMap: removing container" podUID="1036dce3-2ee4-4dc2-a8e9-094d35d989ef" containerName="maas-api"
Oct 31 02:07:41.790122 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:07:41.789762    4964 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1036dce3-2ee4-4dc2-a8e9-094d35d989ef" containerName="maas-api"
Oct 31 02:07:41.790122 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:07:41.789774    4964 cpu_manager.go:414] "containerMap: removing container" podUID="64532a2f-bc54-4250-a617-95865b1d7e26" containerName="payload-pre-processing"
Oct 31 02:07:41.790122 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:07:41.789780    4964 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="64532a2f-bc54-4250-a617-95865b1d7e26" containerName="payload-pre-processing"
Oct 31 02:07:41.790122 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:07:41.789810    4964 cpu_manager.go:414] "containerMap: removing container" podUID="383b0139-d5c3-4492-8245-8faac4af0a10" containerName="istio-proxy"
Oct 31 02:07:41.790122 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:07:41.789818    4964 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="383b0139-d5c3-4492-8245-8faac4af0a10" containerName="istio-proxy"
Oct 31 02:07:41.790122 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:07:41.789830    4964 cpu_manager.go:414] "containerMap: removing container" podUID="9700e3ca-d2e5-4171-a3b0-3506d01d7a00" containerName="main"
Oct 31 02:07:41.790122 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:07:41.789837    4964 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9700e3ca-d2e5-4171-a3b0-3506d01d7a00" containerName="main"
Oct 31 02:07:41.790122 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:07:41.789953    4964 memory_manager.go:365] "RemoveStaleState removing state" podUID="5ce8150f-4c48-4d65-8416-442f611c53fc" containerName="maas-api"
Oct 31 02:07:41.790122 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:07:41.789965    4964 memory_manager.go:365] "RemoveStaleState removing state" podUID="55ee39ac-8129-4228-b0d7-a9b7841d3969" containerName="istio-proxy"
Oct 31 02:07:41.790122 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:07:41.789976    4964 memory_manager.go:365] "RemoveStaleState removing state" podUID="5b8433e9-df92-402c-9873-83646d9bfee5" containerName="main"
Oct 31 02:07:41.790122 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:07:41.789985    4964 memory_manager.go:365] "RemoveStaleState removing state" podUID="05dff628-00f6-4008-8eee-f7e7dcdf8729" containerName="maas-api"
Oct 31 02:07:41.790122 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:07:41.789994    4964 memory_manager.go:365] "RemoveStaleState removing state" podUID="55ee39ac-8129-4228-b0d7-a9b7841d3969" containerName="istio-proxy"
Oct 31 02:07:41.790122 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:07:41.790020    4964 memory_manager.go:365] "RemoveStaleState removing state" podUID="d2790587-1361-49b0-a964-2e7fa86effad" containerName="payload-processing"
Oct 31 02:07:41.790122 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:07:41.790030    4964 memory_manager.go:365] "RemoveStaleState removing state" podUID="8508d596-d89b-4f67-9a57-f6ba4890ce5e" containerName="istio-proxy"
Oct 31 02:07:41.790122 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:07:41.790040    4964 memory_manager.go:365] "RemoveStaleState removing state" podUID="1036dce3-2ee4-4dc2-a8e9-094d35d989ef" containerName="maas-api"
Oct 31 02:07:41.790122 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:07:41.790048    4964 memory_manager.go:365] "RemoveStaleState removing state" podUID="64532a2f-bc54-4250-a617-95865b1d7e26" containerName="payload-pre-processing"
Oct 31 02:07:41.790122 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:07:41.790060    4964 memory_manager.go:365] "RemoveStaleState removing state" podUID="383b0139-d5c3-4492-8245-8faac4af0a10" containerName="istio-proxy"
Oct 31 02:07:41.790122 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:07:41.790069    4964 memory_manager.go:365] "RemoveStaleState removing state" podUID="9700e3ca-d2e5-4171-a3b0-3506d01d7a00" containerName="main"
Oct 31 02:07:41.790122 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:07:41.790078    4964 memory_manager.go:365] "RemoveStaleState removing state" podUID="8508d596-d89b-4f67-9a57-f6ba4890ce5e" containerName="istio-proxy"
Oct 31 02:07:41.790122 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:07:41.790088    4964 memory_manager.go:365] "RemoveStaleState removing state" podUID="3d0e640a-41cb-48fe-9b7f-8927fd534ca4" containerName="main"
Oct 31 02:07:41.790122 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:07:41.790098    4964 memory_manager.go:365] "RemoveStaleState removing state" podUID="3d4d5c24-0d5f-4183-acab-5e2ae32050f5" containerName="main"
Oct 31 02:07:41.790122 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:07:41.790108    4964 memory_manager.go:365] "RemoveStaleState removing state" podUID="380f14cc-28b6-4557-8dd5-7a36f0bbb3be" containerName="payload-processing"
Oct 31 02:07:41.795398 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:07:41.795376    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-183266/autores-model-183266-kserve-758f9bc869-tc88v"
Oct 31 02:07:41.834259 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:07:41.834227    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-autores-183266/autores-model-183266-kserve-758f9bc869-tc88v"]
Oct 31 02:07:41.878680 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:07:41.878651    4964 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-autores-183266\"/\"openshift-service-ca.crt\""
Oct 31 02:07:41.878680 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:07:41.878654    4964 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-autores-183266\"/\"kube-root-ca.crt\""
Oct 31 02:07:41.900549 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:07:41.900526    4964 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-autores-183266\"/\"default-dockercfg-9qwtp\""
Oct 31 02:07:41.900549 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:07:41.900534    4964 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-autores-183266\"/\"autores-model-183266-kserve-self-signed-certs\""
Oct 31 02:07:41.909408 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:07:41.909378    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 02:07:41.923470 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:07:41.923444    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5c156010-30a3-434b-a027-305788f7ed3b-home\") pod \"autores-model-183266-kserve-758f9bc869-tc88v\" (UID: \"5c156010-30a3-434b-a027-305788f7ed3b\") " pod="ai-tenant-e2e-autores-183266/autores-model-183266-kserve-758f9bc869-tc88v"
Oct 31 02:07:41.923470 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:07:41.923479    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5c156010-30a3-434b-a027-305788f7ed3b-tmp-dir\") pod \"autores-model-183266-kserve-758f9bc869-tc88v\" (UID: \"5c156010-30a3-434b-a027-305788f7ed3b\") " pod="ai-tenant-e2e-autores-183266/autores-model-183266-kserve-758f9bc869-tc88v"
Oct 31 02:07:41.923726 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:07:41.923658    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5c156010-30a3-434b-a027-305788f7ed3b-tls-certs\") pod \"autores-model-183266-kserve-758f9bc869-tc88v\" (UID: \"5c156010-30a3-434b-a027-305788f7ed3b\") " pod="ai-tenant-e2e-autores-183266/autores-model-183266-kserve-758f9bc869-tc88v"
Oct 31 02:07:41.923777 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:07:41.923727    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5c156010-30a3-434b-a027-305788f7ed3b-model-cache\") pod \"autores-model-183266-kserve-758f9bc869-tc88v\" (UID: \"5c156010-30a3-434b-a027-305788f7ed3b\") " pod="ai-tenant-e2e-autores-183266/autores-model-183266-kserve-758f9bc869-tc88v"
Oct 31 02:07:41.923990 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:07:41.923960    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wk4pg\" (UniqueName: \"kubernetes.io/projected/5c156010-30a3-434b-a027-305788f7ed3b-kube-api-access-wk4pg\") pod \"autores-model-183266-kserve-758f9bc869-tc88v\" (UID: \"5c156010-30a3-434b-a027-305788f7ed3b\") " pod="ai-tenant-e2e-autores-183266/autores-model-183266-kserve-758f9bc869-tc88v"
Oct 31 02:07:41.924079 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:07:41.924060    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5c156010-30a3-434b-a027-305788f7ed3b-dshm\") pod \"autores-model-183266-kserve-758f9bc869-tc88v\" (UID: \"5c156010-30a3-434b-a027-305788f7ed3b\") " pod="ai-tenant-e2e-autores-183266/autores-model-183266-kserve-758f9bc869-tc88v"
Oct 31 02:07:42.024911 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:07:42.024885    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wk4pg\" (UniqueName: \"kubernetes.io/projected/5c156010-30a3-434b-a027-305788f7ed3b-kube-api-access-wk4pg\") pod \"autores-model-183266-kserve-758f9bc869-tc88v\" (UID: \"5c156010-30a3-434b-a027-305788f7ed3b\") " pod="ai-tenant-e2e-autores-183266/autores-model-183266-kserve-758f9bc869-tc88v"
Oct 31 02:07:42.025051 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:07:42.024937    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5c156010-30a3-434b-a027-305788f7ed3b-dshm\") pod \"autores-model-183266-kserve-758f9bc869-tc88v\" (UID: \"5c156010-30a3-434b-a027-305788f7ed3b\") " pod="ai-tenant-e2e-autores-183266/autores-model-183266-kserve-758f9bc869-tc88v"
Oct 31 02:07:42.025051 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:07:42.024962    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5c156010-30a3-434b-a027-305788f7ed3b-home\") pod \"autores-model-183266-kserve-758f9bc869-tc88v\" (UID: \"5c156010-30a3-434b-a027-305788f7ed3b\") " pod="ai-tenant-e2e-autores-183266/autores-model-183266-kserve-758f9bc869-tc88v"
Oct 31 02:07:42.025051 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:07:42.024977    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5c156010-30a3-434b-a027-305788f7ed3b-tmp-dir\") pod \"autores-model-183266-kserve-758f9bc869-tc88v\" (UID: \"5c156010-30a3-434b-a027-305788f7ed3b\") " pod="ai-tenant-e2e-autores-183266/autores-model-183266-kserve-758f9bc869-tc88v"
Oct 31 02:07:42.025051 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:07:42.025007    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5c156010-30a3-434b-a027-305788f7ed3b-tls-certs\") pod \"autores-model-183266-kserve-758f9bc869-tc88v\" (UID: \"5c156010-30a3-434b-a027-305788f7ed3b\") " pod="ai-tenant-e2e-autores-183266/autores-model-183266-kserve-758f9bc869-tc88v"
Oct 31 02:07:42.025051 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:07:42.025043    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5c156010-30a3-434b-a027-305788f7ed3b-model-cache\") pod \"autores-model-183266-kserve-758f9bc869-tc88v\" (UID: \"5c156010-30a3-434b-a027-305788f7ed3b\") " pod="ai-tenant-e2e-autores-183266/autores-model-183266-kserve-758f9bc869-tc88v"
Oct 31 02:07:42.025520 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:07:42.025487    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5c156010-30a3-434b-a027-305788f7ed3b-home\") pod \"autores-model-183266-kserve-758f9bc869-tc88v\" (UID: \"5c156010-30a3-434b-a027-305788f7ed3b\") " pod="ai-tenant-e2e-autores-183266/autores-model-183266-kserve-758f9bc869-tc88v"
Oct 31 02:07:42.025642 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:07:42.025529    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5c156010-30a3-434b-a027-305788f7ed3b-model-cache\") pod \"autores-model-183266-kserve-758f9bc869-tc88v\" (UID: \"5c156010-30a3-434b-a027-305788f7ed3b\") " pod="ai-tenant-e2e-autores-183266/autores-model-183266-kserve-758f9bc869-tc88v"
Oct 31 02:07:42.025642 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:07:42.025567    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5c156010-30a3-434b-a027-305788f7ed3b-tmp-dir\") pod \"autores-model-183266-kserve-758f9bc869-tc88v\" (UID: \"5c156010-30a3-434b-a027-305788f7ed3b\") " pod="ai-tenant-e2e-autores-183266/autores-model-183266-kserve-758f9bc869-tc88v"
Oct 31 02:07:42.027658 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:07:42.027637    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5c156010-30a3-434b-a027-305788f7ed3b-dshm\") pod \"autores-model-183266-kserve-758f9bc869-tc88v\" (UID: \"5c156010-30a3-434b-a027-305788f7ed3b\") " pod="ai-tenant-e2e-autores-183266/autores-model-183266-kserve-758f9bc869-tc88v"
Oct 31 02:07:42.027819 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:07:42.027799    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5c156010-30a3-434b-a027-305788f7ed3b-tls-certs\") pod \"autores-model-183266-kserve-758f9bc869-tc88v\" (UID: \"5c156010-30a3-434b-a027-305788f7ed3b\") " pod="ai-tenant-e2e-autores-183266/autores-model-183266-kserve-758f9bc869-tc88v"
Oct 31 02:07:42.081227 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:07:42.081165    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wk4pg\" (UniqueName: \"kubernetes.io/projected/5c156010-30a3-434b-a027-305788f7ed3b-kube-api-access-wk4pg\") pod \"autores-model-183266-kserve-758f9bc869-tc88v\" (UID: \"5c156010-30a3-434b-a027-305788f7ed3b\") " pod="ai-tenant-e2e-autores-183266/autores-model-183266-kserve-758f9bc869-tc88v"
Oct 31 02:07:42.111592 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:07:42.111570    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-183266/autores-model-183266-kserve-758f9bc869-tc88v"
Oct 31 02:07:42.498330 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:07:42.498294    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-autores-183266/autores-model-183266-kserve-758f9bc869-tc88v"]
Oct 31 02:07:42.500048 ip-10-0-133-197 kubenswrapper[4964]: W1031 02:07:42.500010    4964 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5c156010_30a3_434b_a027_305788f7ed3b.slice/crio-a5ef71f0fd288a7dee2337fca79e87861830129d96c8d3bbf032f9542671756d WatchSource:0}: Error finding container a5ef71f0fd288a7dee2337fca79e87861830129d96c8d3bbf032f9542671756d: Status 404 returned error can't find the container with id a5ef71f0fd288a7dee2337fca79e87861830129d96c8d3bbf032f9542671756d
Oct 31 02:07:43.047509 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:07:43.047470    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-183266/autores-model-183266-kserve-758f9bc869-tc88v" event={"ID":"5c156010-30a3-434b-a027-305788f7ed3b","Type":"ContainerStarted","Data":"6cd3dbd08ec827dfca8b23c511e7a552729deedc91c122e687bc4d7986ad4f0b"}
Oct 31 02:07:43.047668 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:07:43.047521    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-183266/autores-model-183266-kserve-758f9bc869-tc88v" event={"ID":"5c156010-30a3-434b-a027-305788f7ed3b","Type":"ContainerStarted","Data":"a5ef71f0fd288a7dee2337fca79e87861830129d96c8d3bbf032f9542671756d"}
Oct 31 02:07:43.112123 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:07:43.112084    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-autores-183266/autores-model-183266-kserve-758f9bc869-tc88v"
Oct 31 02:07:43.162550 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:07:43.162498    4964 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-autores-183266/autores-model-183266-kserve-758f9bc869-tc88v" podStartSLOduration=2.162483498 podStartE2EDuration="2.162483498s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 02:07:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 02:07:43.161403395 +0000 UTC m=+3831.637969772" watchObservedRunningTime="2026-10-31 02:07:43.162483498 +0000 UTC m=+3831.639049914"
Oct 31 02:07:46.007735 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:07:46.007701    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 02:07:46.893555 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:07:46.893511    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 02:07:47.079952 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:07:47.079917    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 02:07:47.260459 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:07:47.260412    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 02:07:51.200813 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:07:51.200775    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 02:07:52.112265 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:07:52.112220    4964 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-autores-183266/autores-model-183266-kserve-758f9bc869-tc88v"
Oct 31 02:07:52.127552 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:07:52.127513    4964 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-autores-183266/autores-model-183266-kserve-758f9bc869-tc88v"
Oct 31 02:07:53.128724 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:07:53.128691    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-autores-183266/autores-model-183266-kserve-758f9bc869-tc88v"
Oct 31 02:07:56.312368 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:07:56.311868    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 02:07:59.307023 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:07:59.306983    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 02:07:59.957391 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:07:59.957354    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 02:08:00.145311 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:08:00.145264    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 02:08:01.094419 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:08:01.094379    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 02:08:03.294621 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:08:03.294588    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 02:08:05.904647 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:08:05.904613    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 02:08:06.821760 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:08:06.821727    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 02:08:06.991258 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:08:06.991222    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 02:08:07.164383 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:08:07.164257    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 02:08:10.694917 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:08:10.694885    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 02:08:11.239857 ip-10-0-133-197 kubenswrapper[4964]: E1031 02:08:11.239816    4964 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q56x5" containerName="extractor"
Oct 31 02:08:11.562005 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:08:11.561902    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 02:08:12.860485 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:08:12.860448    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 02:08:15.439381 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:08:15.439348    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 02:08:18.896225 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:08:18.896188    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 02:08:19.075138 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:08:19.075101    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 02:08:20.263559 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:08:20.263520    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 02:08:22.623310 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:08:22.623273    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 02:08:24.988930 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:08:24.988897    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 02:08:29.136744 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:08:29.136702    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 02:08:30.661330 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:08:30.661267    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 02:08:30.879712 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:08:30.879671    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 02:08:31.047967 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:08:31.047927    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 02:08:31.610422 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:08:31.610384    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 02:08:36.143914 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:08:36.143881    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 02:08:38.244335 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:08:38.244288    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 02:08:38.457938 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:08:38.457900    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 02:08:38.637217 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:08:38.637116    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 02:08:39.137937 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:08:39.137901    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 02:08:40.692674 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:08:40.692639    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 02:08:41.238566 ip-10-0-133-197 kubenswrapper[4964]: E1031 02:08:41.238521    4964 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q56x5" containerName="extractor"
Oct 31 02:08:42.286893 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:08:42.286844    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 02:08:42.509221 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:08:42.509183    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 02:08:42.699286 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:08:42.699189    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 02:08:43.254214 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:08:43.254172    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 02:08:44.262303 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:08:44.262265    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 02:08:47.696200 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:08:47.696164    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 02:08:48.968563 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:08:48.968529    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 02:08:49.291140 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:08:49.291105    4964 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-n5bfg"]
Oct 31 02:08:49.291373 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:08:49.291338    4964 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-n5bfg" podUID="597e819f-88c8-4061-8c6a-9adf9a909d30" containerName="istio-proxy" containerID="cri-o://845653a94de03bc46729c960dabb67faf546b7e333c275cc86e818a380da79ef" gracePeriod=30
Oct 31 02:08:49.916308 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:08:49.916273    4964 patch_prober.go:29] interesting pod/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-n5bfg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 02:08:49.916638 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:08:49.916357    4964 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-n5bfg" podUID="597e819f-88c8-4061-8c6a-9adf9a909d30" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 02:08:51.311744 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:08:51.311714    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 02:08:51.490456 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:08:51.490418    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 02:08:51.655986 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:08:51.655892    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 02:08:51.872424 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:08:51.872383    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 02:08:52.376524 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:08:52.376485    4964 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-9257n_f0c0ada7-7688-4dc2-97fc-813e4d0c4ef3/console-operator/2.log"
Oct 31 02:08:52.391389 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:08:52.391364    4964 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-9257n_f0c0ada7-7688-4dc2-97fc-813e4d0c4ef3/console-operator/2.log"
Oct 31 02:08:53.403687 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:08:53.403649    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 02:08:53.630130 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:08:53.630096    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 02:08:53.795264 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:08:53.795227    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 02:08:54.382136 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:08:54.382098    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 02:08:54.551201 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:08:54.551170    4964 generic.go:356] "Generic (PLEG): container finished" podID="597e819f-88c8-4061-8c6a-9adf9a909d30" containerID="845653a94de03bc46729c960dabb67faf546b7e333c275cc86e818a380da79ef" exitCode=0
Oct 31 02:08:54.551669 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:08:54.551242    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-n5bfg" event={"ID":"597e819f-88c8-4061-8c6a-9adf9a909d30","Type":"ContainerDied","Data":"845653a94de03bc46729c960dabb67faf546b7e333c275cc86e818a380da79ef"}
Oct 31 02:08:54.581824 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:08:54.581796    4964 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-n5bfg"
Oct 31 02:08:54.664524 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:08:54.664487    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/597e819f-88c8-4061-8c6a-9adf9a909d30-kube-api-access-wlcrh\" (UniqueName: \"kubernetes.io/projected/597e819f-88c8-4061-8c6a-9adf9a909d30-kube-api-access-wlcrh\") pod \"597e819f-88c8-4061-8c6a-9adf9a909d30\" (UID: \"597e819f-88c8-4061-8c6a-9adf9a909d30\") "
Oct 31 02:08:54.664740 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:08:54.664555    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/597e819f-88c8-4061-8c6a-9adf9a909d30-istio-token\" (UniqueName: \"kubernetes.io/projected/597e819f-88c8-4061-8c6a-9adf9a909d30-istio-token\") pod \"597e819f-88c8-4061-8c6a-9adf9a909d30\" (UID: \"597e819f-88c8-4061-8c6a-9adf9a909d30\") "
Oct 31 02:08:54.664740 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:08:54.664607    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/597e819f-88c8-4061-8c6a-9adf9a909d30-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/597e819f-88c8-4061-8c6a-9adf9a909d30-workload-socket\") pod \"597e819f-88c8-4061-8c6a-9adf9a909d30\" (UID: \"597e819f-88c8-4061-8c6a-9adf9a909d30\") "
Oct 31 02:08:54.664740 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:08:54.664638    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/597e819f-88c8-4061-8c6a-9adf9a909d30-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/597e819f-88c8-4061-8c6a-9adf9a909d30-credential-socket\") pod \"597e819f-88c8-4061-8c6a-9adf9a909d30\" (UID: \"597e819f-88c8-4061-8c6a-9adf9a909d30\") "
Oct 31 02:08:54.664740 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:08:54.664735    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/597e819f-88c8-4061-8c6a-9adf9a909d30-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/597e819f-88c8-4061-8c6a-9adf9a909d30-workload-certs\") pod \"597e819f-88c8-4061-8c6a-9adf9a909d30\" (UID: \"597e819f-88c8-4061-8c6a-9adf9a909d30\") "
Oct 31 02:08:54.665017 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:08:54.664769    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/597e819f-88c8-4061-8c6a-9adf9a909d30-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/597e819f-88c8-4061-8c6a-9adf9a909d30-istiod-ca-cert\") pod \"597e819f-88c8-4061-8c6a-9adf9a909d30\" (UID: \"597e819f-88c8-4061-8c6a-9adf9a909d30\") "
Oct 31 02:08:54.665017 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:08:54.664921    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/597e819f-88c8-4061-8c6a-9adf9a909d30-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/597e819f-88c8-4061-8c6a-9adf9a909d30-istio-podinfo\") pod \"597e819f-88c8-4061-8c6a-9adf9a909d30\" (UID: \"597e819f-88c8-4061-8c6a-9adf9a909d30\") "
Oct 31 02:08:54.665017 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:08:54.664964    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/597e819f-88c8-4061-8c6a-9adf9a909d30-workload-socket" pod "597e819f-88c8-4061-8c6a-9adf9a909d30" (UID: "597e819f-88c8-4061-8c6a-9adf9a909d30"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:08:54.665017 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:08:54.664978    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/597e819f-88c8-4061-8c6a-9adf9a909d30-credential-socket" pod "597e819f-88c8-4061-8c6a-9adf9a909d30" (UID: "597e819f-88c8-4061-8c6a-9adf9a909d30"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:08:54.665017 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:08:54.664998    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/597e819f-88c8-4061-8c6a-9adf9a909d30-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/597e819f-88c8-4061-8c6a-9adf9a909d30-istio-envoy\") pod \"597e819f-88c8-4061-8c6a-9adf9a909d30\" (UID: \"597e819f-88c8-4061-8c6a-9adf9a909d30\") "
Oct 31 02:08:54.665263 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:08:54.665053    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/597e819f-88c8-4061-8c6a-9adf9a909d30-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/597e819f-88c8-4061-8c6a-9adf9a909d30-istio-data\") pod \"597e819f-88c8-4061-8c6a-9adf9a909d30\" (UID: \"597e819f-88c8-4061-8c6a-9adf9a909d30\") "
Oct 31 02:08:54.665263 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:08:54.665142    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/597e819f-88c8-4061-8c6a-9adf9a909d30-workload-certs" pod "597e819f-88c8-4061-8c6a-9adf9a909d30" (UID: "597e819f-88c8-4061-8c6a-9adf9a909d30"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:08:54.665263 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:08:54.665232    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/597e819f-88c8-4061-8c6a-9adf9a909d30-istiod-ca-cert" pod "597e819f-88c8-4061-8c6a-9adf9a909d30" (UID: "597e819f-88c8-4061-8c6a-9adf9a909d30"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 02:08:54.665496 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:08:54.665445    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/597e819f-88c8-4061-8c6a-9adf9a909d30-istio-data" pod "597e819f-88c8-4061-8c6a-9adf9a909d30" (UID: "597e819f-88c8-4061-8c6a-9adf9a909d30"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:08:54.665960 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:08:54.665701    4964 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/597e819f-88c8-4061-8c6a-9adf9a909d30-workload-socket\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:08:54.665960 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:08:54.665842    4964 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/597e819f-88c8-4061-8c6a-9adf9a909d30-credential-socket\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:08:54.665960 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:08:54.665907    4964 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/597e819f-88c8-4061-8c6a-9adf9a909d30-workload-certs\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:08:54.665960 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:08:54.665920    4964 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/597e819f-88c8-4061-8c6a-9adf9a909d30-istiod-ca-cert\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:08:54.665960 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:08:54.665934    4964 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/597e819f-88c8-4061-8c6a-9adf9a909d30-istio-data\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:08:54.667232 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:08:54.667205    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/597e819f-88c8-4061-8c6a-9adf9a909d30-istio-token" pod "597e819f-88c8-4061-8c6a-9adf9a909d30" (UID: "597e819f-88c8-4061-8c6a-9adf9a909d30"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:08:54.667384 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:08:54.667354    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/597e819f-88c8-4061-8c6a-9adf9a909d30-istio-podinfo" pod "597e819f-88c8-4061-8c6a-9adf9a909d30" (UID: "597e819f-88c8-4061-8c6a-9adf9a909d30"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 02:08:54.667625 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:08:54.667601    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/597e819f-88c8-4061-8c6a-9adf9a909d30-kube-api-access-wlcrh" pod "597e819f-88c8-4061-8c6a-9adf9a909d30" (UID: "597e819f-88c8-4061-8c6a-9adf9a909d30"). InnerVolumeSpecName "kube-api-access-wlcrh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:08:54.667702 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:08:54.667641    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/597e819f-88c8-4061-8c6a-9adf9a909d30-istio-envoy" pod "597e819f-88c8-4061-8c6a-9adf9a909d30" (UID: "597e819f-88c8-4061-8c6a-9adf9a909d30"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:08:54.768947 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:08:54.768905    4964 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/597e819f-88c8-4061-8c6a-9adf9a909d30-istio-token\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:08:54.768947 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:08:54.768946    4964 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/597e819f-88c8-4061-8c6a-9adf9a909d30-istio-podinfo\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:08:54.769237 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:08:54.768963    4964 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/597e819f-88c8-4061-8c6a-9adf9a909d30-istio-envoy\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:08:54.769237 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:08:54.768979    4964 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wlcrh\" (UniqueName: \"kubernetes.io/projected/597e819f-88c8-4061-8c6a-9adf9a909d30-kube-api-access-wlcrh\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:08:55.559587 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:08:55.559552    4964 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-n5bfg"
Oct 31 02:08:55.559587 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:08:55.559563    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-n5bfg" event={"ID":"597e819f-88c8-4061-8c6a-9adf9a909d30","Type":"ContainerDied","Data":"7382473ed4ecbf14e6f1ff122eab7e8c69f584f816ea88c1cd39a4ebdfafd20a"}
Oct 31 02:08:55.560168 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:08:55.559649    4964 scope.go:122] "RemoveContainer" containerID="845653a94de03bc46729c960dabb67faf546b7e333c275cc86e818a380da79ef"
Oct 31 02:08:55.791508 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:08:55.791469    4964 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-n5bfg"]
Oct 31 02:08:55.823294 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:08:55.823204    4964 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-n5bfg"]
Oct 31 02:08:57.776975 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:08:57.776943    4964 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="597e819f-88c8-4061-8c6a-9adf9a909d30" path="/var/lib/kubelet/pods/597e819f-88c8-4061-8c6a-9adf9a909d30/volumes"
Oct 31 02:08:58.679116 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:08:58.679083    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 02:08:59.654039 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:08:59.654004    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 02:09:00.903628 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:09:00.903533    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 02:09:01.091353 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:09:01.091275    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 02:09:01.179474 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:09:01.179387    4964 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-c4d438-openshift-default-6d9d56f475-c995r"]
Oct 31 02:09:01.179674 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:09:01.179632    4964 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w3-c4d438-openshift-default-6d9d56f475-c995r" podUID="1ec66b87-7d07-45fd-8469-e1bd2c83f651" containerName="istio-proxy" containerID="cri-o://4b04497f96486b8416976008199fb69286b68711327136c14cb456dc1ea3e4fd" gracePeriod=30
Oct 31 02:09:01.266728 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:09:01.266686    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 02:09:03.034688 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:09:03.034644    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 02:09:03.633196 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:09:03.633155    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 02:09:03.873572 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:09:03.873539    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 02:09:03.893031 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:09:03.892950    4964 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-autores-183266/autores-model-183266-kserve-758f9bc869-tc88v"]
Oct 31 02:09:04.105051 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:09:04.105015    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 02:09:06.254810 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:09:06.254771    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 02:09:06.472751 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:09:06.472724    4964 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-c4d438-openshift-default-6d9d56f475-c995r"
Oct 31 02:09:06.479661 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:09:06.479630    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 02:09:06.624209 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:09:06.624175    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1ec66b87-7d07-45fd-8469-e1bd2c83f651-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ec66b87-7d07-45fd-8469-e1bd2c83f651-workload-socket\") pod \"1ec66b87-7d07-45fd-8469-e1bd2c83f651\" (UID: \"1ec66b87-7d07-45fd-8469-e1bd2c83f651\") "
Oct 31 02:09:06.624443 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:09:06.624217    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1ec66b87-7d07-45fd-8469-e1bd2c83f651-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1ec66b87-7d07-45fd-8469-e1bd2c83f651-workload-certs\") pod \"1ec66b87-7d07-45fd-8469-e1bd2c83f651\" (UID: \"1ec66b87-7d07-45fd-8469-e1bd2c83f651\") "
Oct 31 02:09:06.624443 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:09:06.624264    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1ec66b87-7d07-45fd-8469-e1bd2c83f651-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1ec66b87-7d07-45fd-8469-e1bd2c83f651-istio-podinfo\") pod \"1ec66b87-7d07-45fd-8469-e1bd2c83f651\" (UID: \"1ec66b87-7d07-45fd-8469-e1bd2c83f651\") "
Oct 31 02:09:06.624443 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:09:06.624375    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1ec66b87-7d07-45fd-8469-e1bd2c83f651-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1ec66b87-7d07-45fd-8469-e1bd2c83f651-istiod-ca-cert\") pod \"1ec66b87-7d07-45fd-8469-e1bd2c83f651\" (UID: \"1ec66b87-7d07-45fd-8469-e1bd2c83f651\") "
Oct 31 02:09:06.624443 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:09:06.624418    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1ec66b87-7d07-45fd-8469-e1bd2c83f651-istio-token\" (UniqueName: \"kubernetes.io/projected/1ec66b87-7d07-45fd-8469-e1bd2c83f651-istio-token\") pod \"1ec66b87-7d07-45fd-8469-e1bd2c83f651\" (UID: \"1ec66b87-7d07-45fd-8469-e1bd2c83f651\") "
Oct 31 02:09:06.624689 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:09:06.624448    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1ec66b87-7d07-45fd-8469-e1bd2c83f651-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1ec66b87-7d07-45fd-8469-e1bd2c83f651-istio-data\") pod \"1ec66b87-7d07-45fd-8469-e1bd2c83f651\" (UID: \"1ec66b87-7d07-45fd-8469-e1bd2c83f651\") "
Oct 31 02:09:06.624689 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:09:06.624505    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1ec66b87-7d07-45fd-8469-e1bd2c83f651-kube-api-access-8pwdt\" (UniqueName: \"kubernetes.io/projected/1ec66b87-7d07-45fd-8469-e1bd2c83f651-kube-api-access-8pwdt\") pod \"1ec66b87-7d07-45fd-8469-e1bd2c83f651\" (UID: \"1ec66b87-7d07-45fd-8469-e1bd2c83f651\") "
Oct 31 02:09:06.624689 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:09:06.624513    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ec66b87-7d07-45fd-8469-e1bd2c83f651-workload-socket" pod "1ec66b87-7d07-45fd-8469-e1bd2c83f651" (UID: "1ec66b87-7d07-45fd-8469-e1bd2c83f651"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:09:06.624689 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:09:06.624551    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1ec66b87-7d07-45fd-8469-e1bd2c83f651-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ec66b87-7d07-45fd-8469-e1bd2c83f651-credential-socket\") pod \"1ec66b87-7d07-45fd-8469-e1bd2c83f651\" (UID: \"1ec66b87-7d07-45fd-8469-e1bd2c83f651\") "
Oct 31 02:09:06.624689 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:09:06.624586    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1ec66b87-7d07-45fd-8469-e1bd2c83f651-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1ec66b87-7d07-45fd-8469-e1bd2c83f651-istio-envoy\") pod \"1ec66b87-7d07-45fd-8469-e1bd2c83f651\" (UID: \"1ec66b87-7d07-45fd-8469-e1bd2c83f651\") "
Oct 31 02:09:06.624689 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:09:06.624646    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ec66b87-7d07-45fd-8469-e1bd2c83f651-workload-certs" pod "1ec66b87-7d07-45fd-8469-e1bd2c83f651" (UID: "1ec66b87-7d07-45fd-8469-e1bd2c83f651"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:09:06.625078 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:09:06.624800    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ec66b87-7d07-45fd-8469-e1bd2c83f651-credential-socket" pod "1ec66b87-7d07-45fd-8469-e1bd2c83f651" (UID: "1ec66b87-7d07-45fd-8469-e1bd2c83f651"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:09:06.625078 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:09:06.624923    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ec66b87-7d07-45fd-8469-e1bd2c83f651-istio-data" pod "1ec66b87-7d07-45fd-8469-e1bd2c83f651" (UID: "1ec66b87-7d07-45fd-8469-e1bd2c83f651"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:09:06.625078 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:09:06.624954    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1ec66b87-7d07-45fd-8469-e1bd2c83f651-istiod-ca-cert" pod "1ec66b87-7d07-45fd-8469-e1bd2c83f651" (UID: "1ec66b87-7d07-45fd-8469-e1bd2c83f651"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 02:09:06.625197 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:09:06.625160    4964 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ec66b87-7d07-45fd-8469-e1bd2c83f651-workload-socket\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:09:06.625197 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:09:06.625181    4964 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1ec66b87-7d07-45fd-8469-e1bd2c83f651-workload-certs\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:09:06.625331 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:09:06.625195    4964 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1ec66b87-7d07-45fd-8469-e1bd2c83f651-istiod-ca-cert\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:09:06.625331 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:09:06.625209    4964 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1ec66b87-7d07-45fd-8469-e1bd2c83f651-istio-data\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:09:06.625331 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:09:06.625223    4964 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ec66b87-7d07-45fd-8469-e1bd2c83f651-credential-socket\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:09:06.627217 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:09:06.627170    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ec66b87-7d07-45fd-8469-e1bd2c83f651-istio-token" pod "1ec66b87-7d07-45fd-8469-e1bd2c83f651" (UID: "1ec66b87-7d07-45fd-8469-e1bd2c83f651"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:09:06.627217 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:09:06.627201    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ec66b87-7d07-45fd-8469-e1bd2c83f651-istio-envoy" pod "1ec66b87-7d07-45fd-8469-e1bd2c83f651" (UID: "1ec66b87-7d07-45fd-8469-e1bd2c83f651"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:09:06.627470 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:09:06.627397    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1ec66b87-7d07-45fd-8469-e1bd2c83f651-istio-podinfo" pod "1ec66b87-7d07-45fd-8469-e1bd2c83f651" (UID: "1ec66b87-7d07-45fd-8469-e1bd2c83f651"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 02:09:06.627543 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:09:06.627524    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ec66b87-7d07-45fd-8469-e1bd2c83f651-kube-api-access-8pwdt" pod "1ec66b87-7d07-45fd-8469-e1bd2c83f651" (UID: "1ec66b87-7d07-45fd-8469-e1bd2c83f651"). InnerVolumeSpecName "kube-api-access-8pwdt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:09:06.659913 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:09:06.659878    4964 generic.go:356] "Generic (PLEG): container finished" podID="1ec66b87-7d07-45fd-8469-e1bd2c83f651" containerID="4b04497f96486b8416976008199fb69286b68711327136c14cb456dc1ea3e4fd" exitCode=0
Oct 31 02:09:06.660095 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:09:06.659956    4964 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-c4d438-openshift-default-6d9d56f475-c995r"
Oct 31 02:09:06.660095 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:09:06.659955    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-c4d438-openshift-default-6d9d56f475-c995r" event={"ID":"1ec66b87-7d07-45fd-8469-e1bd2c83f651","Type":"ContainerDied","Data":"4b04497f96486b8416976008199fb69286b68711327136c14cb456dc1ea3e4fd"}
Oct 31 02:09:06.660095 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:09:06.660087    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-c4d438-openshift-default-6d9d56f475-c995r" event={"ID":"1ec66b87-7d07-45fd-8469-e1bd2c83f651","Type":"ContainerDied","Data":"93d5d2e82295d3e412063cd592affd5f8b9b5598ae8f38b3957c04eebca75112"}
Oct 31 02:09:06.660275 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:09:06.660112    4964 scope.go:122] "RemoveContainer" containerID="4b04497f96486b8416976008199fb69286b68711327136c14cb456dc1ea3e4fd"
Oct 31 02:09:06.671053 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:09:06.671032    4964 scope.go:122] "RemoveContainer" containerID="224f2098b2772bf59ef5570eb3e4e2d7efb5d0551b47d9ed009178abe42408bf"
Oct 31 02:09:06.681253 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:09:06.681229    4964 scope.go:122] "RemoveContainer" containerID="4b04497f96486b8416976008199fb69286b68711327136c14cb456dc1ea3e4fd"
Oct 31 02:09:06.681603 ip-10-0-133-197 kubenswrapper[4964]: E1031 02:09:06.681575    4964 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4b04497f96486b8416976008199fb69286b68711327136c14cb456dc1ea3e4fd\": container with ID starting with 4b04497f96486b8416976008199fb69286b68711327136c14cb456dc1ea3e4fd not found: ID does not exist" containerID="4b04497f96486b8416976008199fb69286b68711327136c14cb456dc1ea3e4fd"
Oct 31 02:09:06.681699 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:09:06.681618    4964 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4b04497f96486b8416976008199fb69286b68711327136c14cb456dc1ea3e4fd"} err="failed to get container status \"4b04497f96486b8416976008199fb69286b68711327136c14cb456dc1ea3e4fd\": rpc error: code = NotFound desc = could not find container \"4b04497f96486b8416976008199fb69286b68711327136c14cb456dc1ea3e4fd\": container with ID starting with 4b04497f96486b8416976008199fb69286b68711327136c14cb456dc1ea3e4fd not found: ID does not exist"
Oct 31 02:09:06.681699 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:09:06.681638    4964 scope.go:122] "RemoveContainer" containerID="224f2098b2772bf59ef5570eb3e4e2d7efb5d0551b47d9ed009178abe42408bf"
Oct 31 02:09:06.681911 ip-10-0-133-197 kubenswrapper[4964]: E1031 02:09:06.681895    4964 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"224f2098b2772bf59ef5570eb3e4e2d7efb5d0551b47d9ed009178abe42408bf\": container with ID starting with 224f2098b2772bf59ef5570eb3e4e2d7efb5d0551b47d9ed009178abe42408bf not found: ID does not exist" containerID="224f2098b2772bf59ef5570eb3e4e2d7efb5d0551b47d9ed009178abe42408bf"
Oct 31 02:09:06.681963 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:09:06.681917    4964 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"224f2098b2772bf59ef5570eb3e4e2d7efb5d0551b47d9ed009178abe42408bf"} err="failed to get container status \"224f2098b2772bf59ef5570eb3e4e2d7efb5d0551b47d9ed009178abe42408bf\": rpc error: code = NotFound desc = could not find container \"224f2098b2772bf59ef5570eb3e4e2d7efb5d0551b47d9ed009178abe42408bf\": container with ID starting with 224f2098b2772bf59ef5570eb3e4e2d7efb5d0551b47d9ed009178abe42408bf not found: ID does not exist"
Oct 31 02:09:06.726906 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:09:06.726835    4964 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8pwdt\" (UniqueName: \"kubernetes.io/projected/1ec66b87-7d07-45fd-8469-e1bd2c83f651-kube-api-access-8pwdt\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:09:06.726906 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:09:06.726864    4964 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1ec66b87-7d07-45fd-8469-e1bd2c83f651-istio-envoy\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:09:06.726906 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:09:06.726881    4964 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1ec66b87-7d07-45fd-8469-e1bd2c83f651-istio-podinfo\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:09:06.726906 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:09:06.726890    4964 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1ec66b87-7d07-45fd-8469-e1bd2c83f651-istio-token\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:09:06.824529 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:09:06.824467    4964 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-c4d438-openshift-default-6d9d56f475-c995r"]
Oct 31 02:09:06.827884 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:09:06.827857    4964 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-c4d438-openshift-default-6d9d56f475-c995r"]
Oct 31 02:09:07.777713 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:09:07.777681    4964 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1ec66b87-7d07-45fd-8469-e1bd2c83f651" path="/var/lib/kubelet/pods/1ec66b87-7d07-45fd-8469-e1bd2c83f651/volumes"
Oct 31 02:09:08.868566 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:09:08.868533    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 02:09:10.998840 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:09:10.998792    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 02:09:11.177042 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:09:11.177007    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 02:09:11.239094 ip-10-0-133-197 kubenswrapper[4964]: E1031 02:09:11.239054    4964 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q56x5" containerName="extractor"
Oct 31 02:09:11.742272 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:09:11.742225    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 02:09:11.919038 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:09:11.919003    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 02:09:13.915380 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:09:13.915298    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 02:09:16.655754 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:09:16.655713    4964 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-a8e157/distinct-2-w3-a8e157-kserve-9ddd45d6-ng7j7"]
Oct 31 02:09:16.706593 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:09:16.706546    4964 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-a8e157/distinct-w3-a8e157-kserve-549559dd5c-t29tw"]
Oct 31 02:09:16.850057 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:09:16.850018    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 02:09:17.023852 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:09:17.023818    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 02:09:18.065688 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:09:18.065653    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 02:09:18.915272 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:09:18.915222    4964 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-autores-183266/autores-model-183266-kserve-758f9bc869-tc88v" podUID="5c156010-30a3-434b-a027-305788f7ed3b" containerName="main" containerID="cri-o://6cd3dbd08ec827dfca8b23c511e7a552729deedc91c122e687bc4d7986ad4f0b" gracePeriod=45
Oct 31 02:09:19.200062 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:09:19.200034    4964 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-183266/autores-model-183266-kserve-758f9bc869-tc88v"
Oct 31 02:09:19.264219 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:09:19.264191    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c156010-30a3-434b-a027-305788f7ed3b-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5c156010-30a3-434b-a027-305788f7ed3b-tmp-dir\") pod \"5c156010-30a3-434b-a027-305788f7ed3b\" (UID: \"5c156010-30a3-434b-a027-305788f7ed3b\") "
Oct 31 02:09:19.264446 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:09:19.264242    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5c156010-30a3-434b-a027-305788f7ed3b-tls-certs\" (UniqueName: \"kubernetes.io/secret/5c156010-30a3-434b-a027-305788f7ed3b-tls-certs\") pod \"5c156010-30a3-434b-a027-305788f7ed3b\" (UID: \"5c156010-30a3-434b-a027-305788f7ed3b\") "
Oct 31 02:09:19.264446 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:09:19.264282    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5c156010-30a3-434b-a027-305788f7ed3b-kube-api-access-wk4pg\" (UniqueName: \"kubernetes.io/projected/5c156010-30a3-434b-a027-305788f7ed3b-kube-api-access-wk4pg\") pod \"5c156010-30a3-434b-a027-305788f7ed3b\" (UID: \"5c156010-30a3-434b-a027-305788f7ed3b\") "
Oct 31 02:09:19.264446 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:09:19.264341    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c156010-30a3-434b-a027-305788f7ed3b-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5c156010-30a3-434b-a027-305788f7ed3b-model-cache\") pod \"5c156010-30a3-434b-a027-305788f7ed3b\" (UID: \"5c156010-30a3-434b-a027-305788f7ed3b\") "
Oct 31 02:09:19.264446 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:09:19.264403    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c156010-30a3-434b-a027-305788f7ed3b-dshm\" (UniqueName: \"kubernetes.io/empty-dir/5c156010-30a3-434b-a027-305788f7ed3b-dshm\") pod \"5c156010-30a3-434b-a027-305788f7ed3b\" (UID: \"5c156010-30a3-434b-a027-305788f7ed3b\") "
Oct 31 02:09:19.264675 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:09:19.264480    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c156010-30a3-434b-a027-305788f7ed3b-home\" (UniqueName: \"kubernetes.io/empty-dir/5c156010-30a3-434b-a027-305788f7ed3b-home\") pod \"5c156010-30a3-434b-a027-305788f7ed3b\" (UID: \"5c156010-30a3-434b-a027-305788f7ed3b\") "
Oct 31 02:09:19.264675 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:09:19.264512    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c156010-30a3-434b-a027-305788f7ed3b-tmp-dir" pod "5c156010-30a3-434b-a027-305788f7ed3b" (UID: "5c156010-30a3-434b-a027-305788f7ed3b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:09:19.264675 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:09:19.264622    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c156010-30a3-434b-a027-305788f7ed3b-model-cache" pod "5c156010-30a3-434b-a027-305788f7ed3b" (UID: "5c156010-30a3-434b-a027-305788f7ed3b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:09:19.264823 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:09:19.264796    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c156010-30a3-434b-a027-305788f7ed3b-home" pod "5c156010-30a3-434b-a027-305788f7ed3b" (UID: "5c156010-30a3-434b-a027-305788f7ed3b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:09:19.264932 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:09:19.264914    4964 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5c156010-30a3-434b-a027-305788f7ed3b-tmp-dir\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:09:19.264998 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:09:19.264938    4964 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5c156010-30a3-434b-a027-305788f7ed3b-model-cache\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:09:19.264998 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:09:19.264952    4964 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5c156010-30a3-434b-a027-305788f7ed3b-home\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:09:19.267094 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:09:19.267055    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c156010-30a3-434b-a027-305788f7ed3b-kube-api-access-wk4pg" pod "5c156010-30a3-434b-a027-305788f7ed3b" (UID: "5c156010-30a3-434b-a027-305788f7ed3b"). InnerVolumeSpecName "kube-api-access-wk4pg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:09:19.267266 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:09:19.267246    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5c156010-30a3-434b-a027-305788f7ed3b-tls-certs" pod "5c156010-30a3-434b-a027-305788f7ed3b" (UID: "5c156010-30a3-434b-a027-305788f7ed3b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 02:09:19.267417 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:09:19.267392    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c156010-30a3-434b-a027-305788f7ed3b-dshm" pod "5c156010-30a3-434b-a027-305788f7ed3b" (UID: "5c156010-30a3-434b-a027-305788f7ed3b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:09:19.307555 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:09:19.307437    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 02:09:19.365701 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:09:19.365664    4964 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5c156010-30a3-434b-a027-305788f7ed3b-tls-certs\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:09:19.365701 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:09:19.365697    4964 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wk4pg\" (UniqueName: \"kubernetes.io/projected/5c156010-30a3-434b-a027-305788f7ed3b-kube-api-access-wk4pg\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:09:19.365915 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:09:19.365710    4964 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5c156010-30a3-434b-a027-305788f7ed3b-dshm\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:09:19.755457 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:09:19.755418    4964 generic.go:356] "Generic (PLEG): container finished" podID="5c156010-30a3-434b-a027-305788f7ed3b" containerID="6cd3dbd08ec827dfca8b23c511e7a552729deedc91c122e687bc4d7986ad4f0b" exitCode=0
Oct 31 02:09:19.755655 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:09:19.755466    4964 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-183266/autores-model-183266-kserve-758f9bc869-tc88v"
Oct 31 02:09:19.755655 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:09:19.755503    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-183266/autores-model-183266-kserve-758f9bc869-tc88v" event={"ID":"5c156010-30a3-434b-a027-305788f7ed3b","Type":"ContainerDied","Data":"6cd3dbd08ec827dfca8b23c511e7a552729deedc91c122e687bc4d7986ad4f0b"}
Oct 31 02:09:19.755655 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:09:19.755545    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-183266/autores-model-183266-kserve-758f9bc869-tc88v" event={"ID":"5c156010-30a3-434b-a027-305788f7ed3b","Type":"ContainerDied","Data":"a5ef71f0fd288a7dee2337fca79e87861830129d96c8d3bbf032f9542671756d"}
Oct 31 02:09:19.755655 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:09:19.755568    4964 scope.go:122] "RemoveContainer" containerID="6cd3dbd08ec827dfca8b23c511e7a552729deedc91c122e687bc4d7986ad4f0b"
Oct 31 02:09:19.772983 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:09:19.772962    4964 scope.go:122] "RemoveContainer" containerID="6cd3dbd08ec827dfca8b23c511e7a552729deedc91c122e687bc4d7986ad4f0b"
Oct 31 02:09:19.773283 ip-10-0-133-197 kubenswrapper[4964]: E1031 02:09:19.773245    4964 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6cd3dbd08ec827dfca8b23c511e7a552729deedc91c122e687bc4d7986ad4f0b\": container with ID starting with 6cd3dbd08ec827dfca8b23c511e7a552729deedc91c122e687bc4d7986ad4f0b not found: ID does not exist" containerID="6cd3dbd08ec827dfca8b23c511e7a552729deedc91c122e687bc4d7986ad4f0b"
Oct 31 02:09:19.773412 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:09:19.773290    4964 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6cd3dbd08ec827dfca8b23c511e7a552729deedc91c122e687bc4d7986ad4f0b"} err="failed to get container status \"6cd3dbd08ec827dfca8b23c511e7a552729deedc91c122e687bc4d7986ad4f0b\": rpc error: code = NotFound desc = could not find container \"6cd3dbd08ec827dfca8b23c511e7a552729deedc91c122e687bc4d7986ad4f0b\": container with ID starting with 6cd3dbd08ec827dfca8b23c511e7a552729deedc91c122e687bc4d7986ad4f0b not found: ID does not exist"
Oct 31 02:09:19.919297 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:09:19.919260    4964 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-autores-183266/autores-model-183266-kserve-758f9bc869-tc88v"]
Oct 31 02:09:19.924048 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:09:19.924022    4964 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-autores-183266/autores-model-183266-kserve-758f9bc869-tc88v"]
Oct 31 02:09:20.130679 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:09:20.130592    4964 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-autores-183266/autores-model-183266-kserve-758f9bc869-tc88v" podUID="5c156010-30a3-434b-a027-305788f7ed3b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.192:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 02:09:21.776424 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:09:21.776388    4964 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5c156010-30a3-434b-a027-305788f7ed3b" path="/var/lib/kubelet/pods/5c156010-30a3-434b-a027-305788f7ed3b/volumes"
Oct 31 02:09:23.586499 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:09:23.586466    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 02:09:23.753338 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:09:23.753264    4964 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-a8e157-69ffccd74c-z9zls"]
Oct 31 02:09:23.860524 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:09:23.860434    4964 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-a8e157-5597f4b787-gvgv8"]
Oct 31 02:09:23.909962 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:09:23.909913    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 02:09:24.144566 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:09:24.144467    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 02:09:24.314489 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:09:24.314449    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 02:09:25.093303 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:09:25.093254    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 02:09:25.315858 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:09:25.315825    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 02:09:25.488255 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:09:25.488207    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 02:09:28.754518 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:09:28.754476    4964 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w3-a8e157-69ffccd74c-z9zls" podUID="04c0b568-8251-4888-902b-e46c9483d931" containerName="payload-processing" containerID="cri-o://f42e6fe1b20bd1dc35351757e16228ae4d7b8cd79084274a94a12adfd3b29d95" gracePeriod=25
Oct 31 02:09:28.861001 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:09:28.860959    4964 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-a8e157-5597f4b787-gvgv8" podUID="3aae9913-7c43-45b0-87c8-bf1da6bf9183" containerName="payload-pre-processing" containerID="cri-o://47627fe7ac47a10530c752db69066a5f7e04a9952e5cd181616361006c1a9f88" gracePeriod=25
Oct 31 02:09:30.835885 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:09:30.835848    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 02:09:31.677633 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:09:31.677584    4964 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-a8e157/distinct-2-w3-a8e157-kserve-9ddd45d6-ng7j7" podUID="bc7a71df-c523-4590-91df-cf8fdf3ed19e" containerName="main" containerID="cri-o://cfcbc7b773c61d1d43ce734aabc9a090f19a1794c12f9dac57a6821f7fef7361" gracePeriod=45
Oct 31 02:09:31.729088 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:09:31.729045    4964 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-a8e157/distinct-w3-a8e157-kserve-549559dd5c-t29tw" podUID="4b0fd818-20c2-4dd6-9846-f361c236dbd9" containerName="main" containerID="cri-o://0aad8fa9926e3f2093e2880f7603cac39c6ce924e62dc29d804a5c96f7cae641" gracePeriod=45
Oct 31 02:09:31.844440 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:09:31.844409    4964 generic.go:356] "Generic (PLEG): container finished" podID="bc7a71df-c523-4590-91df-cf8fdf3ed19e" containerID="cfcbc7b773c61d1d43ce734aabc9a090f19a1794c12f9dac57a6821f7fef7361" exitCode=0
Oct 31 02:09:31.844840 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:09:31.844476    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-a8e157/distinct-2-w3-a8e157-kserve-9ddd45d6-ng7j7" event={"ID":"bc7a71df-c523-4590-91df-cf8fdf3ed19e","Type":"ContainerDied","Data":"cfcbc7b773c61d1d43ce734aabc9a090f19a1794c12f9dac57a6821f7fef7361"}
Oct 31 02:09:32.023032 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:09:32.023005    4964 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-a8e157/distinct-2-w3-a8e157-kserve-9ddd45d6-ng7j7"
Oct 31 02:09:32.027675 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:09:32.027652    4964 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-a8e157/distinct-w3-a8e157-kserve-549559dd5c-t29tw"
Oct 31 02:09:32.110612 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:09:32.110568    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4b0fd818-20c2-4dd6-9846-f361c236dbd9-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4b0fd818-20c2-4dd6-9846-f361c236dbd9-model-cache\") pod \"4b0fd818-20c2-4dd6-9846-f361c236dbd9\" (UID: \"4b0fd818-20c2-4dd6-9846-f361c236dbd9\") "
Oct 31 02:09:32.110800 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:09:32.110624    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4b0fd818-20c2-4dd6-9846-f361c236dbd9-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4b0fd818-20c2-4dd6-9846-f361c236dbd9-tmp-dir\") pod \"4b0fd818-20c2-4dd6-9846-f361c236dbd9\" (UID: \"4b0fd818-20c2-4dd6-9846-f361c236dbd9\") "
Oct 31 02:09:32.110800 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:09:32.110661    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bc7a71df-c523-4590-91df-cf8fdf3ed19e-dshm\" (UniqueName: \"kubernetes.io/empty-dir/bc7a71df-c523-4590-91df-cf8fdf3ed19e-dshm\") pod \"bc7a71df-c523-4590-91df-cf8fdf3ed19e\" (UID: \"bc7a71df-c523-4590-91df-cf8fdf3ed19e\") "
Oct 31 02:09:32.110800 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:09:32.110712    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4b0fd818-20c2-4dd6-9846-f361c236dbd9-dshm\" (UniqueName: \"kubernetes.io/empty-dir/4b0fd818-20c2-4dd6-9846-f361c236dbd9-dshm\") pod \"4b0fd818-20c2-4dd6-9846-f361c236dbd9\" (UID: \"4b0fd818-20c2-4dd6-9846-f361c236dbd9\") "
Oct 31 02:09:32.110800 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:09:32.110774    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bc7a71df-c523-4590-91df-cf8fdf3ed19e-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bc7a71df-c523-4590-91df-cf8fdf3ed19e-model-cache\") pod \"bc7a71df-c523-4590-91df-cf8fdf3ed19e\" (UID: \"bc7a71df-c523-4590-91df-cf8fdf3ed19e\") "
Oct 31 02:09:32.111053 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:09:32.110823    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4b0fd818-20c2-4dd6-9846-f361c236dbd9-tls-certs\" (UniqueName: \"kubernetes.io/secret/4b0fd818-20c2-4dd6-9846-f361c236dbd9-tls-certs\") pod \"4b0fd818-20c2-4dd6-9846-f361c236dbd9\" (UID: \"4b0fd818-20c2-4dd6-9846-f361c236dbd9\") "
Oct 31 02:09:32.111053 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:09:32.110859    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4b0fd818-20c2-4dd6-9846-f361c236dbd9-home\" (UniqueName: \"kubernetes.io/empty-dir/4b0fd818-20c2-4dd6-9846-f361c236dbd9-home\") pod \"4b0fd818-20c2-4dd6-9846-f361c236dbd9\" (UID: \"4b0fd818-20c2-4dd6-9846-f361c236dbd9\") "
Oct 31 02:09:32.111053 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:09:32.110909    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b0fd818-20c2-4dd6-9846-f361c236dbd9-model-cache" pod "4b0fd818-20c2-4dd6-9846-f361c236dbd9" (UID: "4b0fd818-20c2-4dd6-9846-f361c236dbd9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:09:32.111053 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:09:32.110962    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b0fd818-20c2-4dd6-9846-f361c236dbd9-tmp-dir" pod "4b0fd818-20c2-4dd6-9846-f361c236dbd9" (UID: "4b0fd818-20c2-4dd6-9846-f361c236dbd9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:09:32.111282 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:09:32.111140    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc7a71df-c523-4590-91df-cf8fdf3ed19e-model-cache" pod "bc7a71df-c523-4590-91df-cf8fdf3ed19e" (UID: "bc7a71df-c523-4590-91df-cf8fdf3ed19e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:09:32.111282 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:09:32.111239    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b0fd818-20c2-4dd6-9846-f361c236dbd9-home" pod "4b0fd818-20c2-4dd6-9846-f361c236dbd9" (UID: "4b0fd818-20c2-4dd6-9846-f361c236dbd9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:09:32.111441 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:09:32.111418    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4b0fd818-20c2-4dd6-9846-f361c236dbd9-kube-api-access-nlhpz\" (UniqueName: \"kubernetes.io/projected/4b0fd818-20c2-4dd6-9846-f361c236dbd9-kube-api-access-nlhpz\") pod \"4b0fd818-20c2-4dd6-9846-f361c236dbd9\" (UID: \"4b0fd818-20c2-4dd6-9846-f361c236dbd9\") "
Oct 31 02:09:32.111666 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:09:32.111497    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bc7a71df-c523-4590-91df-cf8fdf3ed19e-kube-api-access-98r7b\" (UniqueName: \"kubernetes.io/projected/bc7a71df-c523-4590-91df-cf8fdf3ed19e-kube-api-access-98r7b\") pod \"bc7a71df-c523-4590-91df-cf8fdf3ed19e\" (UID: \"bc7a71df-c523-4590-91df-cf8fdf3ed19e\") "
Oct 31 02:09:32.111666 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:09:32.111532    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bc7a71df-c523-4590-91df-cf8fdf3ed19e-home\" (UniqueName: \"kubernetes.io/empty-dir/bc7a71df-c523-4590-91df-cf8fdf3ed19e-home\") pod \"bc7a71df-c523-4590-91df-cf8fdf3ed19e\" (UID: \"bc7a71df-c523-4590-91df-cf8fdf3ed19e\") "
Oct 31 02:09:32.111666 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:09:32.111576    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bc7a71df-c523-4590-91df-cf8fdf3ed19e-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bc7a71df-c523-4590-91df-cf8fdf3ed19e-tmp-dir\") pod \"bc7a71df-c523-4590-91df-cf8fdf3ed19e\" (UID: \"bc7a71df-c523-4590-91df-cf8fdf3ed19e\") "
Oct 31 02:09:32.111666 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:09:32.111608    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bc7a71df-c523-4590-91df-cf8fdf3ed19e-tls-certs\" (UniqueName: \"kubernetes.io/secret/bc7a71df-c523-4590-91df-cf8fdf3ed19e-tls-certs\") pod \"bc7a71df-c523-4590-91df-cf8fdf3ed19e\" (UID: \"bc7a71df-c523-4590-91df-cf8fdf3ed19e\") "
Oct 31 02:09:32.112010 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:09:32.111939    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc7a71df-c523-4590-91df-cf8fdf3ed19e-home" pod "bc7a71df-c523-4590-91df-cf8fdf3ed19e" (UID: "bc7a71df-c523-4590-91df-cf8fdf3ed19e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:09:32.112530 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:09:32.112119    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc7a71df-c523-4590-91df-cf8fdf3ed19e-tmp-dir" pod "bc7a71df-c523-4590-91df-cf8fdf3ed19e" (UID: "bc7a71df-c523-4590-91df-cf8fdf3ed19e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:09:32.112530 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:09:32.112473    4964 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4b0fd818-20c2-4dd6-9846-f361c236dbd9-model-cache\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:09:32.112530 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:09:32.112494    4964 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4b0fd818-20c2-4dd6-9846-f361c236dbd9-tmp-dir\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:09:32.112530 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:09:32.112508    4964 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bc7a71df-c523-4590-91df-cf8fdf3ed19e-model-cache\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:09:32.112530 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:09:32.112523    4964 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4b0fd818-20c2-4dd6-9846-f361c236dbd9-home\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:09:32.112859 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:09:32.112534    4964 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bc7a71df-c523-4590-91df-cf8fdf3ed19e-home\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:09:32.112859 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:09:32.112547    4964 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bc7a71df-c523-4590-91df-cf8fdf3ed19e-tmp-dir\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:09:32.113594 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:09:32.113536    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b0fd818-20c2-4dd6-9846-f361c236dbd9-dshm" pod "4b0fd818-20c2-4dd6-9846-f361c236dbd9" (UID: "4b0fd818-20c2-4dd6-9846-f361c236dbd9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:09:32.113686 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:09:32.113598    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc7a71df-c523-4590-91df-cf8fdf3ed19e-dshm" pod "bc7a71df-c523-4590-91df-cf8fdf3ed19e" (UID: "bc7a71df-c523-4590-91df-cf8fdf3ed19e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:09:32.114759 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:09:32.114728    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4b0fd818-20c2-4dd6-9846-f361c236dbd9-tls-certs" pod "4b0fd818-20c2-4dd6-9846-f361c236dbd9" (UID: "4b0fd818-20c2-4dd6-9846-f361c236dbd9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 02:09:32.114982 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:09:32.114954    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc7a71df-c523-4590-91df-cf8fdf3ed19e-kube-api-access-98r7b" pod "bc7a71df-c523-4590-91df-cf8fdf3ed19e" (UID: "bc7a71df-c523-4590-91df-cf8fdf3ed19e"). InnerVolumeSpecName "kube-api-access-98r7b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:09:32.115571 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:09:32.115553    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bc7a71df-c523-4590-91df-cf8fdf3ed19e-tls-certs" pod "bc7a71df-c523-4590-91df-cf8fdf3ed19e" (UID: "bc7a71df-c523-4590-91df-cf8fdf3ed19e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 02:09:32.115571 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:09:32.115555    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b0fd818-20c2-4dd6-9846-f361c236dbd9-kube-api-access-nlhpz" pod "4b0fd818-20c2-4dd6-9846-f361c236dbd9" (UID: "4b0fd818-20c2-4dd6-9846-f361c236dbd9"). InnerVolumeSpecName "kube-api-access-nlhpz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:09:32.214100 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:09:32.214057    4964 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4b0fd818-20c2-4dd6-9846-f361c236dbd9-tls-certs\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:09:32.214100 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:09:32.214102    4964 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nlhpz\" (UniqueName: \"kubernetes.io/projected/4b0fd818-20c2-4dd6-9846-f361c236dbd9-kube-api-access-nlhpz\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:09:32.214338 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:09:32.214119    4964 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-98r7b\" (UniqueName: \"kubernetes.io/projected/bc7a71df-c523-4590-91df-cf8fdf3ed19e-kube-api-access-98r7b\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:09:32.214338 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:09:32.214150    4964 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bc7a71df-c523-4590-91df-cf8fdf3ed19e-tls-certs\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:09:32.214338 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:09:32.214165    4964 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bc7a71df-c523-4590-91df-cf8fdf3ed19e-dshm\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:09:32.214338 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:09:32.214179    4964 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4b0fd818-20c2-4dd6-9846-f361c236dbd9-dshm\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:09:32.851698 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:09:32.851664    4964 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-a8e157/distinct-2-w3-a8e157-kserve-9ddd45d6-ng7j7"
Oct 31 02:09:32.851698 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:09:32.851692    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-a8e157/distinct-2-w3-a8e157-kserve-9ddd45d6-ng7j7" event={"ID":"bc7a71df-c523-4590-91df-cf8fdf3ed19e","Type":"ContainerDied","Data":"70d64fa2281f36b15e8541dd1793528c025e09a6fec3858757b6cad3fc84621a"}
Oct 31 02:09:32.852257 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:09:32.851744    4964 scope.go:122] "RemoveContainer" containerID="cfcbc7b773c61d1d43ce734aabc9a090f19a1794c12f9dac57a6821f7fef7361"
Oct 31 02:09:32.853180 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:09:32.853156    4964 generic.go:356] "Generic (PLEG): container finished" podID="4b0fd818-20c2-4dd6-9846-f361c236dbd9" containerID="0aad8fa9926e3f2093e2880f7603cac39c6ce924e62dc29d804a5c96f7cae641" exitCode=0
Oct 31 02:09:32.853255 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:09:32.853192    4964 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-a8e157/distinct-w3-a8e157-kserve-549559dd5c-t29tw"
Oct 31 02:09:32.853432 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:09:32.853244    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-a8e157/distinct-w3-a8e157-kserve-549559dd5c-t29tw" event={"ID":"4b0fd818-20c2-4dd6-9846-f361c236dbd9","Type":"ContainerDied","Data":"0aad8fa9926e3f2093e2880f7603cac39c6ce924e62dc29d804a5c96f7cae641"}
Oct 31 02:09:32.853432 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:09:32.853292    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-a8e157/distinct-w3-a8e157-kserve-549559dd5c-t29tw" event={"ID":"4b0fd818-20c2-4dd6-9846-f361c236dbd9","Type":"ContainerDied","Data":"d578446f99c916b93ae99a425267165a338fb47f8e574b3115cdd0215f78c5c4"}
Oct 31 02:09:32.864884 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:09:32.864605    4964 scope.go:122] "RemoveContainer" containerID="0aad8fa9926e3f2093e2880f7603cac39c6ce924e62dc29d804a5c96f7cae641"
Oct 31 02:09:32.874302 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:09:32.874272    4964 scope.go:122] "RemoveContainer" containerID="0aad8fa9926e3f2093e2880f7603cac39c6ce924e62dc29d804a5c96f7cae641"
Oct 31 02:09:32.874710 ip-10-0-133-197 kubenswrapper[4964]: E1031 02:09:32.874689    4964 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0aad8fa9926e3f2093e2880f7603cac39c6ce924e62dc29d804a5c96f7cae641\": container with ID starting with 0aad8fa9926e3f2093e2880f7603cac39c6ce924e62dc29d804a5c96f7cae641 not found: ID does not exist" containerID="0aad8fa9926e3f2093e2880f7603cac39c6ce924e62dc29d804a5c96f7cae641"
Oct 31 02:09:32.874763 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:09:32.874720    4964 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0aad8fa9926e3f2093e2880f7603cac39c6ce924e62dc29d804a5c96f7cae641"} err="failed to get container status \"0aad8fa9926e3f2093e2880f7603cac39c6ce924e62dc29d804a5c96f7cae641\": rpc error: code = NotFound desc = could not find container \"0aad8fa9926e3f2093e2880f7603cac39c6ce924e62dc29d804a5c96f7cae641\": container with ID starting with 0aad8fa9926e3f2093e2880f7603cac39c6ce924e62dc29d804a5c96f7cae641 not found: ID does not exist"
Oct 31 02:09:33.014473 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:09:33.014441    4964 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-a8e157/distinct-w3-a8e157-kserve-549559dd5c-t29tw"]
Oct 31 02:09:33.019849 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:09:33.019821    4964 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-a8e157/distinct-w3-a8e157-kserve-549559dd5c-t29tw"]
Oct 31 02:09:33.128118 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:09:33.128040    4964 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-a8e157/distinct-2-w3-a8e157-kserve-9ddd45d6-ng7j7"]
Oct 31 02:09:33.131710 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:09:33.131685    4964 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-a8e157/distinct-2-w3-a8e157-kserve-9ddd45d6-ng7j7"]
Oct 31 02:09:33.776849 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:09:33.776818    4964 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4b0fd818-20c2-4dd6-9846-f361c236dbd9" path="/var/lib/kubelet/pods/4b0fd818-20c2-4dd6-9846-f361c236dbd9/volumes"
Oct 31 02:09:33.777248 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:09:33.777235    4964 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bc7a71df-c523-4590-91df-cf8fdf3ed19e" path="/var/lib/kubelet/pods/bc7a71df-c523-4590-91df-cf8fdf3ed19e/volumes"
Oct 31 02:09:33.810221 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:09:33.810192    4964 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-a8e157-5597f4b787-gvgv8" podUID="3aae9913-7c43-45b0-87c8-bf1da6bf9183" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 02:09:33.810445 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:09:33.810189    4964 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w3-a8e157-69ffccd74c-z9zls" podUID="04c0b568-8251-4888-902b-e46c9483d931" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 02:09:35.340187 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:09:35.340153    4964 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-9e3b95-6654679bcb-dxw74"]
Oct 31 02:09:35.340714 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:09:35.340469    4964 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-9e3b95-6654679bcb-dxw74" podUID="081cab7b-aa01-4d09-b244-92855e9b0249" containerName="maas-api" containerID="cri-o://fecdedb8917aa829ca6b252f204f180caa58035295f108d71aac43d23eb0e1f4" gracePeriod=30
Oct 31 02:09:35.495110 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:09:35.495066    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 02:09:35.622134 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:09:35.622110    4964 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-9e3b95-6654679bcb-dxw74"
Oct 31 02:09:35.753976 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:09:35.753945    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/081cab7b-aa01-4d09-b244-92855e9b0249-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/081cab7b-aa01-4d09-b244-92855e9b0249-openshift-service-ca\") pod \"081cab7b-aa01-4d09-b244-92855e9b0249\" (UID: \"081cab7b-aa01-4d09-b244-92855e9b0249\") "
Oct 31 02:09:35.754162 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:09:35.754112    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/081cab7b-aa01-4d09-b244-92855e9b0249-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/081cab7b-aa01-4d09-b244-92855e9b0249-maas-api-tls\") pod \"081cab7b-aa01-4d09-b244-92855e9b0249\" (UID: \"081cab7b-aa01-4d09-b244-92855e9b0249\") "
Oct 31 02:09:35.754162 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:09:35.754145    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/081cab7b-aa01-4d09-b244-92855e9b0249-metrics-certs\" (UniqueName: \"kubernetes.io/secret/081cab7b-aa01-4d09-b244-92855e9b0249-metrics-certs\") pod \"081cab7b-aa01-4d09-b244-92855e9b0249\" (UID: \"081cab7b-aa01-4d09-b244-92855e9b0249\") "
Oct 31 02:09:35.754258 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:09:35.754227    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/081cab7b-aa01-4d09-b244-92855e9b0249-kube-api-access-r4n64\" (UniqueName: \"kubernetes.io/projected/081cab7b-aa01-4d09-b244-92855e9b0249-kube-api-access-r4n64\") pod \"081cab7b-aa01-4d09-b244-92855e9b0249\" (UID: \"081cab7b-aa01-4d09-b244-92855e9b0249\") "
Oct 31 02:09:35.755037 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:09:35.754968    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/081cab7b-aa01-4d09-b244-92855e9b0249-openshift-service-ca" pod "081cab7b-aa01-4d09-b244-92855e9b0249" (UID: "081cab7b-aa01-4d09-b244-92855e9b0249"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 02:09:35.757238 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:09:35.757214    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/081cab7b-aa01-4d09-b244-92855e9b0249-metrics-certs" pod "081cab7b-aa01-4d09-b244-92855e9b0249" (UID: "081cab7b-aa01-4d09-b244-92855e9b0249"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 02:09:35.757403 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:09:35.757236    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/081cab7b-aa01-4d09-b244-92855e9b0249-maas-api-tls" pod "081cab7b-aa01-4d09-b244-92855e9b0249" (UID: "081cab7b-aa01-4d09-b244-92855e9b0249"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 02:09:35.757403 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:09:35.757363    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/081cab7b-aa01-4d09-b244-92855e9b0249-kube-api-access-r4n64" pod "081cab7b-aa01-4d09-b244-92855e9b0249" (UID: "081cab7b-aa01-4d09-b244-92855e9b0249"). InnerVolumeSpecName "kube-api-access-r4n64". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:09:35.855806 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:09:35.855730    4964 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/081cab7b-aa01-4d09-b244-92855e9b0249-maas-api-tls\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:09:35.855806 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:09:35.855757    4964 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/081cab7b-aa01-4d09-b244-92855e9b0249-metrics-certs\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:09:35.855806 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:09:35.855768    4964 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r4n64\" (UniqueName: \"kubernetes.io/projected/081cab7b-aa01-4d09-b244-92855e9b0249-kube-api-access-r4n64\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:09:35.855806 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:09:35.855777    4964 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/081cab7b-aa01-4d09-b244-92855e9b0249-openshift-service-ca\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:09:35.874142 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:09:35.874112    4964 generic.go:356] "Generic (PLEG): container finished" podID="081cab7b-aa01-4d09-b244-92855e9b0249" containerID="fecdedb8917aa829ca6b252f204f180caa58035295f108d71aac43d23eb0e1f4" exitCode=0
Oct 31 02:09:35.874335 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:09:35.874167    4964 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-9e3b95-6654679bcb-dxw74"
Oct 31 02:09:35.874335 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:09:35.874191    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-9e3b95-6654679bcb-dxw74" event={"ID":"081cab7b-aa01-4d09-b244-92855e9b0249","Type":"ContainerDied","Data":"fecdedb8917aa829ca6b252f204f180caa58035295f108d71aac43d23eb0e1f4"}
Oct 31 02:09:35.874335 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:09:35.874249    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-9e3b95-6654679bcb-dxw74" event={"ID":"081cab7b-aa01-4d09-b244-92855e9b0249","Type":"ContainerDied","Data":"115ef4999c79691725b1b0ae9f3d541575e2984089453cb4efbd16b5f2b9f29c"}
Oct 31 02:09:35.874335 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:09:35.874273    4964 scope.go:122] "RemoveContainer" containerID="fecdedb8917aa829ca6b252f204f180caa58035295f108d71aac43d23eb0e1f4"
Oct 31 02:09:35.884297 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:09:35.884279    4964 scope.go:122] "RemoveContainer" containerID="fecdedb8917aa829ca6b252f204f180caa58035295f108d71aac43d23eb0e1f4"
Oct 31 02:09:35.884598 ip-10-0-133-197 kubenswrapper[4964]: E1031 02:09:35.884576    4964 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fecdedb8917aa829ca6b252f204f180caa58035295f108d71aac43d23eb0e1f4\": container with ID starting with fecdedb8917aa829ca6b252f204f180caa58035295f108d71aac43d23eb0e1f4 not found: ID does not exist" containerID="fecdedb8917aa829ca6b252f204f180caa58035295f108d71aac43d23eb0e1f4"
Oct 31 02:09:35.884664 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:09:35.884610    4964 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fecdedb8917aa829ca6b252f204f180caa58035295f108d71aac43d23eb0e1f4"} err="failed to get container status \"fecdedb8917aa829ca6b252f204f180caa58035295f108d71aac43d23eb0e1f4\": rpc error: code = NotFound desc = could not find container \"fecdedb8917aa829ca6b252f204f180caa58035295f108d71aac43d23eb0e1f4\": container with ID starting with fecdedb8917aa829ca6b252f204f180caa58035295f108d71aac43d23eb0e1f4 not found: ID does not exist"
Oct 31 02:09:36.013827 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:09:36.013799    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 02:09:36.036809 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:09:36.036769    4964 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-9e3b95-6654679bcb-dxw74"]
Oct 31 02:09:36.039842 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:09:36.039817    4964 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-9e3b95-6654679bcb-dxw74"]
Oct 31 02:09:36.987723 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:09:36.987692    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 02:09:37.207268 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:09:37.207232    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 02:09:37.377747 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:09:37.377663    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 02:09:37.778004 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:09:37.777969    4964 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="081cab7b-aa01-4d09-b244-92855e9b0249" path="/var/lib/kubelet/pods/081cab7b-aa01-4d09-b244-92855e9b0249/volumes"
Oct 31 02:09:37.933577 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:09:37.933540    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 02:09:39.666770 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:09:39.666739    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 02:09:41.238079 ip-10-0-133-197 kubenswrapper[4964]: E1031 02:09:41.238042    4964 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q56x5" containerName="extractor"
Oct 31 02:09:41.851275 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:09:41.851242    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 02:09:42.048273 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:09:42.048239    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 02:09:42.215615 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:09:42.215517    4964 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-183266-openshift-default-74c6876c79-qqthj"]
Oct 31 02:09:42.215812 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:09:42.215766    4964 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-autores-183266-openshift-default-74c6876c79-qqthj" podUID="c7eddf8c-a855-4a72-b3ed-eacf3aaed5af" containerName="istio-proxy" containerID="cri-o://d7ba0c36d9f6cb9f697bc897d8af55562419ea96a5e71aeb5eba1022f6077f67" gracePeriod=30
Oct 31 02:09:42.349894 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:09:42.349830    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 02:09:42.551039 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:09:42.551003    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 02:09:43.810026 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:09:43.809983    4964 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-a8e157-5597f4b787-gvgv8" podUID="3aae9913-7c43-45b0-87c8-bf1da6bf9183" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 02:09:43.810481 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:09:43.810125    4964 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w3-a8e157-69ffccd74c-z9zls" podUID="04c0b568-8251-4888-902b-e46c9483d931" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 02:09:44.529062 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:09:44.529028    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 02:09:44.754489 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:09:44.754456    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 02:09:44.930302 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:09:44.930204    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 02:09:45.726213 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:09:45.726179    4964 patch_prober.go:29] interesting pod/e2e-autores-183266-openshift-default-74c6876c79-qqthj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 02:09:45.726426 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:09:45.726248    4964 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-autores-183266-openshift-default-74c6876c79-qqthj" podUID="c7eddf8c-a855-4a72-b3ed-eacf3aaed5af" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 02:09:47.481874 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:09:47.481797    4964 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-183266-openshift-default-74c6876c79-qqthj"
Oct 31 02:09:47.578201 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:09:47.578166    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c7eddf8c-a855-4a72-b3ed-eacf3aaed5af-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7eddf8c-a855-4a72-b3ed-eacf3aaed5af-workload-socket\") pod \"c7eddf8c-a855-4a72-b3ed-eacf3aaed5af\" (UID: \"c7eddf8c-a855-4a72-b3ed-eacf3aaed5af\") "
Oct 31 02:09:47.578431 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:09:47.578254    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c7eddf8c-a855-4a72-b3ed-eacf3aaed5af-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c7eddf8c-a855-4a72-b3ed-eacf3aaed5af-istio-data\") pod \"c7eddf8c-a855-4a72-b3ed-eacf3aaed5af\" (UID: \"c7eddf8c-a855-4a72-b3ed-eacf3aaed5af\") "
Oct 31 02:09:47.578431 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:09:47.578425    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c7eddf8c-a855-4a72-b3ed-eacf3aaed5af-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7eddf8c-a855-4a72-b3ed-eacf3aaed5af-credential-socket\") pod \"c7eddf8c-a855-4a72-b3ed-eacf3aaed5af\" (UID: \"c7eddf8c-a855-4a72-b3ed-eacf3aaed5af\") "
Oct 31 02:09:47.578573 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:09:47.578470    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c7eddf8c-a855-4a72-b3ed-eacf3aaed5af-kube-api-access-zcvmj\" (UniqueName: \"kubernetes.io/projected/c7eddf8c-a855-4a72-b3ed-eacf3aaed5af-kube-api-access-zcvmj\") pod \"c7eddf8c-a855-4a72-b3ed-eacf3aaed5af\" (UID: \"c7eddf8c-a855-4a72-b3ed-eacf3aaed5af\") "
Oct 31 02:09:47.578573 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:09:47.578519    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c7eddf8c-a855-4a72-b3ed-eacf3aaed5af-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c7eddf8c-a855-4a72-b3ed-eacf3aaed5af-workload-certs\") pod \"c7eddf8c-a855-4a72-b3ed-eacf3aaed5af\" (UID: \"c7eddf8c-a855-4a72-b3ed-eacf3aaed5af\") "
Oct 31 02:09:47.578573 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:09:47.578524    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7eddf8c-a855-4a72-b3ed-eacf3aaed5af-workload-socket" pod "c7eddf8c-a855-4a72-b3ed-eacf3aaed5af" (UID: "c7eddf8c-a855-4a72-b3ed-eacf3aaed5af"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:09:47.578573 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:09:47.578537    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7eddf8c-a855-4a72-b3ed-eacf3aaed5af-istio-data" pod "c7eddf8c-a855-4a72-b3ed-eacf3aaed5af" (UID: "c7eddf8c-a855-4a72-b3ed-eacf3aaed5af"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:09:47.578794 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:09:47.578574    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c7eddf8c-a855-4a72-b3ed-eacf3aaed5af-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c7eddf8c-a855-4a72-b3ed-eacf3aaed5af-istio-envoy\") pod \"c7eddf8c-a855-4a72-b3ed-eacf3aaed5af\" (UID: \"c7eddf8c-a855-4a72-b3ed-eacf3aaed5af\") "
Oct 31 02:09:47.578794 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:09:47.578608    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c7eddf8c-a855-4a72-b3ed-eacf3aaed5af-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c7eddf8c-a855-4a72-b3ed-eacf3aaed5af-istiod-ca-cert\") pod \"c7eddf8c-a855-4a72-b3ed-eacf3aaed5af\" (UID: \"c7eddf8c-a855-4a72-b3ed-eacf3aaed5af\") "
Oct 31 02:09:47.578794 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:09:47.578641    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c7eddf8c-a855-4a72-b3ed-eacf3aaed5af-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c7eddf8c-a855-4a72-b3ed-eacf3aaed5af-istio-podinfo\") pod \"c7eddf8c-a855-4a72-b3ed-eacf3aaed5af\" (UID: \"c7eddf8c-a855-4a72-b3ed-eacf3aaed5af\") "
Oct 31 02:09:47.578794 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:09:47.578713    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7eddf8c-a855-4a72-b3ed-eacf3aaed5af-credential-socket" pod "c7eddf8c-a855-4a72-b3ed-eacf3aaed5af" (UID: "c7eddf8c-a855-4a72-b3ed-eacf3aaed5af"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:09:47.578794 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:09:47.578721    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c7eddf8c-a855-4a72-b3ed-eacf3aaed5af-istio-token\" (UniqueName: \"kubernetes.io/projected/c7eddf8c-a855-4a72-b3ed-eacf3aaed5af-istio-token\") pod \"c7eddf8c-a855-4a72-b3ed-eacf3aaed5af\" (UID: \"c7eddf8c-a855-4a72-b3ed-eacf3aaed5af\") "
Oct 31 02:09:47.579073 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:09:47.578857    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7eddf8c-a855-4a72-b3ed-eacf3aaed5af-workload-certs" pod "c7eddf8c-a855-4a72-b3ed-eacf3aaed5af" (UID: "c7eddf8c-a855-4a72-b3ed-eacf3aaed5af"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:09:47.579139 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:09:47.579072    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c7eddf8c-a855-4a72-b3ed-eacf3aaed5af-istiod-ca-cert" pod "c7eddf8c-a855-4a72-b3ed-eacf3aaed5af" (UID: "c7eddf8c-a855-4a72-b3ed-eacf3aaed5af"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 02:09:47.579423 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:09:47.579405    4964 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7eddf8c-a855-4a72-b3ed-eacf3aaed5af-credential-socket\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:09:47.579490 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:09:47.579424    4964 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c7eddf8c-a855-4a72-b3ed-eacf3aaed5af-workload-certs\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:09:47.579490 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:09:47.579432    4964 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c7eddf8c-a855-4a72-b3ed-eacf3aaed5af-istiod-ca-cert\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:09:47.579490 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:09:47.579441    4964 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7eddf8c-a855-4a72-b3ed-eacf3aaed5af-workload-socket\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:09:47.579490 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:09:47.579450    4964 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c7eddf8c-a855-4a72-b3ed-eacf3aaed5af-istio-data\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:09:47.581284 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:09:47.581240    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c7eddf8c-a855-4a72-b3ed-eacf3aaed5af-kube-api-access-zcvmj" pod "c7eddf8c-a855-4a72-b3ed-eacf3aaed5af" (UID: "c7eddf8c-a855-4a72-b3ed-eacf3aaed5af"). InnerVolumeSpecName "kube-api-access-zcvmj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:09:47.581502 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:09:47.581461    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c7eddf8c-a855-4a72-b3ed-eacf3aaed5af-istio-podinfo" pod "c7eddf8c-a855-4a72-b3ed-eacf3aaed5af" (UID: "c7eddf8c-a855-4a72-b3ed-eacf3aaed5af"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 02:09:47.581847 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:09:47.581815    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7eddf8c-a855-4a72-b3ed-eacf3aaed5af-istio-envoy" pod "c7eddf8c-a855-4a72-b3ed-eacf3aaed5af" (UID: "c7eddf8c-a855-4a72-b3ed-eacf3aaed5af"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:09:47.581847 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:09:47.581825    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c7eddf8c-a855-4a72-b3ed-eacf3aaed5af-istio-token" pod "c7eddf8c-a855-4a72-b3ed-eacf3aaed5af" (UID: "c7eddf8c-a855-4a72-b3ed-eacf3aaed5af"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:09:47.595451 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:09:47.595416    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 02:09:47.680243 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:09:47.680213    4964 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c7eddf8c-a855-4a72-b3ed-eacf3aaed5af-istio-envoy\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:09:47.680243 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:09:47.680240    4964 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c7eddf8c-a855-4a72-b3ed-eacf3aaed5af-istio-podinfo\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:09:47.680468 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:09:47.680254    4964 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c7eddf8c-a855-4a72-b3ed-eacf3aaed5af-istio-token\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:09:47.680468 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:09:47.680268    4964 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zcvmj\" (UniqueName: \"kubernetes.io/projected/c7eddf8c-a855-4a72-b3ed-eacf3aaed5af-kube-api-access-zcvmj\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:09:47.845087 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:09:47.845052    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 02:09:47.955608 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:09:47.955573    4964 generic.go:356] "Generic (PLEG): container finished" podID="c7eddf8c-a855-4a72-b3ed-eacf3aaed5af" containerID="d7ba0c36d9f6cb9f697bc897d8af55562419ea96a5e71aeb5eba1022f6077f67" exitCode=0
Oct 31 02:09:47.955608 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:09:47.955611    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-183266-openshift-default-74c6876c79-qqthj" event={"ID":"c7eddf8c-a855-4a72-b3ed-eacf3aaed5af","Type":"ContainerDied","Data":"d7ba0c36d9f6cb9f697bc897d8af55562419ea96a5e71aeb5eba1022f6077f67"}
Oct 31 02:09:47.955852 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:09:47.955624    4964 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-183266-openshift-default-74c6876c79-qqthj"
Oct 31 02:09:47.955852 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:09:47.955652    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-183266-openshift-default-74c6876c79-qqthj" event={"ID":"c7eddf8c-a855-4a72-b3ed-eacf3aaed5af","Type":"ContainerDied","Data":"5ff71655dae35f955e09f80d95ba1a49fd62c9b4330153d14c244136c9bbe70b"}
Oct 31 02:09:47.955852 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:09:47.955671    4964 scope.go:122] "RemoveContainer" containerID="d7ba0c36d9f6cb9f697bc897d8af55562419ea96a5e71aeb5eba1022f6077f67"
Oct 31 02:09:47.965662 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:09:47.965641    4964 scope.go:122] "RemoveContainer" containerID="998678af9b3d5cbb6f2d50be1b9eba437422d6b2be51a90aa02b87d2102e4abe"
Oct 31 02:09:47.977088 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:09:47.977059    4964 scope.go:122] "RemoveContainer" containerID="d7ba0c36d9f6cb9f697bc897d8af55562419ea96a5e71aeb5eba1022f6077f67"
Oct 31 02:09:47.977461 ip-10-0-133-197 kubenswrapper[4964]: E1031 02:09:47.977440    4964 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d7ba0c36d9f6cb9f697bc897d8af55562419ea96a5e71aeb5eba1022f6077f67\": container with ID starting with d7ba0c36d9f6cb9f697bc897d8af55562419ea96a5e71aeb5eba1022f6077f67 not found: ID does not exist" containerID="d7ba0c36d9f6cb9f697bc897d8af55562419ea96a5e71aeb5eba1022f6077f67"
Oct 31 02:09:47.977584 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:09:47.977473    4964 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d7ba0c36d9f6cb9f697bc897d8af55562419ea96a5e71aeb5eba1022f6077f67"} err="failed to get container status \"d7ba0c36d9f6cb9f697bc897d8af55562419ea96a5e71aeb5eba1022f6077f67\": rpc error: code = NotFound desc = could not find container \"d7ba0c36d9f6cb9f697bc897d8af55562419ea96a5e71aeb5eba1022f6077f67\": container with ID starting with d7ba0c36d9f6cb9f697bc897d8af55562419ea96a5e71aeb5eba1022f6077f67 not found: ID does not exist"
Oct 31 02:09:47.977584 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:09:47.977492    4964 scope.go:122] "RemoveContainer" containerID="998678af9b3d5cbb6f2d50be1b9eba437422d6b2be51a90aa02b87d2102e4abe"
Oct 31 02:09:47.977829 ip-10-0-133-197 kubenswrapper[4964]: E1031 02:09:47.977805    4964 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"998678af9b3d5cbb6f2d50be1b9eba437422d6b2be51a90aa02b87d2102e4abe\": container with ID starting with 998678af9b3d5cbb6f2d50be1b9eba437422d6b2be51a90aa02b87d2102e4abe not found: ID does not exist" containerID="998678af9b3d5cbb6f2d50be1b9eba437422d6b2be51a90aa02b87d2102e4abe"
Oct 31 02:09:47.977919 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:09:47.977841    4964 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"998678af9b3d5cbb6f2d50be1b9eba437422d6b2be51a90aa02b87d2102e4abe"} err="failed to get container status \"998678af9b3d5cbb6f2d50be1b9eba437422d6b2be51a90aa02b87d2102e4abe\": rpc error: code = NotFound desc = could not find container \"998678af9b3d5cbb6f2d50be1b9eba437422d6b2be51a90aa02b87d2102e4abe\": container with ID starting with 998678af9b3d5cbb6f2d50be1b9eba437422d6b2be51a90aa02b87d2102e4abe not found: ID does not exist"
Oct 31 02:09:48.045104 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:09:48.045024    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 02:09:48.120877 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:09:48.120841    4964 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-183266-openshift-default-74c6876c79-qqthj"]
Oct 31 02:09:48.151561 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:09:48.151531    4964 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-autores-183266-openshift-default-74c6876c79-qqthj"]
Oct 31 02:09:48.967402 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:09:48.967366    4964 generic.go:356] "Generic (PLEG): container finished" podID="04c0b568-8251-4888-902b-e46c9483d931" containerID="f42e6fe1b20bd1dc35351757e16228ae4d7b8cd79084274a94a12adfd3b29d95" exitCode=0
Oct 31 02:09:48.967758 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:09:48.967409    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-a8e157-69ffccd74c-z9zls" event={"ID":"04c0b568-8251-4888-902b-e46c9483d931","Type":"ContainerDied","Data":"f42e6fe1b20bd1dc35351757e16228ae4d7b8cd79084274a94a12adfd3b29d95"}
Oct 31 02:09:49.036476 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:09:49.036451    4964 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-a8e157-69ffccd74c-z9zls"
Oct 31 02:09:49.095008 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:09:49.094978    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/04c0b568-8251-4888-902b-e46c9483d931-kube-api-access-mmnv7\" (UniqueName: \"kubernetes.io/projected/04c0b568-8251-4888-902b-e46c9483d931-kube-api-access-mmnv7\") pod \"04c0b568-8251-4888-902b-e46c9483d931\" (UID: \"04c0b568-8251-4888-902b-e46c9483d931\") "
Oct 31 02:09:49.095191 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:09:49.095077    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/04c0b568-8251-4888-902b-e46c9483d931-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/04c0b568-8251-4888-902b-e46c9483d931-plugins-config-volume\") pod \"04c0b568-8251-4888-902b-e46c9483d931\" (UID: \"04c0b568-8251-4888-902b-e46c9483d931\") "
Oct 31 02:09:49.095762 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:09:49.095721    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/04c0b568-8251-4888-902b-e46c9483d931-plugins-config-volume" pod "04c0b568-8251-4888-902b-e46c9483d931" (UID: "04c0b568-8251-4888-902b-e46c9483d931"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 02:09:49.097226 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:09:49.097198    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/04c0b568-8251-4888-902b-e46c9483d931-kube-api-access-mmnv7" pod "04c0b568-8251-4888-902b-e46c9483d931" (UID: "04c0b568-8251-4888-902b-e46c9483d931"). InnerVolumeSpecName "kube-api-access-mmnv7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:09:49.121160 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:09:49.121135    4964 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-a8e157-5597f4b787-gvgv8"
Oct 31 02:09:49.196531 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:09:49.196498    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3aae9913-7c43-45b0-87c8-bf1da6bf9183-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3aae9913-7c43-45b0-87c8-bf1da6bf9183-plugins-config-volume\") pod \"3aae9913-7c43-45b0-87c8-bf1da6bf9183\" (UID: \"3aae9913-7c43-45b0-87c8-bf1da6bf9183\") "
Oct 31 02:09:49.196714 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:09:49.196574    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3aae9913-7c43-45b0-87c8-bf1da6bf9183-kube-api-access-vk6hj\" (UniqueName: \"kubernetes.io/projected/3aae9913-7c43-45b0-87c8-bf1da6bf9183-kube-api-access-vk6hj\") pod \"3aae9913-7c43-45b0-87c8-bf1da6bf9183\" (UID: \"3aae9913-7c43-45b0-87c8-bf1da6bf9183\") "
Oct 31 02:09:49.196857 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:09:49.196835    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3aae9913-7c43-45b0-87c8-bf1da6bf9183-plugins-config-volume" pod "3aae9913-7c43-45b0-87c8-bf1da6bf9183" (UID: "3aae9913-7c43-45b0-87c8-bf1da6bf9183"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 02:09:49.196967 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:09:49.196942    4964 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mmnv7\" (UniqueName: \"kubernetes.io/projected/04c0b568-8251-4888-902b-e46c9483d931-kube-api-access-mmnv7\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:09:49.197087 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:09:49.196969    4964 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/04c0b568-8251-4888-902b-e46c9483d931-plugins-config-volume\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:09:49.197087 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:09:49.196982    4964 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3aae9913-7c43-45b0-87c8-bf1da6bf9183-plugins-config-volume\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:09:49.198889 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:09:49.198868    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3aae9913-7c43-45b0-87c8-bf1da6bf9183-kube-api-access-vk6hj" pod "3aae9913-7c43-45b0-87c8-bf1da6bf9183" (UID: "3aae9913-7c43-45b0-87c8-bf1da6bf9183"). InnerVolumeSpecName "kube-api-access-vk6hj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:09:49.298446 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:09:49.298353    4964 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vk6hj\" (UniqueName: \"kubernetes.io/projected/3aae9913-7c43-45b0-87c8-bf1da6bf9183-kube-api-access-vk6hj\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:09:49.779428 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:09:49.779399    4964 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c7eddf8c-a855-4a72-b3ed-eacf3aaed5af" path="/var/lib/kubelet/pods/c7eddf8c-a855-4a72-b3ed-eacf3aaed5af/volumes"
Oct 31 02:09:49.974460 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:09:49.974429    4964 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-a8e157-69ffccd74c-z9zls"
Oct 31 02:09:49.974929 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:09:49.974460    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-a8e157-69ffccd74c-z9zls" event={"ID":"04c0b568-8251-4888-902b-e46c9483d931","Type":"ContainerDied","Data":"f8bba7183db9fd1ed6d3956e12f71f0d1ec43911e6db02d9914a98216cbba6ec"}
Oct 31 02:09:49.974929 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:09:49.974522    4964 scope.go:122] "RemoveContainer" containerID="f42e6fe1b20bd1dc35351757e16228ae4d7b8cd79084274a94a12adfd3b29d95"
Oct 31 02:09:49.976139 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:09:49.976077    4964 generic.go:356] "Generic (PLEG): container finished" podID="3aae9913-7c43-45b0-87c8-bf1da6bf9183" containerID="47627fe7ac47a10530c752db69066a5f7e04a9952e5cd181616361006c1a9f88" exitCode=0
Oct 31 02:09:49.976139 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:09:49.976112    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-a8e157-5597f4b787-gvgv8" event={"ID":"3aae9913-7c43-45b0-87c8-bf1da6bf9183","Type":"ContainerDied","Data":"47627fe7ac47a10530c752db69066a5f7e04a9952e5cd181616361006c1a9f88"}
Oct 31 02:09:49.976139 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:09:49.976156    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-a8e157-5597f4b787-gvgv8" event={"ID":"3aae9913-7c43-45b0-87c8-bf1da6bf9183","Type":"ContainerDied","Data":"d24d70e0b406d51b504f701389ed3c8ccf8f5f1c0410cc0724c24d9e428f5b94"}
Oct 31 02:09:49.976139 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:09:49.976124    4964 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-a8e157-5597f4b787-gvgv8"
Oct 31 02:09:49.986487 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:09:49.986468    4964 scope.go:122] "RemoveContainer" containerID="47627fe7ac47a10530c752db69066a5f7e04a9952e5cd181616361006c1a9f88"
Oct 31 02:09:49.995372 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:09:49.995349    4964 scope.go:122] "RemoveContainer" containerID="47627fe7ac47a10530c752db69066a5f7e04a9952e5cd181616361006c1a9f88"
Oct 31 02:09:49.995677 ip-10-0-133-197 kubenswrapper[4964]: E1031 02:09:49.995660    4964 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"47627fe7ac47a10530c752db69066a5f7e04a9952e5cd181616361006c1a9f88\": container with ID starting with 47627fe7ac47a10530c752db69066a5f7e04a9952e5cd181616361006c1a9f88 not found: ID does not exist" containerID="47627fe7ac47a10530c752db69066a5f7e04a9952e5cd181616361006c1a9f88"
Oct 31 02:09:49.995743 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:09:49.995688    4964 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"47627fe7ac47a10530c752db69066a5f7e04a9952e5cd181616361006c1a9f88"} err="failed to get container status \"47627fe7ac47a10530c752db69066a5f7e04a9952e5cd181616361006c1a9f88\": rpc error: code = NotFound desc = could not find container \"47627fe7ac47a10530c752db69066a5f7e04a9952e5cd181616361006c1a9f88\": container with ID starting with 47627fe7ac47a10530c752db69066a5f7e04a9952e5cd181616361006c1a9f88 not found: ID does not exist"
Oct 31 02:09:50.137671 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:09:50.137588    4964 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-a8e157-5597f4b787-gvgv8"]
Oct 31 02:09:50.187674 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:09:50.187640    4964 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-a8e157-5597f4b787-gvgv8"]
Oct 31 02:09:50.248866 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:09:50.248805    4964 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-a8e157-69ffccd74c-z9zls"]
Oct 31 02:09:50.251884 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:09:50.251862    4964 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-a8e157-69ffccd74c-z9zls"]
Oct 31 02:09:51.776563 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:09:51.776534    4964 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="04c0b568-8251-4888-902b-e46c9483d931" path="/var/lib/kubelet/pods/04c0b568-8251-4888-902b-e46c9483d931/volumes"
Oct 31 02:09:51.776953 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:09:51.776895    4964 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3aae9913-7c43-45b0-87c8-bf1da6bf9183" path="/var/lib/kubelet/pods/3aae9913-7c43-45b0-87c8-bf1da6bf9183/volumes"
Oct 31 02:09:52.496724 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:09:52.496691    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 02:09:53.219859 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:09:53.219812    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 02:09:53.443014 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:09:53.442983    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 02:09:54.939833 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:09:54.939790    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 02:09:55.163855 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:09:55.163819    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 02:09:56.542799 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:09:56.542748    4964 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-057e2d-7bd65c5c5f-zq9zv"]
Oct 31 02:09:56.543241 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:09:56.542992    4964 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-057e2d-7bd65c5c5f-zq9zv" podUID="1898febe-c54e-4e24-a8d9-7f7ed5fe5ada" containerName="maas-api" containerID="cri-o://1945f3f29fb4009bdf005c8fa195700889536695c8a6c6ab092851a8a594f441" gracePeriod=30
Oct 31 02:09:56.690051 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:09:56.690016    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 02:09:56.813181 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:09:56.813157    4964 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-057e2d-7bd65c5c5f-zq9zv"
Oct 31 02:09:56.878882 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:09:56.878844    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1898febe-c54e-4e24-a8d9-7f7ed5fe5ada-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1898febe-c54e-4e24-a8d9-7f7ed5fe5ada-maas-api-tls\") pod \"1898febe-c54e-4e24-a8d9-7f7ed5fe5ada\" (UID: \"1898febe-c54e-4e24-a8d9-7f7ed5fe5ada\") "
Oct 31 02:09:56.879075 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:09:56.878988    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1898febe-c54e-4e24-a8d9-7f7ed5fe5ada-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1898febe-c54e-4e24-a8d9-7f7ed5fe5ada-openshift-service-ca\") pod \"1898febe-c54e-4e24-a8d9-7f7ed5fe5ada\" (UID: \"1898febe-c54e-4e24-a8d9-7f7ed5fe5ada\") "
Oct 31 02:09:56.879075 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:09:56.879021    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1898febe-c54e-4e24-a8d9-7f7ed5fe5ada-metrics-certs\" (UniqueName: \"kubernetes.io/secret/1898febe-c54e-4e24-a8d9-7f7ed5fe5ada-metrics-certs\") pod \"1898febe-c54e-4e24-a8d9-7f7ed5fe5ada\" (UID: \"1898febe-c54e-4e24-a8d9-7f7ed5fe5ada\") "
Oct 31 02:09:56.879075 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:09:56.879049    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1898febe-c54e-4e24-a8d9-7f7ed5fe5ada-kube-api-access-lv88h\" (UniqueName: \"kubernetes.io/projected/1898febe-c54e-4e24-a8d9-7f7ed5fe5ada-kube-api-access-lv88h\") pod \"1898febe-c54e-4e24-a8d9-7f7ed5fe5ada\" (UID: \"1898febe-c54e-4e24-a8d9-7f7ed5fe5ada\") "
Oct 31 02:09:56.879472 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:09:56.879445    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1898febe-c54e-4e24-a8d9-7f7ed5fe5ada-openshift-service-ca" pod "1898febe-c54e-4e24-a8d9-7f7ed5fe5ada" (UID: "1898febe-c54e-4e24-a8d9-7f7ed5fe5ada"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 02:09:56.881540 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:09:56.881516    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1898febe-c54e-4e24-a8d9-7f7ed5fe5ada-kube-api-access-lv88h" pod "1898febe-c54e-4e24-a8d9-7f7ed5fe5ada" (UID: "1898febe-c54e-4e24-a8d9-7f7ed5fe5ada"). InnerVolumeSpecName "kube-api-access-lv88h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:09:56.882062 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:09:56.882039    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1898febe-c54e-4e24-a8d9-7f7ed5fe5ada-metrics-certs" pod "1898febe-c54e-4e24-a8d9-7f7ed5fe5ada" (UID: "1898febe-c54e-4e24-a8d9-7f7ed5fe5ada"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 02:09:56.882062 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:09:56.882052    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1898febe-c54e-4e24-a8d9-7f7ed5fe5ada-maas-api-tls" pod "1898febe-c54e-4e24-a8d9-7f7ed5fe5ada" (UID: "1898febe-c54e-4e24-a8d9-7f7ed5fe5ada"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 02:09:56.980646 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:09:56.980605    4964 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1898febe-c54e-4e24-a8d9-7f7ed5fe5ada-maas-api-tls\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:09:56.980646 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:09:56.980635    4964 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1898febe-c54e-4e24-a8d9-7f7ed5fe5ada-openshift-service-ca\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:09:56.980646 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:09:56.980645    4964 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1898febe-c54e-4e24-a8d9-7f7ed5fe5ada-metrics-certs\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:09:56.980646 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:09:56.980654    4964 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lv88h\" (UniqueName: \"kubernetes.io/projected/1898febe-c54e-4e24-a8d9-7f7ed5fe5ada-kube-api-access-lv88h\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:09:57.026513 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:09:57.026477    4964 generic.go:356] "Generic (PLEG): container finished" podID="1898febe-c54e-4e24-a8d9-7f7ed5fe5ada" containerID="1945f3f29fb4009bdf005c8fa195700889536695c8a6c6ab092851a8a594f441" exitCode=0
Oct 31 02:09:57.026700 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:09:57.026534    4964 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-057e2d-7bd65c5c5f-zq9zv"
Oct 31 02:09:57.026700 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:09:57.026534    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-057e2d-7bd65c5c5f-zq9zv" event={"ID":"1898febe-c54e-4e24-a8d9-7f7ed5fe5ada","Type":"ContainerDied","Data":"1945f3f29fb4009bdf005c8fa195700889536695c8a6c6ab092851a8a594f441"}
Oct 31 02:09:57.026700 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:09:57.026598    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-057e2d-7bd65c5c5f-zq9zv" event={"ID":"1898febe-c54e-4e24-a8d9-7f7ed5fe5ada","Type":"ContainerDied","Data":"2d5f2828966d9520e4895d38a8284d4a95879cadf75079dcbdb3605f33b35697"}
Oct 31 02:09:57.026700 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:09:57.026629    4964 scope.go:122] "RemoveContainer" containerID="1945f3f29fb4009bdf005c8fa195700889536695c8a6c6ab092851a8a594f441"
Oct 31 02:09:57.037962 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:09:57.037944    4964 scope.go:122] "RemoveContainer" containerID="1945f3f29fb4009bdf005c8fa195700889536695c8a6c6ab092851a8a594f441"
Oct 31 02:09:57.038287 ip-10-0-133-197 kubenswrapper[4964]: E1031 02:09:57.038269    4964 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1945f3f29fb4009bdf005c8fa195700889536695c8a6c6ab092851a8a594f441\": container with ID starting with 1945f3f29fb4009bdf005c8fa195700889536695c8a6c6ab092851a8a594f441 not found: ID does not exist" containerID="1945f3f29fb4009bdf005c8fa195700889536695c8a6c6ab092851a8a594f441"
Oct 31 02:09:57.038423 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:09:57.038295    4964 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1945f3f29fb4009bdf005c8fa195700889536695c8a6c6ab092851a8a594f441"} err="failed to get container status \"1945f3f29fb4009bdf005c8fa195700889536695c8a6c6ab092851a8a594f441\": rpc error: code = NotFound desc = could not find container \"1945f3f29fb4009bdf005c8fa195700889536695c8a6c6ab092851a8a594f441\": container with ID starting with 1945f3f29fb4009bdf005c8fa195700889536695c8a6c6ab092851a8a594f441 not found: ID does not exist"
Oct 31 02:09:57.109956 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:09:57.109749    4964 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-5632ea-678cc4999-rv754"]
Oct 31 02:09:57.189603 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:09:57.189571    4964 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-057e2d-7bd65c5c5f-zq9zv"]
Oct 31 02:09:57.192729 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:09:57.192697    4964 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-057e2d-7bd65c5c5f-zq9zv"]
Oct 31 02:09:57.414630 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:09:57.414554    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 02:09:57.634308 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:09:57.634274    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 02:09:57.776706 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:09:57.776677    4964 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1898febe-c54e-4e24-a8d9-7f7ed5fe5ada" path="/var/lib/kubelet/pods/1898febe-c54e-4e24-a8d9-7f7ed5fe5ada/volumes"
Oct 31 02:09:58.197000 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:09:58.196898    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 02:09:58.422734 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:09:58.422705    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 02:09:58.600800 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:09:58.600758    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 02:09:59.144995 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:09:59.144961    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 02:09:59.997000 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:09:59.996967    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 02:10:00.742216 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:10:00.742187    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 02:10:02.111017 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:10:02.110963    4964 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-5632ea-678cc4999-rv754" podUID="ba3f39ff-dde5-4b50-b624-d3144586f11f" containerName="payload-pre-processing" containerID="cri-o://d4f4abaa61972841995520a7c02102eb71efad52528a827861636cfb769319f1" gracePeriod=25
Oct 31 02:10:02.377875 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:10:02.377808    4964 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-5632ea-678cc4999-rv754"
Oct 31 02:10:02.540245 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:10:02.540208    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ba3f39ff-dde5-4b50-b624-d3144586f11f-kube-api-access-xmm5l\" (UniqueName: \"kubernetes.io/projected/ba3f39ff-dde5-4b50-b624-d3144586f11f-kube-api-access-xmm5l\") pod \"ba3f39ff-dde5-4b50-b624-d3144586f11f\" (UID: \"ba3f39ff-dde5-4b50-b624-d3144586f11f\") "
Oct 31 02:10:02.540414 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:10:02.540384    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ba3f39ff-dde5-4b50-b624-d3144586f11f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ba3f39ff-dde5-4b50-b624-d3144586f11f-plugins-config-volume\") pod \"ba3f39ff-dde5-4b50-b624-d3144586f11f\" (UID: \"ba3f39ff-dde5-4b50-b624-d3144586f11f\") "
Oct 31 02:10:02.540840 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:10:02.540813    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ba3f39ff-dde5-4b50-b624-d3144586f11f-plugins-config-volume" pod "ba3f39ff-dde5-4b50-b624-d3144586f11f" (UID: "ba3f39ff-dde5-4b50-b624-d3144586f11f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 02:10:02.542676 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:10:02.542653    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba3f39ff-dde5-4b50-b624-d3144586f11f-kube-api-access-xmm5l" pod "ba3f39ff-dde5-4b50-b624-d3144586f11f" (UID: "ba3f39ff-dde5-4b50-b624-d3144586f11f"). InnerVolumeSpecName "kube-api-access-xmm5l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:10:02.641742 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:10:02.641649    4964 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ba3f39ff-dde5-4b50-b624-d3144586f11f-plugins-config-volume\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:10:02.641742 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:10:02.641689    4964 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xmm5l\" (UniqueName: \"kubernetes.io/projected/ba3f39ff-dde5-4b50-b624-d3144586f11f-kube-api-access-xmm5l\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:10:02.771847 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:10:02.771814    4964 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-a8e157-584855c7c8-b66mk"]
Oct 31 02:10:02.772071 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:10:02.771998    4964 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-a8e157-584855c7c8-b66mk" podUID="e8e369b7-335b-4ac4-8e8a-3879cf430b76" containerName="maas-api" containerID="cri-o://f9f9df7866ba5e58f49b0fdc6a4e6d3b7fba4aa69a89d73206b6f5419aa92268" gracePeriod=30
Oct 31 02:10:02.915646 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:10:02.915553    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 02:10:03.012752 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:10:03.012726    4964 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-a8e157-584855c7c8-b66mk"
Oct 31 02:10:03.068155 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:10:03.068124    4964 generic.go:356] "Generic (PLEG): container finished" podID="e8e369b7-335b-4ac4-8e8a-3879cf430b76" containerID="f9f9df7866ba5e58f49b0fdc6a4e6d3b7fba4aa69a89d73206b6f5419aa92268" exitCode=0
Oct 31 02:10:03.068405 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:10:03.068165    4964 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-a8e157-584855c7c8-b66mk"
Oct 31 02:10:03.068405 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:10:03.068199    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-a8e157-584855c7c8-b66mk" event={"ID":"e8e369b7-335b-4ac4-8e8a-3879cf430b76","Type":"ContainerDied","Data":"f9f9df7866ba5e58f49b0fdc6a4e6d3b7fba4aa69a89d73206b6f5419aa92268"}
Oct 31 02:10:03.068405 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:10:03.068239    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-a8e157-584855c7c8-b66mk" event={"ID":"e8e369b7-335b-4ac4-8e8a-3879cf430b76","Type":"ContainerDied","Data":"385d1773da1e9db42be37c4660d0a86e0118c376edfb629c5294e43826c9c327"}
Oct 31 02:10:03.068405 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:10:03.068265    4964 scope.go:122] "RemoveContainer" containerID="f9f9df7866ba5e58f49b0fdc6a4e6d3b7fba4aa69a89d73206b6f5419aa92268"
Oct 31 02:10:03.069608 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:10:03.069589    4964 generic.go:356] "Generic (PLEG): container finished" podID="ba3f39ff-dde5-4b50-b624-d3144586f11f" containerID="d4f4abaa61972841995520a7c02102eb71efad52528a827861636cfb769319f1" exitCode=0
Oct 31 02:10:03.069710 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:10:03.069623    4964 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-5632ea-678cc4999-rv754"
Oct 31 02:10:03.069710 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:10:03.069680    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-5632ea-678cc4999-rv754" event={"ID":"ba3f39ff-dde5-4b50-b624-d3144586f11f","Type":"ContainerDied","Data":"d4f4abaa61972841995520a7c02102eb71efad52528a827861636cfb769319f1"}
Oct 31 02:10:03.069828 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:10:03.069726    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-5632ea-678cc4999-rv754" event={"ID":"ba3f39ff-dde5-4b50-b624-d3144586f11f","Type":"ContainerDied","Data":"ceb93aad076f6057f4c7cbf3ab3c06620cccf5846bf2781ffdf91e29ff92df8e"}
Oct 31 02:10:03.078289 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:10:03.078271    4964 scope.go:122] "RemoveContainer" containerID="f9f9df7866ba5e58f49b0fdc6a4e6d3b7fba4aa69a89d73206b6f5419aa92268"
Oct 31 02:10:03.078687 ip-10-0-133-197 kubenswrapper[4964]: E1031 02:10:03.078668    4964 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f9f9df7866ba5e58f49b0fdc6a4e6d3b7fba4aa69a89d73206b6f5419aa92268\": container with ID starting with f9f9df7866ba5e58f49b0fdc6a4e6d3b7fba4aa69a89d73206b6f5419aa92268 not found: ID does not exist" containerID="f9f9df7866ba5e58f49b0fdc6a4e6d3b7fba4aa69a89d73206b6f5419aa92268"
Oct 31 02:10:03.078774 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:10:03.078694    4964 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f9f9df7866ba5e58f49b0fdc6a4e6d3b7fba4aa69a89d73206b6f5419aa92268"} err="failed to get container status \"f9f9df7866ba5e58f49b0fdc6a4e6d3b7fba4aa69a89d73206b6f5419aa92268\": rpc error: code = NotFound desc = could not find container \"f9f9df7866ba5e58f49b0fdc6a4e6d3b7fba4aa69a89d73206b6f5419aa92268\": container with ID starting with f9f9df7866ba5e58f49b0fdc6a4e6d3b7fba4aa69a89d73206b6f5419aa92268 not found: ID does not exist"
Oct 31 02:10:03.078774 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:10:03.078714    4964 scope.go:122] "RemoveContainer" containerID="d4f4abaa61972841995520a7c02102eb71efad52528a827861636cfb769319f1"
Oct 31 02:10:03.087358 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:10:03.087339    4964 scope.go:122] "RemoveContainer" containerID="d4f4abaa61972841995520a7c02102eb71efad52528a827861636cfb769319f1"
Oct 31 02:10:03.087615 ip-10-0-133-197 kubenswrapper[4964]: E1031 02:10:03.087596    4964 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d4f4abaa61972841995520a7c02102eb71efad52528a827861636cfb769319f1\": container with ID starting with d4f4abaa61972841995520a7c02102eb71efad52528a827861636cfb769319f1 not found: ID does not exist" containerID="d4f4abaa61972841995520a7c02102eb71efad52528a827861636cfb769319f1"
Oct 31 02:10:03.087682 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:10:03.087621    4964 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d4f4abaa61972841995520a7c02102eb71efad52528a827861636cfb769319f1"} err="failed to get container status \"d4f4abaa61972841995520a7c02102eb71efad52528a827861636cfb769319f1\": rpc error: code = NotFound desc = could not find container \"d4f4abaa61972841995520a7c02102eb71efad52528a827861636cfb769319f1\": container with ID starting with d4f4abaa61972841995520a7c02102eb71efad52528a827861636cfb769319f1 not found: ID does not exist"
Oct 31 02:10:03.146875 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:10:03.146839    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e8e369b7-335b-4ac4-8e8a-3879cf430b76-metrics-certs\" (UniqueName: \"kubernetes.io/secret/e8e369b7-335b-4ac4-8e8a-3879cf430b76-metrics-certs\") pod \"e8e369b7-335b-4ac4-8e8a-3879cf430b76\" (UID: \"e8e369b7-335b-4ac4-8e8a-3879cf430b76\") "
Oct 31 02:10:03.147349 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:10:03.146985    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e8e369b7-335b-4ac4-8e8a-3879cf430b76-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e8e369b7-335b-4ac4-8e8a-3879cf430b76-maas-api-tls\") pod \"e8e369b7-335b-4ac4-8e8a-3879cf430b76\" (UID: \"e8e369b7-335b-4ac4-8e8a-3879cf430b76\") "
Oct 31 02:10:03.147349 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:10:03.147032    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e8e369b7-335b-4ac4-8e8a-3879cf430b76-kube-api-access-rmzxn\" (UniqueName: \"kubernetes.io/projected/e8e369b7-335b-4ac4-8e8a-3879cf430b76-kube-api-access-rmzxn\") pod \"e8e369b7-335b-4ac4-8e8a-3879cf430b76\" (UID: \"e8e369b7-335b-4ac4-8e8a-3879cf430b76\") "
Oct 31 02:10:03.147349 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:10:03.147105    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e8e369b7-335b-4ac4-8e8a-3879cf430b76-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e8e369b7-335b-4ac4-8e8a-3879cf430b76-openshift-service-ca\") pod \"e8e369b7-335b-4ac4-8e8a-3879cf430b76\" (UID: \"e8e369b7-335b-4ac4-8e8a-3879cf430b76\") "
Oct 31 02:10:03.147539 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:10:03.147503    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e8e369b7-335b-4ac4-8e8a-3879cf430b76-openshift-service-ca" pod "e8e369b7-335b-4ac4-8e8a-3879cf430b76" (UID: "e8e369b7-335b-4ac4-8e8a-3879cf430b76"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 02:10:03.149390 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:10:03.149295    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e8e369b7-335b-4ac4-8e8a-3879cf430b76-kube-api-access-rmzxn" pod "e8e369b7-335b-4ac4-8e8a-3879cf430b76" (UID: "e8e369b7-335b-4ac4-8e8a-3879cf430b76"). InnerVolumeSpecName "kube-api-access-rmzxn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:10:03.149643 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:10:03.149621    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e8e369b7-335b-4ac4-8e8a-3879cf430b76-metrics-certs" pod "e8e369b7-335b-4ac4-8e8a-3879cf430b76" (UID: "e8e369b7-335b-4ac4-8e8a-3879cf430b76"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 02:10:03.149751 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:10:03.149732    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e8e369b7-335b-4ac4-8e8a-3879cf430b76-maas-api-tls" pod "e8e369b7-335b-4ac4-8e8a-3879cf430b76" (UID: "e8e369b7-335b-4ac4-8e8a-3879cf430b76"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 02:10:03.248889 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:10:03.248855    4964 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e8e369b7-335b-4ac4-8e8a-3879cf430b76-maas-api-tls\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:10:03.248889 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:10:03.248889    4964 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rmzxn\" (UniqueName: \"kubernetes.io/projected/e8e369b7-335b-4ac4-8e8a-3879cf430b76-kube-api-access-rmzxn\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:10:03.249111 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:10:03.248899    4964 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e8e369b7-335b-4ac4-8e8a-3879cf430b76-openshift-service-ca\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:10:03.249111 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:10:03.248908    4964 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e8e369b7-335b-4ac4-8e8a-3879cf430b76-metrics-certs\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:10:03.291205 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:10:03.291168    4964 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-5632ea-678cc4999-rv754"]
Oct 31 02:10:03.297780 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:10:03.297743    4964 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-5632ea-678cc4999-rv754"]
Oct 31 02:10:03.488500 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:10:03.488463    4964 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-a8e157-584855c7c8-b66mk"]
Oct 31 02:10:03.491504 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:10:03.491481    4964 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-a8e157-584855c7c8-b66mk"]
Oct 31 02:10:03.776499 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:10:03.776468    4964 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ba3f39ff-dde5-4b50-b624-d3144586f11f" path="/var/lib/kubelet/pods/ba3f39ff-dde5-4b50-b624-d3144586f11f/volumes"
Oct 31 02:10:03.776934 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:10:03.776919    4964 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e8e369b7-335b-4ac4-8e8a-3879cf430b76" path="/var/lib/kubelet/pods/e8e369b7-335b-4ac4-8e8a-3879cf430b76/volumes"
Oct 31 02:10:03.970380 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:10:03.970309    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 02:10:04.024465 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:10:04.024433    4964 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-5e916d-547c767844-llcts"]
Oct 31 02:10:04.024624 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:10:04.024593    4964 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-5e916d-547c767844-llcts" podUID="7c420de6-67b9-4958-8083-ebd8bd939418" containerName="maas-api" containerID="cri-o://192a883bc42d0d874efdf649d8e388469bbae96ef777de96d86c6f727b6cc7f5" gracePeriod=30
Oct 31 02:10:04.170358 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:10:04.170303    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 02:10:04.287052 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:10:04.286995    4964 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-5e916d-547c767844-llcts"
Oct 31 02:10:04.463268 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:10:04.463240    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7c420de6-67b9-4958-8083-ebd8bd939418-kube-api-access-xxhbl\" (UniqueName: \"kubernetes.io/projected/7c420de6-67b9-4958-8083-ebd8bd939418-kube-api-access-xxhbl\") pod \"7c420de6-67b9-4958-8083-ebd8bd939418\" (UID: \"7c420de6-67b9-4958-8083-ebd8bd939418\") "
Oct 31 02:10:04.463474 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:10:04.463278    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7c420de6-67b9-4958-8083-ebd8bd939418-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7c420de6-67b9-4958-8083-ebd8bd939418-openshift-service-ca\") pod \"7c420de6-67b9-4958-8083-ebd8bd939418\" (UID: \"7c420de6-67b9-4958-8083-ebd8bd939418\") "
Oct 31 02:10:04.463474 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:10:04.463330    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7c420de6-67b9-4958-8083-ebd8bd939418-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7c420de6-67b9-4958-8083-ebd8bd939418-maas-api-tls\") pod \"7c420de6-67b9-4958-8083-ebd8bd939418\" (UID: \"7c420de6-67b9-4958-8083-ebd8bd939418\") "
Oct 31 02:10:04.463474 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:10:04.463402    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7c420de6-67b9-4958-8083-ebd8bd939418-metrics-certs\" (UniqueName: \"kubernetes.io/secret/7c420de6-67b9-4958-8083-ebd8bd939418-metrics-certs\") pod \"7c420de6-67b9-4958-8083-ebd8bd939418\" (UID: \"7c420de6-67b9-4958-8083-ebd8bd939418\") "
Oct 31 02:10:04.463742 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:10:04.463712    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7c420de6-67b9-4958-8083-ebd8bd939418-openshift-service-ca" pod "7c420de6-67b9-4958-8083-ebd8bd939418" (UID: "7c420de6-67b9-4958-8083-ebd8bd939418"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 02:10:04.465715 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:10:04.465686    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7c420de6-67b9-4958-8083-ebd8bd939418-maas-api-tls" pod "7c420de6-67b9-4958-8083-ebd8bd939418" (UID: "7c420de6-67b9-4958-8083-ebd8bd939418"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 02:10:04.465715 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:10:04.465700    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c420de6-67b9-4958-8083-ebd8bd939418-kube-api-access-xxhbl" pod "7c420de6-67b9-4958-8083-ebd8bd939418" (UID: "7c420de6-67b9-4958-8083-ebd8bd939418"). InnerVolumeSpecName "kube-api-access-xxhbl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:10:04.465868 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:10:04.465771    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7c420de6-67b9-4958-8083-ebd8bd939418-metrics-certs" pod "7c420de6-67b9-4958-8083-ebd8bd939418" (UID: "7c420de6-67b9-4958-8083-ebd8bd939418"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 02:10:04.565287 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:10:04.565207    4964 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xxhbl\" (UniqueName: \"kubernetes.io/projected/7c420de6-67b9-4958-8083-ebd8bd939418-kube-api-access-xxhbl\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:10:04.565287 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:10:04.565234    4964 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7c420de6-67b9-4958-8083-ebd8bd939418-openshift-service-ca\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:10:04.565287 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:10:04.565244    4964 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7c420de6-67b9-4958-8083-ebd8bd939418-maas-api-tls\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:10:04.565287 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:10:04.565254    4964 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7c420de6-67b9-4958-8083-ebd8bd939418-metrics-certs\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:10:04.614758 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:10:04.614717    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 02:10:04.909876 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:10:04.909797    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 02:10:05.090883 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:10:05.090848    4964 generic.go:356] "Generic (PLEG): container finished" podID="7c420de6-67b9-4958-8083-ebd8bd939418" containerID="192a883bc42d0d874efdf649d8e388469bbae96ef777de96d86c6f727b6cc7f5" exitCode=0
Oct 31 02:10:05.091071 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:10:05.090900    4964 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-5e916d-547c767844-llcts"
Oct 31 02:10:05.091071 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:10:05.090934    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-5e916d-547c767844-llcts" event={"ID":"7c420de6-67b9-4958-8083-ebd8bd939418","Type":"ContainerDied","Data":"192a883bc42d0d874efdf649d8e388469bbae96ef777de96d86c6f727b6cc7f5"}
Oct 31 02:10:05.091071 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:10:05.090994    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-5e916d-547c767844-llcts" event={"ID":"7c420de6-67b9-4958-8083-ebd8bd939418","Type":"ContainerDied","Data":"d4b0b242530e6afa02361b9137767df2c3f4863d1f9ae76fb8a46ab05ba7485a"}
Oct 31 02:10:05.091071 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:10:05.091023    4964 scope.go:122] "RemoveContainer" containerID="192a883bc42d0d874efdf649d8e388469bbae96ef777de96d86c6f727b6cc7f5"
Oct 31 02:10:05.102940 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:10:05.102915    4964 scope.go:122] "RemoveContainer" containerID="192a883bc42d0d874efdf649d8e388469bbae96ef777de96d86c6f727b6cc7f5"
Oct 31 02:10:05.103247 ip-10-0-133-197 kubenswrapper[4964]: E1031 02:10:05.103226    4964 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"192a883bc42d0d874efdf649d8e388469bbae96ef777de96d86c6f727b6cc7f5\": container with ID starting with 192a883bc42d0d874efdf649d8e388469bbae96ef777de96d86c6f727b6cc7f5 not found: ID does not exist" containerID="192a883bc42d0d874efdf649d8e388469bbae96ef777de96d86c6f727b6cc7f5"
Oct 31 02:10:05.103329 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:10:05.103257    4964 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"192a883bc42d0d874efdf649d8e388469bbae96ef777de96d86c6f727b6cc7f5"} err="failed to get container status \"192a883bc42d0d874efdf649d8e388469bbae96ef777de96d86c6f727b6cc7f5\": rpc error: code = NotFound desc = could not find container \"192a883bc42d0d874efdf649d8e388469bbae96ef777de96d86c6f727b6cc7f5\": container with ID starting with 192a883bc42d0d874efdf649d8e388469bbae96ef777de96d86c6f727b6cc7f5 not found: ID does not exist"
Oct 31 02:10:05.255101 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:10:05.255063    4964 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-5e916d-547c767844-llcts"]
Oct 31 02:10:05.303224 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:10:05.303195    4964 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-5e916d-547c767844-llcts"]
Oct 31 02:10:05.549906 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:10:05.549804    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 02:10:05.754660 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:10:05.754611    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 02:10:05.777172 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:10:05.777136    4964 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7c420de6-67b9-4958-8083-ebd8bd939418" path="/var/lib/kubelet/pods/7c420de6-67b9-4958-8083-ebd8bd939418/volumes"
Oct 31 02:10:05.936843 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:10:05.936744    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 02:10:06.149969 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:10:06.149934    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 02:10:06.325834 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:10:06.325797    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 02:10:10.368449 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:10:10.368416    4964 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-5632ea-b5b6857b8-5zj46"]
Oct 31 02:10:10.368875 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:10:10.368640    4964 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-5632ea-b5b6857b8-5zj46" podUID="290631aa-9208-4dbb-8a0c-aa79cea9e6f8" containerName="maas-api" containerID="cri-o://46cd860ac553eca1c24bf50488224f62e68e46a1d98903afd843b6b95b2cf31e" gracePeriod=30
Oct 31 02:10:10.556440 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:10:10.556288    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 02:10:10.626392 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:10:10.626294    4964 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-5632ea-b5b6857b8-5zj46"
Oct 31 02:10:10.734921 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:10:10.734886    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/290631aa-9208-4dbb-8a0c-aa79cea9e6f8-metrics-certs\" (UniqueName: \"kubernetes.io/secret/290631aa-9208-4dbb-8a0c-aa79cea9e6f8-metrics-certs\") pod \"290631aa-9208-4dbb-8a0c-aa79cea9e6f8\" (UID: \"290631aa-9208-4dbb-8a0c-aa79cea9e6f8\") "
Oct 31 02:10:10.735137 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:10:10.734936    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/290631aa-9208-4dbb-8a0c-aa79cea9e6f8-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/290631aa-9208-4dbb-8a0c-aa79cea9e6f8-maas-api-tls\") pod \"290631aa-9208-4dbb-8a0c-aa79cea9e6f8\" (UID: \"290631aa-9208-4dbb-8a0c-aa79cea9e6f8\") "
Oct 31 02:10:10.735137 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:10:10.734990    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/290631aa-9208-4dbb-8a0c-aa79cea9e6f8-kube-api-access-4xcb7\" (UniqueName: \"kubernetes.io/projected/290631aa-9208-4dbb-8a0c-aa79cea9e6f8-kube-api-access-4xcb7\") pod \"290631aa-9208-4dbb-8a0c-aa79cea9e6f8\" (UID: \"290631aa-9208-4dbb-8a0c-aa79cea9e6f8\") "
Oct 31 02:10:10.735137 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:10:10.735032    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/290631aa-9208-4dbb-8a0c-aa79cea9e6f8-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/290631aa-9208-4dbb-8a0c-aa79cea9e6f8-openshift-service-ca\") pod \"290631aa-9208-4dbb-8a0c-aa79cea9e6f8\" (UID: \"290631aa-9208-4dbb-8a0c-aa79cea9e6f8\") "
Oct 31 02:10:10.735460 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:10:10.735432    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/290631aa-9208-4dbb-8a0c-aa79cea9e6f8-openshift-service-ca" pod "290631aa-9208-4dbb-8a0c-aa79cea9e6f8" (UID: "290631aa-9208-4dbb-8a0c-aa79cea9e6f8"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 02:10:10.737225 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:10:10.737199    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/290631aa-9208-4dbb-8a0c-aa79cea9e6f8-maas-api-tls" pod "290631aa-9208-4dbb-8a0c-aa79cea9e6f8" (UID: "290631aa-9208-4dbb-8a0c-aa79cea9e6f8"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 02:10:10.737225 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:10:10.737215    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/290631aa-9208-4dbb-8a0c-aa79cea9e6f8-metrics-certs" pod "290631aa-9208-4dbb-8a0c-aa79cea9e6f8" (UID: "290631aa-9208-4dbb-8a0c-aa79cea9e6f8"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 02:10:10.737454 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:10:10.737254    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/290631aa-9208-4dbb-8a0c-aa79cea9e6f8-kube-api-access-4xcb7" pod "290631aa-9208-4dbb-8a0c-aa79cea9e6f8" (UID: "290631aa-9208-4dbb-8a0c-aa79cea9e6f8"). InnerVolumeSpecName "kube-api-access-4xcb7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:10:10.835914 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:10:10.835879    4964 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/290631aa-9208-4dbb-8a0c-aa79cea9e6f8-metrics-certs\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:10:10.835914 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:10:10.835906    4964 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/290631aa-9208-4dbb-8a0c-aa79cea9e6f8-maas-api-tls\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:10:10.835914 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:10:10.835916    4964 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4xcb7\" (UniqueName: \"kubernetes.io/projected/290631aa-9208-4dbb-8a0c-aa79cea9e6f8-kube-api-access-4xcb7\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:10:10.836186 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:10:10.835925    4964 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/290631aa-9208-4dbb-8a0c-aa79cea9e6f8-openshift-service-ca\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:10:11.125428 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:10:11.125388    4964 generic.go:356] "Generic (PLEG): container finished" podID="290631aa-9208-4dbb-8a0c-aa79cea9e6f8" containerID="46cd860ac553eca1c24bf50488224f62e68e46a1d98903afd843b6b95b2cf31e" exitCode=0
Oct 31 02:10:11.125596 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:10:11.125446    4964 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-5632ea-b5b6857b8-5zj46"
Oct 31 02:10:11.125596 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:10:11.125453    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-5632ea-b5b6857b8-5zj46" event={"ID":"290631aa-9208-4dbb-8a0c-aa79cea9e6f8","Type":"ContainerDied","Data":"46cd860ac553eca1c24bf50488224f62e68e46a1d98903afd843b6b95b2cf31e"}
Oct 31 02:10:11.125596 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:10:11.125506    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-5632ea-b5b6857b8-5zj46" event={"ID":"290631aa-9208-4dbb-8a0c-aa79cea9e6f8","Type":"ContainerDied","Data":"40b8119b50769dff2377c83ea574e47f48e504f2a07254e66a5f7aeb072de6a9"}
Oct 31 02:10:11.125596 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:10:11.125533    4964 scope.go:122] "RemoveContainer" containerID="46cd860ac553eca1c24bf50488224f62e68e46a1d98903afd843b6b95b2cf31e"
Oct 31 02:10:11.135569 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:10:11.135553    4964 scope.go:122] "RemoveContainer" containerID="46cd860ac553eca1c24bf50488224f62e68e46a1d98903afd843b6b95b2cf31e"
Oct 31 02:10:11.135841 ip-10-0-133-197 kubenswrapper[4964]: E1031 02:10:11.135821    4964 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"46cd860ac553eca1c24bf50488224f62e68e46a1d98903afd843b6b95b2cf31e\": container with ID starting with 46cd860ac553eca1c24bf50488224f62e68e46a1d98903afd843b6b95b2cf31e not found: ID does not exist" containerID="46cd860ac553eca1c24bf50488224f62e68e46a1d98903afd843b6b95b2cf31e"
Oct 31 02:10:11.135907 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:10:11.135850    4964 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"46cd860ac553eca1c24bf50488224f62e68e46a1d98903afd843b6b95b2cf31e"} err="failed to get container status \"46cd860ac553eca1c24bf50488224f62e68e46a1d98903afd843b6b95b2cf31e\": rpc error: code = NotFound desc = could not find container \"46cd860ac553eca1c24bf50488224f62e68e46a1d98903afd843b6b95b2cf31e\": container with ID starting with 46cd860ac553eca1c24bf50488224f62e68e46a1d98903afd843b6b95b2cf31e not found: ID does not exist"
Oct 31 02:10:11.237211 ip-10-0-133-197 kubenswrapper[4964]: E1031 02:10:11.237177    4964 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q56x5" containerName="extractor"
Oct 31 02:10:11.288976 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:10:11.288946    4964 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-5632ea-b5b6857b8-5zj46"]
Oct 31 02:10:11.299156 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:10:11.292922    4964 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-5632ea-b5b6857b8-5zj46"]
Oct 31 02:10:11.776060 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:10:11.776026    4964 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="290631aa-9208-4dbb-8a0c-aa79cea9e6f8" path="/var/lib/kubelet/pods/290631aa-9208-4dbb-8a0c-aa79cea9e6f8/volumes"
Oct 31 02:10:11.866198 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:10:11.866165    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 02:10:12.042034 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:10:12.041947    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 02:10:12.243413 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:10:12.243378    4964 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-7jd55"]
Oct 31 02:10:12.243606 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:10:12.243583    4964 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-7jd55" podUID="fd5d3644-7596-44e1-844f-fdc3c68b5021" containerName="istio-proxy" containerID="cri-o://6a604a4e640a88052a646705ff4256ad69d192ed642c00387523baa5279161f6" gracePeriod=30
Oct 31 02:10:12.288701 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:10:12.288652    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 02:10:12.470554 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:10:12.470523    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 02:10:13.035922 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:10:13.035886    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 02:10:16.077542 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:10:16.077505    4964 patch_prober.go:29] interesting pod/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-7jd55 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 02:10:16.077914 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:10:16.077562    4964 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-7jd55" podUID="fd5d3644-7596-44e1-844f-fdc3c68b5021" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 02:10:16.864775 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:10:16.864733    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 02:10:17.403673 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:10:17.403638    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 02:10:17.497092 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:10:17.497064    4964 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-7jd55"
Oct 31 02:10:17.600637 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:10:17.600560    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fd5d3644-7596-44e1-844f-fdc3c68b5021-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fd5d3644-7596-44e1-844f-fdc3c68b5021-istio-data\") pod \"fd5d3644-7596-44e1-844f-fdc3c68b5021\" (UID: \"fd5d3644-7596-44e1-844f-fdc3c68b5021\") "
Oct 31 02:10:17.600637 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:10:17.600613    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fd5d3644-7596-44e1-844f-fdc3c68b5021-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fd5d3644-7596-44e1-844f-fdc3c68b5021-workload-certs\") pod \"fd5d3644-7596-44e1-844f-fdc3c68b5021\" (UID: \"fd5d3644-7596-44e1-844f-fdc3c68b5021\") "
Oct 31 02:10:17.600853 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:10:17.600671    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fd5d3644-7596-44e1-844f-fdc3c68b5021-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd5d3644-7596-44e1-844f-fdc3c68b5021-credential-socket\") pod \"fd5d3644-7596-44e1-844f-fdc3c68b5021\" (UID: \"fd5d3644-7596-44e1-844f-fdc3c68b5021\") "
Oct 31 02:10:17.600853 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:10:17.600709    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fd5d3644-7596-44e1-844f-fdc3c68b5021-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fd5d3644-7596-44e1-844f-fdc3c68b5021-istio-podinfo\") pod \"fd5d3644-7596-44e1-844f-fdc3c68b5021\" (UID: \"fd5d3644-7596-44e1-844f-fdc3c68b5021\") "
Oct 31 02:10:17.600853 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:10:17.600728    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fd5d3644-7596-44e1-844f-fdc3c68b5021-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fd5d3644-7596-44e1-844f-fdc3c68b5021-istio-envoy\") pod \"fd5d3644-7596-44e1-844f-fdc3c68b5021\" (UID: \"fd5d3644-7596-44e1-844f-fdc3c68b5021\") "
Oct 31 02:10:17.600853 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:10:17.600749    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fd5d3644-7596-44e1-844f-fdc3c68b5021-istio-token\" (UniqueName: \"kubernetes.io/projected/fd5d3644-7596-44e1-844f-fdc3c68b5021-istio-token\") pod \"fd5d3644-7596-44e1-844f-fdc3c68b5021\" (UID: \"fd5d3644-7596-44e1-844f-fdc3c68b5021\") "
Oct 31 02:10:17.600853 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:10:17.600775    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fd5d3644-7596-44e1-844f-fdc3c68b5021-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd5d3644-7596-44e1-844f-fdc3c68b5021-workload-socket\") pod \"fd5d3644-7596-44e1-844f-fdc3c68b5021\" (UID: \"fd5d3644-7596-44e1-844f-fdc3c68b5021\") "
Oct 31 02:10:17.600853 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:10:17.600813    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fd5d3644-7596-44e1-844f-fdc3c68b5021-kube-api-access-hlbtc\" (UniqueName: \"kubernetes.io/projected/fd5d3644-7596-44e1-844f-fdc3c68b5021-kube-api-access-hlbtc\") pod \"fd5d3644-7596-44e1-844f-fdc3c68b5021\" (UID: \"fd5d3644-7596-44e1-844f-fdc3c68b5021\") "
Oct 31 02:10:17.601157 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:10:17.600867    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fd5d3644-7596-44e1-844f-fdc3c68b5021-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fd5d3644-7596-44e1-844f-fdc3c68b5021-istiod-ca-cert\") pod \"fd5d3644-7596-44e1-844f-fdc3c68b5021\" (UID: \"fd5d3644-7596-44e1-844f-fdc3c68b5021\") "
Oct 31 02:10:17.601157 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:10:17.600854    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd5d3644-7596-44e1-844f-fdc3c68b5021-istio-data" pod "fd5d3644-7596-44e1-844f-fdc3c68b5021" (UID: "fd5d3644-7596-44e1-844f-fdc3c68b5021"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:10:17.601157 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:10:17.600910    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd5d3644-7596-44e1-844f-fdc3c68b5021-credential-socket" pod "fd5d3644-7596-44e1-844f-fdc3c68b5021" (UID: "fd5d3644-7596-44e1-844f-fdc3c68b5021"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:10:17.601157 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:10:17.601121    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd5d3644-7596-44e1-844f-fdc3c68b5021-workload-socket" pod "fd5d3644-7596-44e1-844f-fdc3c68b5021" (UID: "fd5d3644-7596-44e1-844f-fdc3c68b5021"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:10:17.601157 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:10:17.601149    4964 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd5d3644-7596-44e1-844f-fdc3c68b5021-credential-socket\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:10:17.601505 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:10:17.601167    4964 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fd5d3644-7596-44e1-844f-fdc3c68b5021-istio-data\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:10:17.601505 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:10:17.601270    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd5d3644-7596-44e1-844f-fdc3c68b5021-workload-certs" pod "fd5d3644-7596-44e1-844f-fdc3c68b5021" (UID: "fd5d3644-7596-44e1-844f-fdc3c68b5021"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:10:17.601505 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:10:17.601442    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fd5d3644-7596-44e1-844f-fdc3c68b5021-istiod-ca-cert" pod "fd5d3644-7596-44e1-844f-fdc3c68b5021" (UID: "fd5d3644-7596-44e1-844f-fdc3c68b5021"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 02:10:17.603591 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:10:17.603565    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fd5d3644-7596-44e1-844f-fdc3c68b5021-istio-podinfo" pod "fd5d3644-7596-44e1-844f-fdc3c68b5021" (UID: "fd5d3644-7596-44e1-844f-fdc3c68b5021"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 02:10:17.603591 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:10:17.603580    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd5d3644-7596-44e1-844f-fdc3c68b5021-istio-token" pod "fd5d3644-7596-44e1-844f-fdc3c68b5021" (UID: "fd5d3644-7596-44e1-844f-fdc3c68b5021"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:10:17.603788 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:10:17.603661    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd5d3644-7596-44e1-844f-fdc3c68b5021-kube-api-access-hlbtc" pod "fd5d3644-7596-44e1-844f-fdc3c68b5021" (UID: "fd5d3644-7596-44e1-844f-fdc3c68b5021"). InnerVolumeSpecName "kube-api-access-hlbtc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:10:17.603788 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:10:17.603656    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd5d3644-7596-44e1-844f-fdc3c68b5021-istio-envoy" pod "fd5d3644-7596-44e1-844f-fdc3c68b5021" (UID: "fd5d3644-7596-44e1-844f-fdc3c68b5021"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:10:17.701899 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:10:17.701870    4964 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fd5d3644-7596-44e1-844f-fdc3c68b5021-workload-certs\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:10:17.701899 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:10:17.701900    4964 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fd5d3644-7596-44e1-844f-fdc3c68b5021-istio-podinfo\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:10:17.702105 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:10:17.701914    4964 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fd5d3644-7596-44e1-844f-fdc3c68b5021-istio-envoy\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:10:17.702105 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:10:17.701928    4964 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fd5d3644-7596-44e1-844f-fdc3c68b5021-istio-token\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:10:17.702105 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:10:17.701941    4964 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd5d3644-7596-44e1-844f-fdc3c68b5021-workload-socket\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:10:17.702105 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:10:17.701954    4964 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hlbtc\" (UniqueName: \"kubernetes.io/projected/fd5d3644-7596-44e1-844f-fdc3c68b5021-kube-api-access-hlbtc\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:10:17.702105 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:10:17.701969    4964 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fd5d3644-7596-44e1-844f-fdc3c68b5021-istiod-ca-cert\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:10:18.164439 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:10:18.164409    4964 generic.go:356] "Generic (PLEG): container finished" podID="fd5d3644-7596-44e1-844f-fdc3c68b5021" containerID="6a604a4e640a88052a646705ff4256ad69d192ed642c00387523baa5279161f6" exitCode=0
Oct 31 02:10:18.164638 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:10:18.164469    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-7jd55" event={"ID":"fd5d3644-7596-44e1-844f-fdc3c68b5021","Type":"ContainerDied","Data":"6a604a4e640a88052a646705ff4256ad69d192ed642c00387523baa5279161f6"}
Oct 31 02:10:18.164638 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:10:18.164471    4964 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-7jd55"
Oct 31 02:10:18.164638 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:10:18.164508    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-7jd55" event={"ID":"fd5d3644-7596-44e1-844f-fdc3c68b5021","Type":"ContainerDied","Data":"1292dcba10258abc0014748ad6d7e9069b2eda01c02ef390c96a9b4925246d40"}
Oct 31 02:10:18.164638 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:10:18.164529    4964 scope.go:122] "RemoveContainer" containerID="6a604a4e640a88052a646705ff4256ad69d192ed642c00387523baa5279161f6"
Oct 31 02:10:18.174100 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:10:18.174080    4964 scope.go:122] "RemoveContainer" containerID="6a604a4e640a88052a646705ff4256ad69d192ed642c00387523baa5279161f6"
Oct 31 02:10:18.174451 ip-10-0-133-197 kubenswrapper[4964]: E1031 02:10:18.174417    4964 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6a604a4e640a88052a646705ff4256ad69d192ed642c00387523baa5279161f6\": container with ID starting with 6a604a4e640a88052a646705ff4256ad69d192ed642c00387523baa5279161f6 not found: ID does not exist" containerID="6a604a4e640a88052a646705ff4256ad69d192ed642c00387523baa5279161f6"
Oct 31 02:10:18.174571 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:10:18.174460    4964 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6a604a4e640a88052a646705ff4256ad69d192ed642c00387523baa5279161f6"} err="failed to get container status \"6a604a4e640a88052a646705ff4256ad69d192ed642c00387523baa5279161f6\": rpc error: code = NotFound desc = could not find container \"6a604a4e640a88052a646705ff4256ad69d192ed642c00387523baa5279161f6\": container with ID starting with 6a604a4e640a88052a646705ff4256ad69d192ed642c00387523baa5279161f6 not found: ID does not exist"
Oct 31 02:10:18.328243 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:10:18.328202    4964 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-7jd55"]
Oct 31 02:10:18.377647 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:10:18.377608    4964 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-7jd55"]
Oct 31 02:10:19.037956 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:10:19.037891    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 02:10:19.362935 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:10:19.362838    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 02:10:19.545778 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:10:19.545738    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 02:10:19.776514 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:10:19.776487    4964 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fd5d3644-7596-44e1-844f-fdc3c68b5021" path="/var/lib/kubelet/pods/fd5d3644-7596-44e1-844f-fdc3c68b5021/volumes"
Oct 31 02:10:19.875866 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:10:19.875833    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 02:10:22.535064 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:10:22.535031    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 02:10:22.720375 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:10:22.720339    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 02:10:23.015781 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:10:23.015750    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 02:10:24.181560 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:10:24.181521    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 02:10:24.363805 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:10:24.363771    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 02:10:24.792393 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:10:24.792363    4964 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-00e840/distinct-2-w2-00e840-kserve-6564ddcb89-cftvc"]
Oct 31 02:10:24.841748 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:10:24.841713    4964 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-00e840/premium-w2-00e840-kserve-6f6d876787-cv7w8"]
Oct 31 02:10:24.934875 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:10:24.934842    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 02:10:27.832216 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:10:27.832185    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 02:10:32.226216 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:10:32.226129    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 02:10:32.397647 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:10:32.397604    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 02:10:32.912775 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:10:32.912738    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 02:10:34.672509 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:10:34.672471    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 02:10:34.851937 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:10:34.851900    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 02:10:37.797989 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:10:37.797939    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 02:10:39.813337 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:10:39.813264    4964 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-00e840/distinct-2-w2-00e840-kserve-6564ddcb89-cftvc" podUID="facb04d4-8cd2-48aa-9b75-f165d7ca3f4a" containerName="main" containerID="cri-o://f3adde0f6d194273609102065a86275d9f95736af59ae08a21ff2f80280da2ba" gracePeriod=45
Oct 31 02:10:39.861092 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:10:39.861058    4964 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-00e840/premium-w2-00e840-kserve-6f6d876787-cv7w8" podUID="b8c12d5f-8ada-4c53-880c-446f5f0b5998" containerName="main" containerID="cri-o://3fd1dcaf3ee3ea29a6b8b325d1d184ff9277baa98504e22d1a089dae3b345baa" gracePeriod=45
Oct 31 02:10:39.975905 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:10:39.975868    4964 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-00e840/premium-w2-00e840-kserve-6f6d876787-cv7w8" podUID="b8c12d5f-8ada-4c53-880c-446f5f0b5998" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": dial tcp 10.132.0.73:8000: connect: connection refused"
Oct 31 02:10:40.121674 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:10:40.121644    4964 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-00e840/distinct-2-w2-00e840-kserve-6564ddcb89-cftvc"
Oct 31 02:10:40.153860 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:10:40.153832    4964 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-00e840/premium-w2-00e840-kserve-6f6d876787-cv7w8"
Oct 31 02:10:40.234177 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:10:40.234145    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/facb04d4-8cd2-48aa-9b75-f165d7ca3f4a-kube-api-access-27dw2\" (UniqueName: \"kubernetes.io/projected/facb04d4-8cd2-48aa-9b75-f165d7ca3f4a-kube-api-access-27dw2\") pod \"facb04d4-8cd2-48aa-9b75-f165d7ca3f4a\" (UID: \"facb04d4-8cd2-48aa-9b75-f165d7ca3f4a\") "
Oct 31 02:10:40.234404 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:10:40.234182    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b8c12d5f-8ada-4c53-880c-446f5f0b5998-kube-api-access-x6tqs\" (UniqueName: \"kubernetes.io/projected/b8c12d5f-8ada-4c53-880c-446f5f0b5998-kube-api-access-x6tqs\") pod \"b8c12d5f-8ada-4c53-880c-446f5f0b5998\" (UID: \"b8c12d5f-8ada-4c53-880c-446f5f0b5998\") "
Oct 31 02:10:40.234404 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:10:40.234220    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/facb04d4-8cd2-48aa-9b75-f165d7ca3f4a-home\" (UniqueName: \"kubernetes.io/empty-dir/facb04d4-8cd2-48aa-9b75-f165d7ca3f4a-home\") pod \"facb04d4-8cd2-48aa-9b75-f165d7ca3f4a\" (UID: \"facb04d4-8cd2-48aa-9b75-f165d7ca3f4a\") "
Oct 31 02:10:40.234404 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:10:40.234259    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/facb04d4-8cd2-48aa-9b75-f165d7ca3f4a-dshm\" (UniqueName: \"kubernetes.io/empty-dir/facb04d4-8cd2-48aa-9b75-f165d7ca3f4a-dshm\") pod \"facb04d4-8cd2-48aa-9b75-f165d7ca3f4a\" (UID: \"facb04d4-8cd2-48aa-9b75-f165d7ca3f4a\") "
Oct 31 02:10:40.234404 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:10:40.234295    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b8c12d5f-8ada-4c53-880c-446f5f0b5998-home\" (UniqueName: \"kubernetes.io/empty-dir/b8c12d5f-8ada-4c53-880c-446f5f0b5998-home\") pod \"b8c12d5f-8ada-4c53-880c-446f5f0b5998\" (UID: \"b8c12d5f-8ada-4c53-880c-446f5f0b5998\") "
Oct 31 02:10:40.234404 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:10:40.234358    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b8c12d5f-8ada-4c53-880c-446f5f0b5998-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b8c12d5f-8ada-4c53-880c-446f5f0b5998-model-cache\") pod \"b8c12d5f-8ada-4c53-880c-446f5f0b5998\" (UID: \"b8c12d5f-8ada-4c53-880c-446f5f0b5998\") "
Oct 31 02:10:40.234743 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:10:40.234465    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b8c12d5f-8ada-4c53-880c-446f5f0b5998-tls-certs\" (UniqueName: \"kubernetes.io/secret/b8c12d5f-8ada-4c53-880c-446f5f0b5998-tls-certs\") pod \"b8c12d5f-8ada-4c53-880c-446f5f0b5998\" (UID: \"b8c12d5f-8ada-4c53-880c-446f5f0b5998\") "
Oct 31 02:10:40.234743 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:10:40.234582    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/facb04d4-8cd2-48aa-9b75-f165d7ca3f4a-home" pod "facb04d4-8cd2-48aa-9b75-f165d7ca3f4a" (UID: "facb04d4-8cd2-48aa-9b75-f165d7ca3f4a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:10:40.234743 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:10:40.234632    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8c12d5f-8ada-4c53-880c-446f5f0b5998-home" pod "b8c12d5f-8ada-4c53-880c-446f5f0b5998" (UID: "b8c12d5f-8ada-4c53-880c-446f5f0b5998"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:10:40.234743 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:10:40.234714    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8c12d5f-8ada-4c53-880c-446f5f0b5998-model-cache" pod "b8c12d5f-8ada-4c53-880c-446f5f0b5998" (UID: "b8c12d5f-8ada-4c53-880c-446f5f0b5998"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:10:40.234975 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:10:40.234754    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b8c12d5f-8ada-4c53-880c-446f5f0b5998-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b8c12d5f-8ada-4c53-880c-446f5f0b5998-tmp-dir\") pod \"b8c12d5f-8ada-4c53-880c-446f5f0b5998\" (UID: \"b8c12d5f-8ada-4c53-880c-446f5f0b5998\") "
Oct 31 02:10:40.234975 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:10:40.234857    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/facb04d4-8cd2-48aa-9b75-f165d7ca3f4a-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/facb04d4-8cd2-48aa-9b75-f165d7ca3f4a-model-cache\") pod \"facb04d4-8cd2-48aa-9b75-f165d7ca3f4a\" (UID: \"facb04d4-8cd2-48aa-9b75-f165d7ca3f4a\") "
Oct 31 02:10:40.234975 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:10:40.234936    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8c12d5f-8ada-4c53-880c-446f5f0b5998-tmp-dir" pod "b8c12d5f-8ada-4c53-880c-446f5f0b5998" (UID: "b8c12d5f-8ada-4c53-880c-446f5f0b5998"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:10:40.237357 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:10:40.235079    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/facb04d4-8cd2-48aa-9b75-f165d7ca3f4a-model-cache" pod "facb04d4-8cd2-48aa-9b75-f165d7ca3f4a" (UID: "facb04d4-8cd2-48aa-9b75-f165d7ca3f4a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:10:40.237357 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:10:40.235168    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/facb04d4-8cd2-48aa-9b75-f165d7ca3f4a-tls-certs\" (UniqueName: \"kubernetes.io/secret/facb04d4-8cd2-48aa-9b75-f165d7ca3f4a-tls-certs\") pod \"facb04d4-8cd2-48aa-9b75-f165d7ca3f4a\" (UID: \"facb04d4-8cd2-48aa-9b75-f165d7ca3f4a\") "
Oct 31 02:10:40.237357 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:10:40.235222    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/facb04d4-8cd2-48aa-9b75-f165d7ca3f4a-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/facb04d4-8cd2-48aa-9b75-f165d7ca3f4a-tmp-dir\") pod \"facb04d4-8cd2-48aa-9b75-f165d7ca3f4a\" (UID: \"facb04d4-8cd2-48aa-9b75-f165d7ca3f4a\") "
Oct 31 02:10:40.237357 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:10:40.236373    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/facb04d4-8cd2-48aa-9b75-f165d7ca3f4a-tmp-dir" pod "facb04d4-8cd2-48aa-9b75-f165d7ca3f4a" (UID: "facb04d4-8cd2-48aa-9b75-f165d7ca3f4a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:10:40.237357 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:10:40.236410    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b8c12d5f-8ada-4c53-880c-446f5f0b5998-dshm\" (UniqueName: \"kubernetes.io/empty-dir/b8c12d5f-8ada-4c53-880c-446f5f0b5998-dshm\") pod \"b8c12d5f-8ada-4c53-880c-446f5f0b5998\" (UID: \"b8c12d5f-8ada-4c53-880c-446f5f0b5998\") "
Oct 31 02:10:40.237357 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:10:40.236790    4964 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b8c12d5f-8ada-4c53-880c-446f5f0b5998-home\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:10:40.237357 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:10:40.236808    4964 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b8c12d5f-8ada-4c53-880c-446f5f0b5998-model-cache\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:10:40.238361 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:10:40.238258    4964 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b8c12d5f-8ada-4c53-880c-446f5f0b5998-tmp-dir\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:10:40.238361 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:10:40.238285    4964 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/facb04d4-8cd2-48aa-9b75-f165d7ca3f4a-model-cache\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:10:40.238361 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:10:40.238301    4964 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/facb04d4-8cd2-48aa-9b75-f165d7ca3f4a-tmp-dir\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:10:40.238597 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:10:40.238528    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b8c12d5f-8ada-4c53-880c-446f5f0b5998-kube-api-access-x6tqs" pod "b8c12d5f-8ada-4c53-880c-446f5f0b5998" (UID: "b8c12d5f-8ada-4c53-880c-446f5f0b5998"). InnerVolumeSpecName "kube-api-access-x6tqs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:10:40.238693 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:10:40.238669    4964 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/facb04d4-8cd2-48aa-9b75-f165d7ca3f4a-home\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:10:40.238884 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:10:40.238855    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8c12d5f-8ada-4c53-880c-446f5f0b5998-dshm" pod "b8c12d5f-8ada-4c53-880c-446f5f0b5998" (UID: "b8c12d5f-8ada-4c53-880c-446f5f0b5998"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:10:40.239153 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:10:40.239123    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/facb04d4-8cd2-48aa-9b75-f165d7ca3f4a-kube-api-access-27dw2" pod "facb04d4-8cd2-48aa-9b75-f165d7ca3f4a" (UID: "facb04d4-8cd2-48aa-9b75-f165d7ca3f4a"). InnerVolumeSpecName "kube-api-access-27dw2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:10:40.240160 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:10:40.239481    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/facb04d4-8cd2-48aa-9b75-f165d7ca3f4a-dshm" pod "facb04d4-8cd2-48aa-9b75-f165d7ca3f4a" (UID: "facb04d4-8cd2-48aa-9b75-f165d7ca3f4a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:10:40.240506 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:10:40.240482    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b8c12d5f-8ada-4c53-880c-446f5f0b5998-tls-certs" pod "b8c12d5f-8ada-4c53-880c-446f5f0b5998" (UID: "b8c12d5f-8ada-4c53-880c-446f5f0b5998"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 02:10:40.240929 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:10:40.240900    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/facb04d4-8cd2-48aa-9b75-f165d7ca3f4a-tls-certs" pod "facb04d4-8cd2-48aa-9b75-f165d7ca3f4a" (UID: "facb04d4-8cd2-48aa-9b75-f165d7ca3f4a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 02:10:40.246571 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:10:40.246540    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 02:10:40.297903 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:10:40.297867    4964 generic.go:356] "Generic (PLEG): container finished" podID="facb04d4-8cd2-48aa-9b75-f165d7ca3f4a" containerID="f3adde0f6d194273609102065a86275d9f95736af59ae08a21ff2f80280da2ba" exitCode=0
Oct 31 02:10:40.298083 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:10:40.297919    4964 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-00e840/distinct-2-w2-00e840-kserve-6564ddcb89-cftvc"
Oct 31 02:10:40.298083 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:10:40.297958    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-00e840/distinct-2-w2-00e840-kserve-6564ddcb89-cftvc" event={"ID":"facb04d4-8cd2-48aa-9b75-f165d7ca3f4a","Type":"ContainerDied","Data":"f3adde0f6d194273609102065a86275d9f95736af59ae08a21ff2f80280da2ba"}
Oct 31 02:10:40.298083 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:10:40.298004    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-00e840/distinct-2-w2-00e840-kserve-6564ddcb89-cftvc" event={"ID":"facb04d4-8cd2-48aa-9b75-f165d7ca3f4a","Type":"ContainerDied","Data":"49f8c07fbd634fda9c2e1383f71442efb0d42a58f6a8823ae793c5330bd217ca"}
Oct 31 02:10:40.298083 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:10:40.298025    4964 scope.go:122] "RemoveContainer" containerID="f3adde0f6d194273609102065a86275d9f95736af59ae08a21ff2f80280da2ba"
Oct 31 02:10:40.299732 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:10:40.299706    4964 generic.go:356] "Generic (PLEG): container finished" podID="b8c12d5f-8ada-4c53-880c-446f5f0b5998" containerID="3fd1dcaf3ee3ea29a6b8b325d1d184ff9277baa98504e22d1a089dae3b345baa" exitCode=0
Oct 31 02:10:40.299847 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:10:40.299771    4964 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-00e840/premium-w2-00e840-kserve-6f6d876787-cv7w8"
Oct 31 02:10:40.299847 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:10:40.299826    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-00e840/premium-w2-00e840-kserve-6f6d876787-cv7w8" event={"ID":"b8c12d5f-8ada-4c53-880c-446f5f0b5998","Type":"ContainerDied","Data":"3fd1dcaf3ee3ea29a6b8b325d1d184ff9277baa98504e22d1a089dae3b345baa"}
Oct 31 02:10:40.299934 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:10:40.299855    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-00e840/premium-w2-00e840-kserve-6f6d876787-cv7w8" event={"ID":"b8c12d5f-8ada-4c53-880c-446f5f0b5998","Type":"ContainerDied","Data":"14575c0f4994b4384b6969b59771c8cda6f4b7295fa78094d091b57b9e9607ea"}
Oct 31 02:10:40.308945 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:10:40.308923    4964 scope.go:122] "RemoveContainer" containerID="f3adde0f6d194273609102065a86275d9f95736af59ae08a21ff2f80280da2ba"
Oct 31 02:10:40.309388 ip-10-0-133-197 kubenswrapper[4964]: E1031 02:10:40.309345    4964 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f3adde0f6d194273609102065a86275d9f95736af59ae08a21ff2f80280da2ba\": container with ID starting with f3adde0f6d194273609102065a86275d9f95736af59ae08a21ff2f80280da2ba not found: ID does not exist" containerID="f3adde0f6d194273609102065a86275d9f95736af59ae08a21ff2f80280da2ba"
Oct 31 02:10:40.309509 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:10:40.309393    4964 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f3adde0f6d194273609102065a86275d9f95736af59ae08a21ff2f80280da2ba"} err="failed to get container status \"f3adde0f6d194273609102065a86275d9f95736af59ae08a21ff2f80280da2ba\": rpc error: code = NotFound desc = could not find container \"f3adde0f6d194273609102065a86275d9f95736af59ae08a21ff2f80280da2ba\": container with ID starting with f3adde0f6d194273609102065a86275d9f95736af59ae08a21ff2f80280da2ba not found: ID does not exist"
Oct 31 02:10:40.309509 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:10:40.309414    4964 scope.go:122] "RemoveContainer" containerID="3fd1dcaf3ee3ea29a6b8b325d1d184ff9277baa98504e22d1a089dae3b345baa"
Oct 31 02:10:40.318575 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:10:40.318557    4964 scope.go:122] "RemoveContainer" containerID="3fd1dcaf3ee3ea29a6b8b325d1d184ff9277baa98504e22d1a089dae3b345baa"
Oct 31 02:10:40.318895 ip-10-0-133-197 kubenswrapper[4964]: E1031 02:10:40.318870    4964 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3fd1dcaf3ee3ea29a6b8b325d1d184ff9277baa98504e22d1a089dae3b345baa\": container with ID starting with 3fd1dcaf3ee3ea29a6b8b325d1d184ff9277baa98504e22d1a089dae3b345baa not found: ID does not exist" containerID="3fd1dcaf3ee3ea29a6b8b325d1d184ff9277baa98504e22d1a089dae3b345baa"
Oct 31 02:10:40.318995 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:10:40.318905    4964 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3fd1dcaf3ee3ea29a6b8b325d1d184ff9277baa98504e22d1a089dae3b345baa"} err="failed to get container status \"3fd1dcaf3ee3ea29a6b8b325d1d184ff9277baa98504e22d1a089dae3b345baa\": rpc error: code = NotFound desc = could not find container \"3fd1dcaf3ee3ea29a6b8b325d1d184ff9277baa98504e22d1a089dae3b345baa\": container with ID starting with 3fd1dcaf3ee3ea29a6b8b325d1d184ff9277baa98504e22d1a089dae3b345baa not found: ID does not exist"
Oct 31 02:10:40.339230 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:10:40.339152    4964 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b8c12d5f-8ada-4c53-880c-446f5f0b5998-tls-certs\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:10:40.339230 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:10:40.339177    4964 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/facb04d4-8cd2-48aa-9b75-f165d7ca3f4a-tls-certs\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:10:40.339230 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:10:40.339189    4964 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b8c12d5f-8ada-4c53-880c-446f5f0b5998-dshm\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:10:40.339230 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:10:40.339199    4964 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-27dw2\" (UniqueName: \"kubernetes.io/projected/facb04d4-8cd2-48aa-9b75-f165d7ca3f4a-kube-api-access-27dw2\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:10:40.339230 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:10:40.339209    4964 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x6tqs\" (UniqueName: \"kubernetes.io/projected/b8c12d5f-8ada-4c53-880c-446f5f0b5998-kube-api-access-x6tqs\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:10:40.339230 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:10:40.339217    4964 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/facb04d4-8cd2-48aa-9b75-f165d7ca3f4a-dshm\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:10:40.466427 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:10:40.466392    4964 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-00e840/distinct-2-w2-00e840-kserve-6564ddcb89-cftvc"]
Oct 31 02:10:40.468452 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:10:40.468426    4964 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-00e840/distinct-2-w2-00e840-kserve-6564ddcb89-cftvc"]
Oct 31 02:10:40.575798 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:10:40.575761    4964 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-00e840/premium-w2-00e840-kserve-6f6d876787-cv7w8"]
Oct 31 02:10:40.581348 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:10:40.581279    4964 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-00e840/premium-w2-00e840-kserve-6f6d876787-cv7w8"]
Oct 31 02:10:41.237505 ip-10-0-133-197 kubenswrapper[4964]: E1031 02:10:41.237467    4964 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q56x5" containerName="extractor"
Oct 31 02:10:41.777510 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:10:41.777481    4964 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b8c12d5f-8ada-4c53-880c-446f5f0b5998" path="/var/lib/kubelet/pods/b8c12d5f-8ada-4c53-880c-446f5f0b5998/volumes"
Oct 31 02:10:41.777921 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:10:41.777905    4964 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="facb04d4-8cd2-48aa-9b75-f165d7ca3f4a" path="/var/lib/kubelet/pods/facb04d4-8cd2-48aa-9b75-f165d7ca3f4a/volumes"
Oct 31 02:10:42.596471 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:10:42.596441    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 02:10:46.646662 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:10:46.646622    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 02:10:46.914860 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:10:46.914778    4964 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-71bc98-7fbd6bc889-zp8fz"]
Oct 31 02:10:46.915117 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:10:46.914998    4964 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-71bc98-7fbd6bc889-zp8fz" podUID="26611751-3a0b-488f-a67e-5a79bb448cbb" containerName="maas-api" containerID="cri-o://691a01e8e6ae5458f4153dc526e7deca689e0f73535fd4faa732ec66feaddc41" gracePeriod=30
Oct 31 02:10:47.061660 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:10:47.061622    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 02:10:47.179360 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:10:47.179288    4964 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-71bc98-7fbd6bc889-zp8fz"
Oct 31 02:10:47.314616 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:10:47.314586    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/26611751-3a0b-488f-a67e-5a79bb448cbb-metrics-certs\" (UniqueName: \"kubernetes.io/secret/26611751-3a0b-488f-a67e-5a79bb448cbb-metrics-certs\") pod \"26611751-3a0b-488f-a67e-5a79bb448cbb\" (UID: \"26611751-3a0b-488f-a67e-5a79bb448cbb\") "
Oct 31 02:10:47.314825 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:10:47.314645    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/26611751-3a0b-488f-a67e-5a79bb448cbb-kube-api-access-z8d6q\" (UniqueName: \"kubernetes.io/projected/26611751-3a0b-488f-a67e-5a79bb448cbb-kube-api-access-z8d6q\") pod \"26611751-3a0b-488f-a67e-5a79bb448cbb\" (UID: \"26611751-3a0b-488f-a67e-5a79bb448cbb\") "
Oct 31 02:10:47.314825 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:10:47.314734    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/26611751-3a0b-488f-a67e-5a79bb448cbb-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/26611751-3a0b-488f-a67e-5a79bb448cbb-openshift-service-ca\") pod \"26611751-3a0b-488f-a67e-5a79bb448cbb\" (UID: \"26611751-3a0b-488f-a67e-5a79bb448cbb\") "
Oct 31 02:10:47.314825 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:10:47.314813    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/26611751-3a0b-488f-a67e-5a79bb448cbb-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/26611751-3a0b-488f-a67e-5a79bb448cbb-maas-api-tls\") pod \"26611751-3a0b-488f-a67e-5a79bb448cbb\" (UID: \"26611751-3a0b-488f-a67e-5a79bb448cbb\") "
Oct 31 02:10:47.315171 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:10:47.315140    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/26611751-3a0b-488f-a67e-5a79bb448cbb-openshift-service-ca" pod "26611751-3a0b-488f-a67e-5a79bb448cbb" (UID: "26611751-3a0b-488f-a67e-5a79bb448cbb"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 02:10:47.317025 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:10:47.317002    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/26611751-3a0b-488f-a67e-5a79bb448cbb-metrics-certs" pod "26611751-3a0b-488f-a67e-5a79bb448cbb" (UID: "26611751-3a0b-488f-a67e-5a79bb448cbb"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 02:10:47.317152 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:10:47.317054    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/26611751-3a0b-488f-a67e-5a79bb448cbb-kube-api-access-z8d6q" pod "26611751-3a0b-488f-a67e-5a79bb448cbb" (UID: "26611751-3a0b-488f-a67e-5a79bb448cbb"). InnerVolumeSpecName "kube-api-access-z8d6q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:10:47.317593 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:10:47.317574    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/26611751-3a0b-488f-a67e-5a79bb448cbb-maas-api-tls" pod "26611751-3a0b-488f-a67e-5a79bb448cbb" (UID: "26611751-3a0b-488f-a67e-5a79bb448cbb"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 02:10:47.342348 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:10:47.342287    4964 generic.go:356] "Generic (PLEG): container finished" podID="26611751-3a0b-488f-a67e-5a79bb448cbb" containerID="691a01e8e6ae5458f4153dc526e7deca689e0f73535fd4faa732ec66feaddc41" exitCode=0
Oct 31 02:10:47.342516 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:10:47.342363    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-71bc98-7fbd6bc889-zp8fz" event={"ID":"26611751-3a0b-488f-a67e-5a79bb448cbb","Type":"ContainerDied","Data":"691a01e8e6ae5458f4153dc526e7deca689e0f73535fd4faa732ec66feaddc41"}
Oct 31 02:10:47.342516 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:10:47.342420    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-71bc98-7fbd6bc889-zp8fz" event={"ID":"26611751-3a0b-488f-a67e-5a79bb448cbb","Type":"ContainerDied","Data":"558e38fd130b73b9dcf3cb057b5dc9fc0f6151d5e9860ad7033fd563500d06dc"}
Oct 31 02:10:47.342516 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:10:47.342445    4964 scope.go:122] "RemoveContainer" containerID="691a01e8e6ae5458f4153dc526e7deca689e0f73535fd4faa732ec66feaddc41"
Oct 31 02:10:47.342516 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:10:47.342371    4964 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-71bc98-7fbd6bc889-zp8fz"
Oct 31 02:10:47.352891 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:10:47.352874    4964 scope.go:122] "RemoveContainer" containerID="691a01e8e6ae5458f4153dc526e7deca689e0f73535fd4faa732ec66feaddc41"
Oct 31 02:10:47.353156 ip-10-0-133-197 kubenswrapper[4964]: E1031 02:10:47.353133    4964 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"691a01e8e6ae5458f4153dc526e7deca689e0f73535fd4faa732ec66feaddc41\": container with ID starting with 691a01e8e6ae5458f4153dc526e7deca689e0f73535fd4faa732ec66feaddc41 not found: ID does not exist" containerID="691a01e8e6ae5458f4153dc526e7deca689e0f73535fd4faa732ec66feaddc41"
Oct 31 02:10:47.353258 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:10:47.353164    4964 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"691a01e8e6ae5458f4153dc526e7deca689e0f73535fd4faa732ec66feaddc41"} err="failed to get container status \"691a01e8e6ae5458f4153dc526e7deca689e0f73535fd4faa732ec66feaddc41\": rpc error: code = NotFound desc = could not find container \"691a01e8e6ae5458f4153dc526e7deca689e0f73535fd4faa732ec66feaddc41\": container with ID starting with 691a01e8e6ae5458f4153dc526e7deca689e0f73535fd4faa732ec66feaddc41 not found: ID does not exist"
Oct 31 02:10:47.416100 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:10:47.416071    4964 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/26611751-3a0b-488f-a67e-5a79bb448cbb-openshift-service-ca\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:10:47.416100 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:10:47.416097    4964 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/26611751-3a0b-488f-a67e-5a79bb448cbb-maas-api-tls\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:10:47.416100 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:10:47.416107    4964 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/26611751-3a0b-488f-a67e-5a79bb448cbb-metrics-certs\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:10:47.416364 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:10:47.416118    4964 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z8d6q\" (UniqueName: \"kubernetes.io/projected/26611751-3a0b-488f-a67e-5a79bb448cbb-kube-api-access-z8d6q\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:10:47.509022 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:10:47.508984    4964 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-71bc98-7fbd6bc889-zp8fz"]
Oct 31 02:10:47.513553 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:10:47.513526    4964 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-71bc98-7fbd6bc889-zp8fz"]
Oct 31 02:10:47.775491 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:10:47.775417    4964 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="26611751-3a0b-488f-a67e-5a79bb448cbb" path="/var/lib/kubelet/pods/26611751-3a0b-488f-a67e-5a79bb448cbb/volumes"
Oct 31 02:10:48.589531 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:10:48.589497    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 02:10:48.814009 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:10:48.813972    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 02:10:48.985748 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:10:48.985718    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 02:10:49.534896 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:10:49.534863    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 02:10:53.922395 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:10:53.922357    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 02:10:54.877558 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:10:54.877514    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 02:10:55.106680 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:10:55.106643    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 02:10:56.099029 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:10:56.098994    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 02:10:56.271270 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:10:56.271211    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 02:10:56.456090 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:10:56.455991    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 02:10:56.878250 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:10:56.878214    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 02:10:58.232987 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:10:58.232948    4964 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-00e840-58b9774b57-b6vbm"]
Oct 31 02:10:58.233500 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:10:58.233176    4964 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-00e840-58b9774b57-b6vbm" podUID="750dc732-c296-44cb-9f7f-a7f7be7b4144" containerName="maas-api" containerID="cri-o://c71c70024f09d0094139d714b4881cb0c42675bd9569bd1d03955b9c01ffcdf5" gracePeriod=30
Oct 31 02:10:58.407905 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:10:58.407868    4964 generic.go:356] "Generic (PLEG): container finished" podID="750dc732-c296-44cb-9f7f-a7f7be7b4144" containerID="c71c70024f09d0094139d714b4881cb0c42675bd9569bd1d03955b9c01ffcdf5" exitCode=0
Oct 31 02:10:58.408077 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:10:58.407925    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-00e840-58b9774b57-b6vbm" event={"ID":"750dc732-c296-44cb-9f7f-a7f7be7b4144","Type":"ContainerDied","Data":"c71c70024f09d0094139d714b4881cb0c42675bd9569bd1d03955b9c01ffcdf5"}
Oct 31 02:10:58.410501 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:10:58.410466    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 02:10:58.500115 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:10:58.500049    4964 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-00e840-58b9774b57-b6vbm"
Oct 31 02:10:58.529145 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:10:58.529115    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/750dc732-c296-44cb-9f7f-a7f7be7b4144-metrics-certs\" (UniqueName: \"kubernetes.io/secret/750dc732-c296-44cb-9f7f-a7f7be7b4144-metrics-certs\") pod \"750dc732-c296-44cb-9f7f-a7f7be7b4144\" (UID: \"750dc732-c296-44cb-9f7f-a7f7be7b4144\") "
Oct 31 02:10:58.529347 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:10:58.529214    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/750dc732-c296-44cb-9f7f-a7f7be7b4144-kube-api-access-dtwf6\" (UniqueName: \"kubernetes.io/projected/750dc732-c296-44cb-9f7f-a7f7be7b4144-kube-api-access-dtwf6\") pod \"750dc732-c296-44cb-9f7f-a7f7be7b4144\" (UID: \"750dc732-c296-44cb-9f7f-a7f7be7b4144\") "
Oct 31 02:10:58.529347 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:10:58.529265    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/750dc732-c296-44cb-9f7f-a7f7be7b4144-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/750dc732-c296-44cb-9f7f-a7f7be7b4144-maas-api-tls\") pod \"750dc732-c296-44cb-9f7f-a7f7be7b4144\" (UID: \"750dc732-c296-44cb-9f7f-a7f7be7b4144\") "
Oct 31 02:10:58.529347 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:10:58.529300    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/750dc732-c296-44cb-9f7f-a7f7be7b4144-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/750dc732-c296-44cb-9f7f-a7f7be7b4144-openshift-service-ca\") pod \"750dc732-c296-44cb-9f7f-a7f7be7b4144\" (UID: \"750dc732-c296-44cb-9f7f-a7f7be7b4144\") "
Oct 31 02:10:58.529785 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:10:58.529751    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/750dc732-c296-44cb-9f7f-a7f7be7b4144-openshift-service-ca" pod "750dc732-c296-44cb-9f7f-a7f7be7b4144" (UID: "750dc732-c296-44cb-9f7f-a7f7be7b4144"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 02:10:58.531425 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:10:58.531398    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/750dc732-c296-44cb-9f7f-a7f7be7b4144-metrics-certs" pod "750dc732-c296-44cb-9f7f-a7f7be7b4144" (UID: "750dc732-c296-44cb-9f7f-a7f7be7b4144"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 02:10:58.531542 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:10:58.531500    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/750dc732-c296-44cb-9f7f-a7f7be7b4144-kube-api-access-dtwf6" pod "750dc732-c296-44cb-9f7f-a7f7be7b4144" (UID: "750dc732-c296-44cb-9f7f-a7f7be7b4144"). InnerVolumeSpecName "kube-api-access-dtwf6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:10:58.531783 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:10:58.531764    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/750dc732-c296-44cb-9f7f-a7f7be7b4144-maas-api-tls" pod "750dc732-c296-44cb-9f7f-a7f7be7b4144" (UID: "750dc732-c296-44cb-9f7f-a7f7be7b4144"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 02:10:58.630841 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:10:58.630812    4964 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/750dc732-c296-44cb-9f7f-a7f7be7b4144-metrics-certs\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:10:58.630841 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:10:58.630839    4964 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dtwf6\" (UniqueName: \"kubernetes.io/projected/750dc732-c296-44cb-9f7f-a7f7be7b4144-kube-api-access-dtwf6\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:10:58.630841 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:10:58.630849    4964 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/750dc732-c296-44cb-9f7f-a7f7be7b4144-maas-api-tls\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:10:58.631108 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:10:58.630858    4964 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/750dc732-c296-44cb-9f7f-a7f7be7b4144-openshift-service-ca\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:10:59.415075 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:10:59.415043    4964 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-00e840-58b9774b57-b6vbm"
Oct 31 02:10:59.415075 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:10:59.415060    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-00e840-58b9774b57-b6vbm" event={"ID":"750dc732-c296-44cb-9f7f-a7f7be7b4144","Type":"ContainerDied","Data":"b378eb77f8fb2bb199ffb059a6d0ee57c8a8cc4dbbbb63de3f05b15148a7303f"}
Oct 31 02:10:59.415690 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:10:59.415126    4964 scope.go:122] "RemoveContainer" containerID="c71c70024f09d0094139d714b4881cb0c42675bd9569bd1d03955b9c01ffcdf5"
Oct 31 02:10:59.577743 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:10:59.577706    4964 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-00e840-58b9774b57-b6vbm"]
Oct 31 02:10:59.628278 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:10:59.628233    4964 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-00e840-58b9774b57-b6vbm"]
Oct 31 02:10:59.776801 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:10:59.776768    4964 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="750dc732-c296-44cb-9f7f-a7f7be7b4144" path="/var/lib/kubelet/pods/750dc732-c296-44cb-9f7f-a7f7be7b4144/volumes"
Oct 31 02:10:59.978332 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:10:59.978273    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 02:11:00.309019 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:11:00.308977    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 02:11:00.491822 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:11:00.491782    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 02:11:00.956048 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:11:00.956016    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 02:11:01.782065 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:11:01.782030    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 02:11:01.794584 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:11:01.794550    4964 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-baacee/distinct-w1-baacee-kserve-6c85cfb7d7-stkqq"]
Oct 31 02:11:01.841567 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:11:01.841528    4964 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-baacee/premium-w1-baacee-kserve-69cc8c6456-7ngq9"]
Oct 31 02:11:01.982074 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:11:01.982042    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 02:11:02.161076 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:11:02.160989    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 02:11:02.721039 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:11:02.721005    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 02:11:04.307225 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:11:04.307181    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 02:11:05.537799 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:11:05.537765    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 02:11:06.623741 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:11:06.623702    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 02:11:06.810676 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:11:06.810635    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 02:11:06.993358 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:11:06.993302    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 02:11:08.300339 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:11:08.300289    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 02:11:08.953748 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:11:08.953716    4964 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-baacee-6fcddb5455-j2fpn"]
Oct 31 02:11:09.248794 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:11:09.248752    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 02:11:09.467067 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:11:09.467031    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 02:11:11.242412 ip-10-0-133-197 kubenswrapper[4964]: E1031 02:11:11.242372    4964 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q56x5" containerName="extractor"
Oct 31 02:11:13.954850 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:11:13.954782    4964 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-baacee-6fcddb5455-j2fpn" podUID="d2bef5ff-d60c-4d04-ac18-dd98c3f92457" containerName="payload-pre-processing" containerID="cri-o://811d489d988679399ecc75e71dec6e369930bd163c8fd7618a5a39aaef54919c" gracePeriod=25
Oct 31 02:11:16.816527 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:11:16.816447    4964 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-baacee/distinct-w1-baacee-kserve-6c85cfb7d7-stkqq" podUID="df6d0bdd-382d-49b7-879c-15cb5371246e" containerName="main" containerID="cri-o://aee05ce5bac0b4342a425b1a06eabf8e95c84855bec79021dec94dccfc04eafb" gracePeriod=45
Oct 31 02:11:16.861901 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:11:16.861853    4964 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-baacee/premium-w1-baacee-kserve-69cc8c6456-7ngq9" podUID="15028ba0-c888-4780-b770-6acb2d03d70a" containerName="main" containerID="cri-o://d05e6b37a0478286c4234afe2c8f31db97c13a366644531daa3f5569b46a2c63" gracePeriod=45
Oct 31 02:11:16.987916 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:11:16.987872    4964 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-baacee-6fcddb5455-j2fpn" podUID="d2bef5ff-d60c-4d04-ac18-dd98c3f92457" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 02:11:17.143403 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:11:17.143377    4964 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-baacee/distinct-w1-baacee-kserve-6c85cfb7d7-stkqq"
Oct 31 02:11:17.146714 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:11:17.146688    4964 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-baacee/premium-w1-baacee-kserve-69cc8c6456-7ngq9"
Oct 31 02:11:17.227717 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:11:17.227679    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/15028ba0-c888-4780-b770-6acb2d03d70a-dshm\" (UniqueName: \"kubernetes.io/empty-dir/15028ba0-c888-4780-b770-6acb2d03d70a-dshm\") pod \"15028ba0-c888-4780-b770-6acb2d03d70a\" (UID: \"15028ba0-c888-4780-b770-6acb2d03d70a\") "
Oct 31 02:11:17.227930 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:11:17.227729    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/15028ba0-c888-4780-b770-6acb2d03d70a-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/15028ba0-c888-4780-b770-6acb2d03d70a-model-cache\") pod \"15028ba0-c888-4780-b770-6acb2d03d70a\" (UID: \"15028ba0-c888-4780-b770-6acb2d03d70a\") "
Oct 31 02:11:17.227930 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:11:17.227767    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/df6d0bdd-382d-49b7-879c-15cb5371246e-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/df6d0bdd-382d-49b7-879c-15cb5371246e-model-cache\") pod \"df6d0bdd-382d-49b7-879c-15cb5371246e\" (UID: \"df6d0bdd-382d-49b7-879c-15cb5371246e\") "
Oct 31 02:11:17.227930 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:11:17.227798    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/15028ba0-c888-4780-b770-6acb2d03d70a-home\" (UniqueName: \"kubernetes.io/empty-dir/15028ba0-c888-4780-b770-6acb2d03d70a-home\") pod \"15028ba0-c888-4780-b770-6acb2d03d70a\" (UID: \"15028ba0-c888-4780-b770-6acb2d03d70a\") "
Oct 31 02:11:17.227930 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:11:17.227861    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/15028ba0-c888-4780-b770-6acb2d03d70a-tls-certs\" (UniqueName: \"kubernetes.io/secret/15028ba0-c888-4780-b770-6acb2d03d70a-tls-certs\") pod \"15028ba0-c888-4780-b770-6acb2d03d70a\" (UID: \"15028ba0-c888-4780-b770-6acb2d03d70a\") "
Oct 31 02:11:17.227930 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:11:17.227895    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/df6d0bdd-382d-49b7-879c-15cb5371246e-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/df6d0bdd-382d-49b7-879c-15cb5371246e-tmp-dir\") pod \"df6d0bdd-382d-49b7-879c-15cb5371246e\" (UID: \"df6d0bdd-382d-49b7-879c-15cb5371246e\") "
Oct 31 02:11:17.227930 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:11:17.227925    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/15028ba0-c888-4780-b770-6acb2d03d70a-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/15028ba0-c888-4780-b770-6acb2d03d70a-tmp-dir\") pod \"15028ba0-c888-4780-b770-6acb2d03d70a\" (UID: \"15028ba0-c888-4780-b770-6acb2d03d70a\") "
Oct 31 02:11:17.228299 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:11:17.227970    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/df6d0bdd-382d-49b7-879c-15cb5371246e-dshm\" (UniqueName: \"kubernetes.io/empty-dir/df6d0bdd-382d-49b7-879c-15cb5371246e-dshm\") pod \"df6d0bdd-382d-49b7-879c-15cb5371246e\" (UID: \"df6d0bdd-382d-49b7-879c-15cb5371246e\") "
Oct 31 02:11:17.228299 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:11:17.228048    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/df6d0bdd-382d-49b7-879c-15cb5371246e-kube-api-access-wjk2j\" (UniqueName: \"kubernetes.io/projected/df6d0bdd-382d-49b7-879c-15cb5371246e-kube-api-access-wjk2j\") pod \"df6d0bdd-382d-49b7-879c-15cb5371246e\" (UID: \"df6d0bdd-382d-49b7-879c-15cb5371246e\") "
Oct 31 02:11:17.228299 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:11:17.228061    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15028ba0-c888-4780-b770-6acb2d03d70a-model-cache" pod "15028ba0-c888-4780-b770-6acb2d03d70a" (UID: "15028ba0-c888-4780-b770-6acb2d03d70a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:11:17.228299 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:11:17.228076    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df6d0bdd-382d-49b7-879c-15cb5371246e-model-cache" pod "df6d0bdd-382d-49b7-879c-15cb5371246e" (UID: "df6d0bdd-382d-49b7-879c-15cb5371246e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:11:17.228299 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:11:17.228074    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/df6d0bdd-382d-49b7-879c-15cb5371246e-home\" (UniqueName: \"kubernetes.io/empty-dir/df6d0bdd-382d-49b7-879c-15cb5371246e-home\") pod \"df6d0bdd-382d-49b7-879c-15cb5371246e\" (UID: \"df6d0bdd-382d-49b7-879c-15cb5371246e\") "
Oct 31 02:11:17.228299 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:11:17.228115    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15028ba0-c888-4780-b770-6acb2d03d70a-home" pod "15028ba0-c888-4780-b770-6acb2d03d70a" (UID: "15028ba0-c888-4780-b770-6acb2d03d70a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:11:17.228299 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:11:17.228183    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/df6d0bdd-382d-49b7-879c-15cb5371246e-tls-certs\" (UniqueName: \"kubernetes.io/secret/df6d0bdd-382d-49b7-879c-15cb5371246e-tls-certs\") pod \"df6d0bdd-382d-49b7-879c-15cb5371246e\" (UID: \"df6d0bdd-382d-49b7-879c-15cb5371246e\") "
Oct 31 02:11:17.228299 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:11:17.228220    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/15028ba0-c888-4780-b770-6acb2d03d70a-kube-api-access-j8mwp\" (UniqueName: \"kubernetes.io/projected/15028ba0-c888-4780-b770-6acb2d03d70a-kube-api-access-j8mwp\") pod \"15028ba0-c888-4780-b770-6acb2d03d70a\" (UID: \"15028ba0-c888-4780-b770-6acb2d03d70a\") "
Oct 31 02:11:17.228299 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:11:17.228263    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df6d0bdd-382d-49b7-879c-15cb5371246e-tmp-dir" pod "df6d0bdd-382d-49b7-879c-15cb5371246e" (UID: "df6d0bdd-382d-49b7-879c-15cb5371246e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:11:17.228867 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:11:17.228277    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df6d0bdd-382d-49b7-879c-15cb5371246e-home" pod "df6d0bdd-382d-49b7-879c-15cb5371246e" (UID: "df6d0bdd-382d-49b7-879c-15cb5371246e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:11:17.228867 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:11:17.228393    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15028ba0-c888-4780-b770-6acb2d03d70a-tmp-dir" pod "15028ba0-c888-4780-b770-6acb2d03d70a" (UID: "15028ba0-c888-4780-b770-6acb2d03d70a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:11:17.228867 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:11:17.228715    4964 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/15028ba0-c888-4780-b770-6acb2d03d70a-model-cache\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:11:17.228867 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:11:17.228784    4964 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/df6d0bdd-382d-49b7-879c-15cb5371246e-model-cache\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:11:17.228867 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:11:17.228804    4964 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/15028ba0-c888-4780-b770-6acb2d03d70a-home\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:11:17.231111 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:11:17.231078    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/15028ba0-c888-4780-b770-6acb2d03d70a-tls-certs" pod "15028ba0-c888-4780-b770-6acb2d03d70a" (UID: "15028ba0-c888-4780-b770-6acb2d03d70a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 02:11:17.231796 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:11:17.231443    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15028ba0-c888-4780-b770-6acb2d03d70a-dshm" pod "15028ba0-c888-4780-b770-6acb2d03d70a" (UID: "15028ba0-c888-4780-b770-6acb2d03d70a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:11:17.231796 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:11:17.231561    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/df6d0bdd-382d-49b7-879c-15cb5371246e-tls-certs" pod "df6d0bdd-382d-49b7-879c-15cb5371246e" (UID: "df6d0bdd-382d-49b7-879c-15cb5371246e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 02:11:17.231796 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:11:17.231590    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df6d0bdd-382d-49b7-879c-15cb5371246e-kube-api-access-wjk2j" pod "df6d0bdd-382d-49b7-879c-15cb5371246e" (UID: "df6d0bdd-382d-49b7-879c-15cb5371246e"). InnerVolumeSpecName "kube-api-access-wjk2j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:11:17.231796 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:11:17.231611    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/15028ba0-c888-4780-b770-6acb2d03d70a-kube-api-access-j8mwp" pod "15028ba0-c888-4780-b770-6acb2d03d70a" (UID: "15028ba0-c888-4780-b770-6acb2d03d70a"). InnerVolumeSpecName "kube-api-access-j8mwp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:11:17.232058 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:11:17.231977    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df6d0bdd-382d-49b7-879c-15cb5371246e-dshm" pod "df6d0bdd-382d-49b7-879c-15cb5371246e" (UID: "df6d0bdd-382d-49b7-879c-15cb5371246e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:11:17.329652 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:11:17.329611    4964 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/15028ba0-c888-4780-b770-6acb2d03d70a-tls-certs\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:11:17.329652 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:11:17.329651    4964 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/df6d0bdd-382d-49b7-879c-15cb5371246e-tmp-dir\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:11:17.329882 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:11:17.329666    4964 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/15028ba0-c888-4780-b770-6acb2d03d70a-tmp-dir\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:11:17.329882 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:11:17.329680    4964 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/df6d0bdd-382d-49b7-879c-15cb5371246e-dshm\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:11:17.329882 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:11:17.329696    4964 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wjk2j\" (UniqueName: \"kubernetes.io/projected/df6d0bdd-382d-49b7-879c-15cb5371246e-kube-api-access-wjk2j\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:11:17.329882 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:11:17.329711    4964 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/df6d0bdd-382d-49b7-879c-15cb5371246e-home\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:11:17.329882 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:11:17.329724    4964 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/df6d0bdd-382d-49b7-879c-15cb5371246e-tls-certs\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:11:17.329882 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:11:17.329739    4964 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j8mwp\" (UniqueName: \"kubernetes.io/projected/15028ba0-c888-4780-b770-6acb2d03d70a-kube-api-access-j8mwp\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:11:17.329882 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:11:17.329752    4964 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/15028ba0-c888-4780-b770-6acb2d03d70a-dshm\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:11:17.517274 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:11:17.517240    4964 generic.go:356] "Generic (PLEG): container finished" podID="df6d0bdd-382d-49b7-879c-15cb5371246e" containerID="aee05ce5bac0b4342a425b1a06eabf8e95c84855bec79021dec94dccfc04eafb" exitCode=0
Oct 31 02:11:17.517526 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:11:17.517287    4964 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-baacee/distinct-w1-baacee-kserve-6c85cfb7d7-stkqq"
Oct 31 02:11:17.517526 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:11:17.517360    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-baacee/distinct-w1-baacee-kserve-6c85cfb7d7-stkqq" event={"ID":"df6d0bdd-382d-49b7-879c-15cb5371246e","Type":"ContainerDied","Data":"aee05ce5bac0b4342a425b1a06eabf8e95c84855bec79021dec94dccfc04eafb"}
Oct 31 02:11:17.517526 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:11:17.517405    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-baacee/distinct-w1-baacee-kserve-6c85cfb7d7-stkqq" event={"ID":"df6d0bdd-382d-49b7-879c-15cb5371246e","Type":"ContainerDied","Data":"d7452c10094d65664e60d3e34fdff39bbf5c56880954582560bd8ab989e6d418"}
Oct 31 02:11:17.517526 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:11:17.517429    4964 scope.go:122] "RemoveContainer" containerID="aee05ce5bac0b4342a425b1a06eabf8e95c84855bec79021dec94dccfc04eafb"
Oct 31 02:11:17.518753 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:11:17.518731    4964 generic.go:356] "Generic (PLEG): container finished" podID="15028ba0-c888-4780-b770-6acb2d03d70a" containerID="d05e6b37a0478286c4234afe2c8f31db97c13a366644531daa3f5569b46a2c63" exitCode=0
Oct 31 02:11:17.518898 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:11:17.518791    4964 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-baacee/premium-w1-baacee-kserve-69cc8c6456-7ngq9"
Oct 31 02:11:17.518898 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:11:17.518823    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-baacee/premium-w1-baacee-kserve-69cc8c6456-7ngq9" event={"ID":"15028ba0-c888-4780-b770-6acb2d03d70a","Type":"ContainerDied","Data":"d05e6b37a0478286c4234afe2c8f31db97c13a366644531daa3f5569b46a2c63"}
Oct 31 02:11:17.518898 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:11:17.518864    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-baacee/premium-w1-baacee-kserve-69cc8c6456-7ngq9" event={"ID":"15028ba0-c888-4780-b770-6acb2d03d70a","Type":"ContainerDied","Data":"188ad780121fb18f71f1a0cddf3b70c8d3365e501e55a1e2c993554d84e5b1b2"}
Oct 31 02:11:17.527655 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:11:17.527636    4964 scope.go:122] "RemoveContainer" containerID="aee05ce5bac0b4342a425b1a06eabf8e95c84855bec79021dec94dccfc04eafb"
Oct 31 02:11:17.527999 ip-10-0-133-197 kubenswrapper[4964]: E1031 02:11:17.527979    4964 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aee05ce5bac0b4342a425b1a06eabf8e95c84855bec79021dec94dccfc04eafb\": container with ID starting with aee05ce5bac0b4342a425b1a06eabf8e95c84855bec79021dec94dccfc04eafb not found: ID does not exist" containerID="aee05ce5bac0b4342a425b1a06eabf8e95c84855bec79021dec94dccfc04eafb"
Oct 31 02:11:17.528098 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:11:17.528009    4964 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aee05ce5bac0b4342a425b1a06eabf8e95c84855bec79021dec94dccfc04eafb"} err="failed to get container status \"aee05ce5bac0b4342a425b1a06eabf8e95c84855bec79021dec94dccfc04eafb\": rpc error: code = NotFound desc = could not find container \"aee05ce5bac0b4342a425b1a06eabf8e95c84855bec79021dec94dccfc04eafb\": container with ID starting with aee05ce5bac0b4342a425b1a06eabf8e95c84855bec79021dec94dccfc04eafb not found: ID does not exist"
Oct 31 02:11:17.528098 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:11:17.528025    4964 scope.go:122] "RemoveContainer" containerID="d05e6b37a0478286c4234afe2c8f31db97c13a366644531daa3f5569b46a2c63"
Oct 31 02:11:17.537247 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:11:17.537226    4964 scope.go:122] "RemoveContainer" containerID="d05e6b37a0478286c4234afe2c8f31db97c13a366644531daa3f5569b46a2c63"
Oct 31 02:11:17.537585 ip-10-0-133-197 kubenswrapper[4964]: E1031 02:11:17.537568    4964 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d05e6b37a0478286c4234afe2c8f31db97c13a366644531daa3f5569b46a2c63\": container with ID starting with d05e6b37a0478286c4234afe2c8f31db97c13a366644531daa3f5569b46a2c63 not found: ID does not exist" containerID="d05e6b37a0478286c4234afe2c8f31db97c13a366644531daa3f5569b46a2c63"
Oct 31 02:11:17.537760 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:11:17.537596    4964 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d05e6b37a0478286c4234afe2c8f31db97c13a366644531daa3f5569b46a2c63"} err="failed to get container status \"d05e6b37a0478286c4234afe2c8f31db97c13a366644531daa3f5569b46a2c63\": rpc error: code = NotFound desc = could not find container \"d05e6b37a0478286c4234afe2c8f31db97c13a366644531daa3f5569b46a2c63\": container with ID starting with d05e6b37a0478286c4234afe2c8f31db97c13a366644531daa3f5569b46a2c63 not found: ID does not exist"
Oct 31 02:11:17.679584 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:11:17.679544    4964 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-baacee/premium-w1-baacee-kserve-69cc8c6456-7ngq9"]
Oct 31 02:11:17.681556 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:11:17.681535    4964 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-baacee/premium-w1-baacee-kserve-69cc8c6456-7ngq9"]
Oct 31 02:11:17.735670 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:11:17.735628    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 02:11:17.777394 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:11:17.777297    4964 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="15028ba0-c888-4780-b770-6acb2d03d70a" path="/var/lib/kubelet/pods/15028ba0-c888-4780-b770-6acb2d03d70a/volumes"
Oct 31 02:11:17.787962 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:11:17.787924    4964 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-baacee/distinct-w1-baacee-kserve-6c85cfb7d7-stkqq"]
Oct 31 02:11:17.789737 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:11:17.789718    4964 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-baacee/distinct-w1-baacee-kserve-6c85cfb7d7-stkqq"]
Oct 31 02:11:19.775640 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:11:19.775606    4964 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="df6d0bdd-382d-49b7-879c-15cb5371246e" path="/var/lib/kubelet/pods/df6d0bdd-382d-49b7-879c-15cb5371246e/volumes"
Oct 31 02:11:22.598718 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:11:22.598685    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 02:11:24.379057 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:11:24.379021    4964 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-baacee-77bdffbd-4vbp7"]
Oct 31 02:11:24.379522 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:11:24.379245    4964 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-baacee-77bdffbd-4vbp7" podUID="803fb9d4-2f24-48f3-bea0-f62833d91796" containerName="maas-api" containerID="cri-o://19c7ec33e5e6f13b92f9de6dee708277d3f915fe3cc9784e7782e2777f5b4be1" gracePeriod=30
Oct 31 02:11:24.548193 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:11:24.548157    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 02:11:24.561981 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:11:24.561944    4964 generic.go:356] "Generic (PLEG): container finished" podID="803fb9d4-2f24-48f3-bea0-f62833d91796" containerID="19c7ec33e5e6f13b92f9de6dee708277d3f915fe3cc9784e7782e2777f5b4be1" exitCode=0
Oct 31 02:11:24.562116 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:11:24.562075    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-baacee-77bdffbd-4vbp7" event={"ID":"803fb9d4-2f24-48f3-bea0-f62833d91796","Type":"ContainerDied","Data":"19c7ec33e5e6f13b92f9de6dee708277d3f915fe3cc9784e7782e2777f5b4be1"}
Oct 31 02:11:24.643867 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:11:24.643805    4964 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-baacee-77bdffbd-4vbp7"
Oct 31 02:11:24.801258 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:11:24.801227    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/803fb9d4-2f24-48f3-bea0-f62833d91796-metrics-certs\" (UniqueName: \"kubernetes.io/secret/803fb9d4-2f24-48f3-bea0-f62833d91796-metrics-certs\") pod \"803fb9d4-2f24-48f3-bea0-f62833d91796\" (UID: \"803fb9d4-2f24-48f3-bea0-f62833d91796\") "
Oct 31 02:11:24.801457 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:11:24.801287    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/803fb9d4-2f24-48f3-bea0-f62833d91796-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/803fb9d4-2f24-48f3-bea0-f62833d91796-openshift-service-ca\") pod \"803fb9d4-2f24-48f3-bea0-f62833d91796\" (UID: \"803fb9d4-2f24-48f3-bea0-f62833d91796\") "
Oct 31 02:11:24.801457 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:11:24.801424    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/803fb9d4-2f24-48f3-bea0-f62833d91796-kube-api-access-t6vvh\" (UniqueName: \"kubernetes.io/projected/803fb9d4-2f24-48f3-bea0-f62833d91796-kube-api-access-t6vvh\") pod \"803fb9d4-2f24-48f3-bea0-f62833d91796\" (UID: \"803fb9d4-2f24-48f3-bea0-f62833d91796\") "
Oct 31 02:11:24.801581 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:11:24.801475    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/803fb9d4-2f24-48f3-bea0-f62833d91796-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/803fb9d4-2f24-48f3-bea0-f62833d91796-maas-api-tls\") pod \"803fb9d4-2f24-48f3-bea0-f62833d91796\" (UID: \"803fb9d4-2f24-48f3-bea0-f62833d91796\") "
Oct 31 02:11:24.801695 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:11:24.801672    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/803fb9d4-2f24-48f3-bea0-f62833d91796-openshift-service-ca" pod "803fb9d4-2f24-48f3-bea0-f62833d91796" (UID: "803fb9d4-2f24-48f3-bea0-f62833d91796"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 02:11:24.801796 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:11:24.801780    4964 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/803fb9d4-2f24-48f3-bea0-f62833d91796-openshift-service-ca\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:11:24.803605 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:11:24.803581    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/803fb9d4-2f24-48f3-bea0-f62833d91796-metrics-certs" pod "803fb9d4-2f24-48f3-bea0-f62833d91796" (UID: "803fb9d4-2f24-48f3-bea0-f62833d91796"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 02:11:24.803709 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:11:24.803694    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/803fb9d4-2f24-48f3-bea0-f62833d91796-maas-api-tls" pod "803fb9d4-2f24-48f3-bea0-f62833d91796" (UID: "803fb9d4-2f24-48f3-bea0-f62833d91796"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 02:11:24.803817 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:11:24.803793    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/803fb9d4-2f24-48f3-bea0-f62833d91796-kube-api-access-t6vvh" pod "803fb9d4-2f24-48f3-bea0-f62833d91796" (UID: "803fb9d4-2f24-48f3-bea0-f62833d91796"). InnerVolumeSpecName "kube-api-access-t6vvh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:11:24.902789 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:11:24.902680    4964 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/803fb9d4-2f24-48f3-bea0-f62833d91796-metrics-certs\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:11:24.902789 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:11:24.902724    4964 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t6vvh\" (UniqueName: \"kubernetes.io/projected/803fb9d4-2f24-48f3-bea0-f62833d91796-kube-api-access-t6vvh\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:11:24.902789 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:11:24.902740    4964 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/803fb9d4-2f24-48f3-bea0-f62833d91796-maas-api-tls\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:11:25.568933 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:11:25.568899    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-baacee-77bdffbd-4vbp7" event={"ID":"803fb9d4-2f24-48f3-bea0-f62833d91796","Type":"ContainerDied","Data":"903640ff765eb7eb63d0197ca2baef33ed0abf54633c4b1381c508fdb78ebea2"}
Oct 31 02:11:25.568933 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:11:25.568928    4964 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-baacee-77bdffbd-4vbp7"
Oct 31 02:11:25.569441 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:11:25.568954    4964 scope.go:122] "RemoveContainer" containerID="19c7ec33e5e6f13b92f9de6dee708277d3f915fe3cc9784e7782e2777f5b4be1"
Oct 31 02:11:25.731619 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:11:25.731584    4964 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-baacee-77bdffbd-4vbp7"]
Oct 31 02:11:25.781277 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:11:25.781241    4964 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-baacee-77bdffbd-4vbp7"]
Oct 31 02:11:26.049253 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:11:26.049215    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 02:11:26.342667 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:11:26.342577    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 02:11:26.987908 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:11:26.987866    4964 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-baacee-6fcddb5455-j2fpn" podUID="d2bef5ff-d60c-4d04-ac18-dd98c3f92457" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 02:11:27.017024 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:11:27.016994    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 02:11:27.775891 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:11:27.775853    4964 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="803fb9d4-2f24-48f3-bea0-f62833d91796" path="/var/lib/kubelet/pods/803fb9d4-2f24-48f3-bea0-f62833d91796/volumes"
Oct 31 02:11:32.916267 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:11:32.916227    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 02:11:33.087171 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:11:33.087139    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 02:11:33.291592 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:11:33.291551    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 02:11:34.207966 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:11:34.207936    4964 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-baacee-6fcddb5455-j2fpn"
Oct 31 02:11:34.304552 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:11:34.304503    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d2bef5ff-d60c-4d04-ac18-dd98c3f92457-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d2bef5ff-d60c-4d04-ac18-dd98c3f92457-plugins-config-volume\") pod \"d2bef5ff-d60c-4d04-ac18-dd98c3f92457\" (UID: \"d2bef5ff-d60c-4d04-ac18-dd98c3f92457\") "
Oct 31 02:11:34.304552 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:11:34.304556    4964 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d2bef5ff-d60c-4d04-ac18-dd98c3f92457-kube-api-access-n6bdg\" (UniqueName: \"kubernetes.io/projected/d2bef5ff-d60c-4d04-ac18-dd98c3f92457-kube-api-access-n6bdg\") pod \"d2bef5ff-d60c-4d04-ac18-dd98c3f92457\" (UID: \"d2bef5ff-d60c-4d04-ac18-dd98c3f92457\") "
Oct 31 02:11:34.304916 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:11:34.304894    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d2bef5ff-d60c-4d04-ac18-dd98c3f92457-plugins-config-volume" pod "d2bef5ff-d60c-4d04-ac18-dd98c3f92457" (UID: "d2bef5ff-d60c-4d04-ac18-dd98c3f92457"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 02:11:34.306917 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:11:34.306893    4964 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d2bef5ff-d60c-4d04-ac18-dd98c3f92457-kube-api-access-n6bdg" pod "d2bef5ff-d60c-4d04-ac18-dd98c3f92457" (UID: "d2bef5ff-d60c-4d04-ac18-dd98c3f92457"). InnerVolumeSpecName "kube-api-access-n6bdg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:11:34.405722 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:11:34.405668    4964 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d2bef5ff-d60c-4d04-ac18-dd98c3f92457-plugins-config-volume\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:11:34.405722 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:11:34.405720    4964 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n6bdg\" (UniqueName: \"kubernetes.io/projected/d2bef5ff-d60c-4d04-ac18-dd98c3f92457-kube-api-access-n6bdg\") on node \"ip-10-0-133-197.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:11:34.623192 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:11:34.623098    4964 generic.go:356] "Generic (PLEG): container finished" podID="d2bef5ff-d60c-4d04-ac18-dd98c3f92457" containerID="811d489d988679399ecc75e71dec6e369930bd163c8fd7618a5a39aaef54919c" exitCode=0
Oct 31 02:11:34.623192 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:11:34.623162    4964 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-baacee-6fcddb5455-j2fpn"
Oct 31 02:11:34.623420 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:11:34.623190    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-baacee-6fcddb5455-j2fpn" event={"ID":"d2bef5ff-d60c-4d04-ac18-dd98c3f92457","Type":"ContainerDied","Data":"811d489d988679399ecc75e71dec6e369930bd163c8fd7618a5a39aaef54919c"}
Oct 31 02:11:34.623420 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:11:34.623278    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-baacee-6fcddb5455-j2fpn" event={"ID":"d2bef5ff-d60c-4d04-ac18-dd98c3f92457","Type":"ContainerDied","Data":"8d01963238b03719f8ad233d2b6abaa7876b92d3e155d4c70e97596d1bebbfbe"}
Oct 31 02:11:34.623420 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:11:34.623306    4964 scope.go:122] "RemoveContainer" containerID="811d489d988679399ecc75e71dec6e369930bd163c8fd7618a5a39aaef54919c"
Oct 31 02:11:34.633795 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:11:34.633775    4964 scope.go:122] "RemoveContainer" containerID="811d489d988679399ecc75e71dec6e369930bd163c8fd7618a5a39aaef54919c"
Oct 31 02:11:34.634086 ip-10-0-133-197 kubenswrapper[4964]: E1031 02:11:34.634065    4964 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"811d489d988679399ecc75e71dec6e369930bd163c8fd7618a5a39aaef54919c\": container with ID starting with 811d489d988679399ecc75e71dec6e369930bd163c8fd7618a5a39aaef54919c not found: ID does not exist" containerID="811d489d988679399ecc75e71dec6e369930bd163c8fd7618a5a39aaef54919c"
Oct 31 02:11:34.634159 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:11:34.634096    4964 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"811d489d988679399ecc75e71dec6e369930bd163c8fd7618a5a39aaef54919c"} err="failed to get container status \"811d489d988679399ecc75e71dec6e369930bd163c8fd7618a5a39aaef54919c\": rpc error: code = NotFound desc = could not find container \"811d489d988679399ecc75e71dec6e369930bd163c8fd7618a5a39aaef54919c\": container with ID starting with 811d489d988679399ecc75e71dec6e369930bd163c8fd7618a5a39aaef54919c not found: ID does not exist"
Oct 31 02:11:34.784682 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:11:34.784643    4964 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-baacee-6fcddb5455-j2fpn"]
Oct 31 02:11:34.834345 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:11:34.834301    4964 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-baacee-6fcddb5455-j2fpn"]
Oct 31 02:11:35.230820 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:11:35.230793    4964 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-2nnh7_9b628e58-c287-4716-adbb-6bbdf0dad9fb/manager/0.log"
Oct 31 02:11:35.776450 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:11:35.776411    4964 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d2bef5ff-d60c-4d04-ac18-dd98c3f92457" path="/var/lib/kubelet/pods/d2bef5ff-d60c-4d04-ac18-dd98c3f92457/volumes"
Oct 31 02:11:36.893012 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:11:36.892969    4964 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-9mrpk_0446b6be-06eb-4fde-858b-a36f492f2209/manager/0.log"
Oct 31 02:11:37.365194 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:11:37.365154    4964 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-svrmp_25e6b2b1-a331-483e-86e8-8585418a8e37/manager/0.log"
Oct 31 02:11:38.201606 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:11:38.201576    4964 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-58f5dc44b9-c98xl_cf528aec-95b1-46dd-8f6c-da3ac6cc85d9/manager/0.log"
Oct 31 02:11:38.230233 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:11:38.230197    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 02:11:38.408673 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:11:38.408630    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 02:11:39.374789 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:11:39.374753    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 02:11:40.224917 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:11:40.224885    4964 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-z94np_baeff60c-ce67-45a0-a76b-52220ba1065f/manager/0.log"
Oct 31 02:11:41.244792 ip-10-0-133-197 kubenswrapper[4964]: E1031 02:11:41.244761    4964 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q56x5" containerName="extractor"
Oct 31 02:11:44.903517 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:11:44.903480    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 02:11:47.652118 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:11:47.652078    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 02:11:52.157476 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:11:52.157440    4964 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-58f5dc44b9-c98xl_cf528aec-95b1-46dd-8f6c-da3ac6cc85d9/manager/0.log"
Oct 31 02:11:54.200964 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:11:54.200927    4964 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-z94np_baeff60c-ce67-45a0-a76b-52220ba1065f/manager/0.log"
Oct 31 02:12:00.676334 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:12:00.676221    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 02:12:00.757994 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:12:00.757946    4964 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-2nnh7_9b628e58-c287-4716-adbb-6bbdf0dad9fb/manager/0.log"
Oct 31 02:12:11.242567 ip-10-0-133-197 kubenswrapper[4964]: E1031 02:12:11.242523    4964 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q56x5" containerName="extractor"
Oct 31 02:12:12.133511 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:12:12.133477    4964 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-9mrpk_0446b6be-06eb-4fde-858b-a36f492f2209/manager/0.log"
Oct 31 02:12:22.612210 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:12:22.612170    4964 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-svrmp_25e6b2b1-a331-483e-86e8-8585418a8e37/manager/0.log"
Oct 31 02:12:22.941615 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:12:22.941524    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 02:12:23.594305 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:12:23.594270    4964 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6b748b6585wg7kb_5b65b0b2-79a7-4116-a40e-78e1414c0071/istio-proxy/0.log"
Oct 31 02:12:25.528032 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:12:25.527994    4964 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-tlb5m_c8461fdb-12e5-4691-8b95-139d5a901f0d/payload-processing/0.log"
Oct 31 02:12:25.758631 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:12:25.758598    4964 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-5964fff8f9-8k6pp_e49453a8-c7d9-4248-a8ef-4a819b76806c/router/0.log"
Oct 31 02:12:26.992070 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:12:26.992042    4964 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-7b998b79d9-dls47_a90b1b97-ef44-4b3c-9c5f-b1b8a79dd356/main/0.log"
Oct 31 02:12:27.501554 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:12:27.501525    4964 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-69b588897d-x8krt_59416eef-55d7-4df2-8f28-a64f69ea4568/main/0.log"
Oct 31 02:12:27.980832 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:12:27.980799    4964 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_echo-simulator-kserve-64b6f8f96c-2p64m_b2dfcad6-e2bb-4e79-aa45-9f53c989349d/main/0.log"
Oct 31 02:12:28.215662 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:12:28.215632    4964 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-84b4ff656c-76tzr_a10752ec-71ed-44a2-a3d0-e6f5ad807ce1/main/0.log"
Oct 31 02:12:41.237772 ip-10-0-133-197 kubenswrapper[4964]: E1031 02:12:41.237730    4964 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q56x5" containerName="extractor"
Oct 31 02:12:48.840518 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:12:48.840451    4964 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-7lvdn/must-gather-b779n"]
Oct 31 02:12:48.840994 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:12:48.840967    4964 cpu_manager.go:414] "containerMap: removing container" podUID="c7eddf8c-a855-4a72-b3ed-eacf3aaed5af" containerName="istio-proxy"
Oct 31 02:12:48.840994 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:12:48.840983    4964 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c7eddf8c-a855-4a72-b3ed-eacf3aaed5af" containerName="istio-proxy"
Oct 31 02:12:48.840994 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:12:48.840992    4964 cpu_manager.go:414] "containerMap: removing container" podUID="ba3f39ff-dde5-4b50-b624-d3144586f11f" containerName="payload-pre-processing"
Oct 31 02:12:48.841105 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:12:48.840997    4964 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ba3f39ff-dde5-4b50-b624-d3144586f11f" containerName="payload-pre-processing"
Oct 31 02:12:48.841105 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:12:48.841010    4964 cpu_manager.go:414] "containerMap: removing container" podUID="1ec66b87-7d07-45fd-8469-e1bd2c83f651" containerName="istio-proxy"
Oct 31 02:12:48.841105 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:12:48.841017    4964 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1ec66b87-7d07-45fd-8469-e1bd2c83f651" containerName="istio-proxy"
Oct 31 02:12:48.841105 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:12:48.841025    4964 cpu_manager.go:414] "containerMap: removing container" podUID="3aae9913-7c43-45b0-87c8-bf1da6bf9183" containerName="payload-pre-processing"
Oct 31 02:12:48.841105 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:12:48.841030    4964 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3aae9913-7c43-45b0-87c8-bf1da6bf9183" containerName="payload-pre-processing"
Oct 31 02:12:48.841105 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:12:48.841037    4964 cpu_manager.go:414] "containerMap: removing container" podUID="803fb9d4-2f24-48f3-bea0-f62833d91796" containerName="maas-api"
Oct 31 02:12:48.841105 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:12:48.841041    4964 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="803fb9d4-2f24-48f3-bea0-f62833d91796" containerName="maas-api"
Oct 31 02:12:48.841105 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:12:48.841049    4964 cpu_manager.go:414] "containerMap: removing container" podUID="d2bef5ff-d60c-4d04-ac18-dd98c3f92457" containerName="payload-pre-processing"
Oct 31 02:12:48.841105 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:12:48.841053    4964 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d2bef5ff-d60c-4d04-ac18-dd98c3f92457" containerName="payload-pre-processing"
Oct 31 02:12:48.841105 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:12:48.841061    4964 cpu_manager.go:414] "containerMap: removing container" podUID="1ec66b87-7d07-45fd-8469-e1bd2c83f651" containerName="istio-proxy"
Oct 31 02:12:48.841105 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:12:48.841065    4964 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1ec66b87-7d07-45fd-8469-e1bd2c83f651" containerName="istio-proxy"
Oct 31 02:12:48.841105 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:12:48.841071    4964 cpu_manager.go:414] "containerMap: removing container" podUID="bc7a71df-c523-4590-91df-cf8fdf3ed19e" containerName="main"
Oct 31 02:12:48.841105 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:12:48.841076    4964 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bc7a71df-c523-4590-91df-cf8fdf3ed19e" containerName="main"
Oct 31 02:12:48.841105 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:12:48.841082    4964 cpu_manager.go:414] "containerMap: removing container" podUID="597e819f-88c8-4061-8c6a-9adf9a909d30" containerName="istio-proxy"
Oct 31 02:12:48.841105 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:12:48.841086    4964 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="597e819f-88c8-4061-8c6a-9adf9a909d30" containerName="istio-proxy"
Oct 31 02:12:48.841105 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:12:48.841091    4964 cpu_manager.go:414] "containerMap: removing container" podUID="c7eddf8c-a855-4a72-b3ed-eacf3aaed5af" containerName="istio-proxy"
Oct 31 02:12:48.841105 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:12:48.841095    4964 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c7eddf8c-a855-4a72-b3ed-eacf3aaed5af" containerName="istio-proxy"
Oct 31 02:12:48.841105 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:12:48.841103    4964 cpu_manager.go:414] "containerMap: removing container" podUID="fd5d3644-7596-44e1-844f-fdc3c68b5021" containerName="istio-proxy"
Oct 31 02:12:48.841105 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:12:48.841107    4964 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fd5d3644-7596-44e1-844f-fdc3c68b5021" containerName="istio-proxy"
Oct 31 02:12:48.841835 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:12:48.841119    4964 cpu_manager.go:414] "containerMap: removing container" podUID="04c0b568-8251-4888-902b-e46c9483d931" containerName="payload-processing"
Oct 31 02:12:48.841835 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:12:48.841123    4964 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="04c0b568-8251-4888-902b-e46c9483d931" containerName="payload-processing"
Oct 31 02:12:48.841835 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:12:48.841130    4964 cpu_manager.go:414] "containerMap: removing container" podUID="facb04d4-8cd2-48aa-9b75-f165d7ca3f4a" containerName="main"
Oct 31 02:12:48.841835 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:12:48.841135    4964 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="facb04d4-8cd2-48aa-9b75-f165d7ca3f4a" containerName="main"
Oct 31 02:12:48.841835 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:12:48.841141    4964 cpu_manager.go:414] "containerMap: removing container" podUID="750dc732-c296-44cb-9f7f-a7f7be7b4144" containerName="maas-api"
Oct 31 02:12:48.841835 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:12:48.841146    4964 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="750dc732-c296-44cb-9f7f-a7f7be7b4144" containerName="maas-api"
Oct 31 02:12:48.841835 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:12:48.841151    4964 cpu_manager.go:414] "containerMap: removing container" podUID="e8e369b7-335b-4ac4-8e8a-3879cf430b76" containerName="maas-api"
Oct 31 02:12:48.841835 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:12:48.841155    4964 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e8e369b7-335b-4ac4-8e8a-3879cf430b76" containerName="maas-api"
Oct 31 02:12:48.841835 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:12:48.841160    4964 cpu_manager.go:414] "containerMap: removing container" podUID="081cab7b-aa01-4d09-b244-92855e9b0249" containerName="maas-api"
Oct 31 02:12:48.841835 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:12:48.841163    4964 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="081cab7b-aa01-4d09-b244-92855e9b0249" containerName="maas-api"
Oct 31 02:12:48.841835 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:12:48.841171    4964 cpu_manager.go:414] "containerMap: removing container" podUID="8508d596-d89b-4f67-9a57-f6ba4890ce5e" containerName="istio-proxy"
Oct 31 02:12:48.841835 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:12:48.841175    4964 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8508d596-d89b-4f67-9a57-f6ba4890ce5e" containerName="istio-proxy"
Oct 31 02:12:48.841835 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:12:48.841181    4964 cpu_manager.go:414] "containerMap: removing container" podUID="df6d0bdd-382d-49b7-879c-15cb5371246e" containerName="main"
Oct 31 02:12:48.841835 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:12:48.841186    4964 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="df6d0bdd-382d-49b7-879c-15cb5371246e" containerName="main"
Oct 31 02:12:48.841835 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:12:48.841193    4964 cpu_manager.go:414] "containerMap: removing container" podUID="4b0fd818-20c2-4dd6-9846-f361c236dbd9" containerName="main"
Oct 31 02:12:48.841835 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:12:48.841197    4964 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4b0fd818-20c2-4dd6-9846-f361c236dbd9" containerName="main"
Oct 31 02:12:48.841835 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:12:48.841204    4964 cpu_manager.go:414] "containerMap: removing container" podUID="1898febe-c54e-4e24-a8d9-7f7ed5fe5ada" containerName="maas-api"
Oct 31 02:12:48.841835 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:12:48.841208    4964 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1898febe-c54e-4e24-a8d9-7f7ed5fe5ada" containerName="maas-api"
Oct 31 02:12:48.841835 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:12:48.841214    4964 cpu_manager.go:414] "containerMap: removing container" podUID="15028ba0-c888-4780-b770-6acb2d03d70a" containerName="main"
Oct 31 02:12:48.841835 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:12:48.841217    4964 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="15028ba0-c888-4780-b770-6acb2d03d70a" containerName="main"
Oct 31 02:12:48.841835 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:12:48.841224    4964 cpu_manager.go:414] "containerMap: removing container" podUID="5c156010-30a3-434b-a027-305788f7ed3b" containerName="main"
Oct 31 02:12:48.841835 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:12:48.841228    4964 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5c156010-30a3-434b-a027-305788f7ed3b" containerName="main"
Oct 31 02:12:48.841835 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:12:48.841234    4964 cpu_manager.go:414] "containerMap: removing container" podUID="290631aa-9208-4dbb-8a0c-aa79cea9e6f8" containerName="maas-api"
Oct 31 02:12:48.841835 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:12:48.841239    4964 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="290631aa-9208-4dbb-8a0c-aa79cea9e6f8" containerName="maas-api"
Oct 31 02:12:48.841835 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:12:48.841250    4964 cpu_manager.go:414] "containerMap: removing container" podUID="26611751-3a0b-488f-a67e-5a79bb448cbb" containerName="maas-api"
Oct 31 02:12:48.841835 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:12:48.841255    4964 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="26611751-3a0b-488f-a67e-5a79bb448cbb" containerName="maas-api"
Oct 31 02:12:48.841835 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:12:48.841265    4964 cpu_manager.go:414] "containerMap: removing container" podUID="7c420de6-67b9-4958-8083-ebd8bd939418" containerName="maas-api"
Oct 31 02:12:48.841835 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:12:48.841269    4964 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7c420de6-67b9-4958-8083-ebd8bd939418" containerName="maas-api"
Oct 31 02:12:48.841835 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:12:48.841279    4964 cpu_manager.go:414] "containerMap: removing container" podUID="b8c12d5f-8ada-4c53-880c-446f5f0b5998" containerName="main"
Oct 31 02:12:48.841835 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:12:48.841285    4964 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b8c12d5f-8ada-4c53-880c-446f5f0b5998" containerName="main"
Oct 31 02:12:48.841835 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:12:48.841495    4964 memory_manager.go:365] "RemoveStaleState removing state" podUID="26611751-3a0b-488f-a67e-5a79bb448cbb" containerName="maas-api"
Oct 31 02:12:48.841835 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:12:48.841516    4964 memory_manager.go:365] "RemoveStaleState removing state" podUID="7c420de6-67b9-4958-8083-ebd8bd939418" containerName="maas-api"
Oct 31 02:12:48.841835 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:12:48.841525    4964 memory_manager.go:365] "RemoveStaleState removing state" podUID="b8c12d5f-8ada-4c53-880c-446f5f0b5998" containerName="main"
Oct 31 02:12:48.841835 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:12:48.841537    4964 memory_manager.go:365] "RemoveStaleState removing state" podUID="c7eddf8c-a855-4a72-b3ed-eacf3aaed5af" containerName="istio-proxy"
Oct 31 02:12:48.841835 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:12:48.841544    4964 memory_manager.go:365] "RemoveStaleState removing state" podUID="ba3f39ff-dde5-4b50-b624-d3144586f11f" containerName="payload-pre-processing"
Oct 31 02:12:48.841835 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:12:48.841554    4964 memory_manager.go:365] "RemoveStaleState removing state" podUID="1ec66b87-7d07-45fd-8469-e1bd2c83f651" containerName="istio-proxy"
Oct 31 02:12:48.841835 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:12:48.841564    4964 memory_manager.go:365] "RemoveStaleState removing state" podUID="3aae9913-7c43-45b0-87c8-bf1da6bf9183" containerName="payload-pre-processing"
Oct 31 02:12:48.841835 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:12:48.841574    4964 memory_manager.go:365] "RemoveStaleState removing state" podUID="803fb9d4-2f24-48f3-bea0-f62833d91796" containerName="maas-api"
Oct 31 02:12:48.841835 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:12:48.841583    4964 memory_manager.go:365] "RemoveStaleState removing state" podUID="d2bef5ff-d60c-4d04-ac18-dd98c3f92457" containerName="payload-pre-processing"
Oct 31 02:12:48.841835 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:12:48.841591    4964 memory_manager.go:365] "RemoveStaleState removing state" podUID="1ec66b87-7d07-45fd-8469-e1bd2c83f651" containerName="istio-proxy"
Oct 31 02:12:48.841835 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:12:48.841595    4964 memory_manager.go:365] "RemoveStaleState removing state" podUID="bc7a71df-c523-4590-91df-cf8fdf3ed19e" containerName="main"
Oct 31 02:12:48.841835 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:12:48.841601    4964 memory_manager.go:365] "RemoveStaleState removing state" podUID="597e819f-88c8-4061-8c6a-9adf9a909d30" containerName="istio-proxy"
Oct 31 02:12:48.841835 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:12:48.841607    4964 memory_manager.go:365] "RemoveStaleState removing state" podUID="c7eddf8c-a855-4a72-b3ed-eacf3aaed5af" containerName="istio-proxy"
Oct 31 02:12:48.841835 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:12:48.841613    4964 memory_manager.go:365] "RemoveStaleState removing state" podUID="fd5d3644-7596-44e1-844f-fdc3c68b5021" containerName="istio-proxy"
Oct 31 02:12:48.841835 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:12:48.841622    4964 memory_manager.go:365] "RemoveStaleState removing state" podUID="04c0b568-8251-4888-902b-e46c9483d931" containerName="payload-processing"
Oct 31 02:12:48.841835 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:12:48.841628    4964 memory_manager.go:365] "RemoveStaleState removing state" podUID="facb04d4-8cd2-48aa-9b75-f165d7ca3f4a" containerName="main"
Oct 31 02:12:48.841835 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:12:48.841633    4964 memory_manager.go:365] "RemoveStaleState removing state" podUID="750dc732-c296-44cb-9f7f-a7f7be7b4144" containerName="maas-api"
Oct 31 02:12:48.841835 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:12:48.841639    4964 memory_manager.go:365] "RemoveStaleState removing state" podUID="e8e369b7-335b-4ac4-8e8a-3879cf430b76" containerName="maas-api"
Oct 31 02:12:48.841835 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:12:48.841646    4964 memory_manager.go:365] "RemoveStaleState removing state" podUID="081cab7b-aa01-4d09-b244-92855e9b0249" containerName="maas-api"
Oct 31 02:12:48.841835 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:12:48.841657    4964 memory_manager.go:365] "RemoveStaleState removing state" podUID="df6d0bdd-382d-49b7-879c-15cb5371246e" containerName="main"
Oct 31 02:12:48.841835 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:12:48.841666    4964 memory_manager.go:365] "RemoveStaleState removing state" podUID="4b0fd818-20c2-4dd6-9846-f361c236dbd9" containerName="main"
Oct 31 02:12:48.841835 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:12:48.841671    4964 memory_manager.go:365] "RemoveStaleState removing state" podUID="1898febe-c54e-4e24-a8d9-7f7ed5fe5ada" containerName="maas-api"
Oct 31 02:12:48.841835 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:12:48.841676    4964 memory_manager.go:365] "RemoveStaleState removing state" podUID="15028ba0-c888-4780-b770-6acb2d03d70a" containerName="main"
Oct 31 02:12:48.841835 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:12:48.841681    4964 memory_manager.go:365] "RemoveStaleState removing state" podUID="5c156010-30a3-434b-a027-305788f7ed3b" containerName="main"
Oct 31 02:12:48.841835 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:12:48.841687    4964 memory_manager.go:365] "RemoveStaleState removing state" podUID="290631aa-9208-4dbb-8a0c-aa79cea9e6f8" containerName="maas-api"
Oct 31 02:12:48.845176 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:12:48.845157    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-7lvdn/must-gather-b779n"
Oct 31 02:12:48.885241 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:12:48.885207    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 02:12:48.901856 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:12:48.901827    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-7lvdn/must-gather-b779n"]
Oct 31 02:12:48.948972 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:12:48.948920    4964 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-7lvdn\"/\"openshift-service-ca.crt\""
Oct 31 02:12:48.948972 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:12:48.948943    4964 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-7lvdn\"/\"kube-root-ca.crt\""
Oct 31 02:12:48.948972 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:12:48.948965    4964 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-7lvdn\"/\"default-dockercfg-trw8z\""
Oct 31 02:12:48.951182 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:12:48.951159    4964 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-frc7g/must-gather-lgmfm"]
Oct 31 02:12:48.954763 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:12:48.954744    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-frc7g/must-gather-lgmfm"
Oct 31 02:12:49.002899 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:12:49.002873    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nbm77\" (UniqueName: \"kubernetes.io/projected/de254cef-7fd4-43b7-a1a6-14ef5f72e542-kube-api-access-nbm77\") pod \"must-gather-b779n\" (UID: \"de254cef-7fd4-43b7-a1a6-14ef5f72e542\") " pod="openshift-must-gather-7lvdn/must-gather-b779n"
Oct 31 02:12:49.003032 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:12:49.002920    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-frc7g/must-gather-lgmfm"]
Oct 31 02:12:49.003032 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:12:49.002940    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/de254cef-7fd4-43b7-a1a6-14ef5f72e542-must-gather-output\") pod \"must-gather-b779n\" (UID: \"de254cef-7fd4-43b7-a1a6-14ef5f72e542\") " pod="openshift-must-gather-7lvdn/must-gather-b779n"
Oct 31 02:12:49.006332 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:12:49.006289    4964 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-frc7g\"/\"kube-root-ca.crt\""
Oct 31 02:12:49.006447 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:12:49.006342    4964 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-frc7g\"/\"openshift-service-ca.crt\""
Oct 31 02:12:49.006447 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:12:49.006389    4964 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-frc7g\"/\"default-dockercfg-lp57d\""
Oct 31 02:12:49.077588 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:12:49.077549    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 02:12:49.103945 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:12:49.103881    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/de254cef-7fd4-43b7-a1a6-14ef5f72e542-must-gather-output\") pod \"must-gather-b779n\" (UID: \"de254cef-7fd4-43b7-a1a6-14ef5f72e542\") " pod="openshift-must-gather-7lvdn/must-gather-b779n"
Oct 31 02:12:49.104079 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:12:49.103945    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d7q7z\" (UniqueName: \"kubernetes.io/projected/d9f54f91-fc93-4081-abcf-5b4bd3b51ccc-kube-api-access-d7q7z\") pod \"must-gather-lgmfm\" (UID: \"d9f54f91-fc93-4081-abcf-5b4bd3b51ccc\") " pod="openshift-must-gather-frc7g/must-gather-lgmfm"
Oct 31 02:12:49.104079 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:12:49.104015    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/d9f54f91-fc93-4081-abcf-5b4bd3b51ccc-must-gather-output\") pod \"must-gather-lgmfm\" (UID: \"d9f54f91-fc93-4081-abcf-5b4bd3b51ccc\") " pod="openshift-must-gather-frc7g/must-gather-lgmfm"
Oct 31 02:12:49.104079 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:12:49.104048    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nbm77\" (UniqueName: \"kubernetes.io/projected/de254cef-7fd4-43b7-a1a6-14ef5f72e542-kube-api-access-nbm77\") pod \"must-gather-b779n\" (UID: \"de254cef-7fd4-43b7-a1a6-14ef5f72e542\") " pod="openshift-must-gather-7lvdn/must-gather-b779n"
Oct 31 02:12:49.104226 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:12:49.104208    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/de254cef-7fd4-43b7-a1a6-14ef5f72e542-must-gather-output\") pod \"must-gather-b779n\" (UID: \"de254cef-7fd4-43b7-a1a6-14ef5f72e542\") " pod="openshift-must-gather-7lvdn/must-gather-b779n"
Oct 31 02:12:49.159953 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:12:49.159923    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nbm77\" (UniqueName: \"kubernetes.io/projected/de254cef-7fd4-43b7-a1a6-14ef5f72e542-kube-api-access-nbm77\") pod \"must-gather-b779n\" (UID: \"de254cef-7fd4-43b7-a1a6-14ef5f72e542\") " pod="openshift-must-gather-7lvdn/must-gather-b779n"
Oct 31 02:12:49.205150 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:12:49.205121    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d7q7z\" (UniqueName: \"kubernetes.io/projected/d9f54f91-fc93-4081-abcf-5b4bd3b51ccc-kube-api-access-d7q7z\") pod \"must-gather-lgmfm\" (UID: \"d9f54f91-fc93-4081-abcf-5b4bd3b51ccc\") " pod="openshift-must-gather-frc7g/must-gather-lgmfm"
Oct 31 02:12:49.205276 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:12:49.205159    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/d9f54f91-fc93-4081-abcf-5b4bd3b51ccc-must-gather-output\") pod \"must-gather-lgmfm\" (UID: \"d9f54f91-fc93-4081-abcf-5b4bd3b51ccc\") " pod="openshift-must-gather-frc7g/must-gather-lgmfm"
Oct 31 02:12:49.205503 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:12:49.205488    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/d9f54f91-fc93-4081-abcf-5b4bd3b51ccc-must-gather-output\") pod \"must-gather-lgmfm\" (UID: \"d9f54f91-fc93-4081-abcf-5b4bd3b51ccc\") " pod="openshift-must-gather-frc7g/must-gather-lgmfm"
Oct 31 02:12:49.262097 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:12:49.262071    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d7q7z\" (UniqueName: \"kubernetes.io/projected/d9f54f91-fc93-4081-abcf-5b4bd3b51ccc-kube-api-access-d7q7z\") pod \"must-gather-lgmfm\" (UID: \"d9f54f91-fc93-4081-abcf-5b4bd3b51ccc\") " pod="openshift-must-gather-frc7g/must-gather-lgmfm"
Oct 31 02:12:49.269003 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:12:49.268988    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-frc7g/must-gather-lgmfm"
Oct 31 02:12:49.459338 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:12:49.459241    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-7lvdn/must-gather-b779n"
Oct 31 02:12:49.644603 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:12:49.644579    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-frc7g/must-gather-lgmfm"]
Oct 31 02:12:49.646976 ip-10-0-133-197 kubenswrapper[4964]: W1031 02:12:49.646946    4964 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd9f54f91_fc93_4081_abcf_5b4bd3b51ccc.slice/crio-b73726ed19374f9c2031bdfcf4a2fa420ab9fd285798bd5c01c3404df1e813f1 WatchSource:0}: Error finding container b73726ed19374f9c2031bdfcf4a2fa420ab9fd285798bd5c01c3404df1e813f1: Status 404 returned error can't find the container with id b73726ed19374f9c2031bdfcf4a2fa420ab9fd285798bd5c01c3404df1e813f1
Oct 31 02:12:49.648834 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:12:49.648816    4964 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 31 02:12:49.833398 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:12:49.833360    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-7lvdn/must-gather-b779n"]
Oct 31 02:12:49.835951 ip-10-0-133-197 kubenswrapper[4964]: W1031 02:12:49.835917    4964 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podde254cef_7fd4_43b7_a1a6_14ef5f72e542.slice/crio-248b1ff4115273b10677d957967d17ba03b3915e4f530a2c56fd9ee95e1254f9 WatchSource:0}: Error finding container 248b1ff4115273b10677d957967d17ba03b3915e4f530a2c56fd9ee95e1254f9: Status 404 returned error can't find the container with id 248b1ff4115273b10677d957967d17ba03b3915e4f530a2c56fd9ee95e1254f9
Oct 31 02:12:50.023656 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:12:50.023612    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-7lvdn/must-gather-b779n" event={"ID":"de254cef-7fd4-43b7-a1a6-14ef5f72e542","Type":"ContainerStarted","Data":"248b1ff4115273b10677d957967d17ba03b3915e4f530a2c56fd9ee95e1254f9"}
Oct 31 02:12:50.024911 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:12:50.024883    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-frc7g/must-gather-lgmfm" event={"ID":"d9f54f91-fc93-4081-abcf-5b4bd3b51ccc","Type":"ContainerStarted","Data":"b73726ed19374f9c2031bdfcf4a2fa420ab9fd285798bd5c01c3404df1e813f1"}
Oct 31 02:12:51.031850 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:12:51.031817    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-frc7g/must-gather-lgmfm" event={"ID":"d9f54f91-fc93-4081-abcf-5b4bd3b51ccc","Type":"ContainerStarted","Data":"51dfbdc17e5d179797f288066282b9c384972c5ed47b4d8eda32ec610f2cfd18"}
Oct 31 02:12:52.053874 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:12:52.053832    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-frc7g/must-gather-lgmfm" event={"ID":"d9f54f91-fc93-4081-abcf-5b4bd3b51ccc","Type":"ContainerStarted","Data":"a7f30ff94118f61a2980b55a6e2fb5e9310eac48a9a94ce30d1083a6fa7a94e0"}
Oct 31 02:12:52.169676 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:12:52.169445    4964 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-frc7g/must-gather-lgmfm" podStartSLOduration=2.903485703 podStartE2EDuration="4.169426931s" totalImagesPullingTime="1.265941228s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 02:12:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 02:12:52.166952371 +0000 UTC m=+4140.643518793" watchObservedRunningTime="2026-10-31 02:12:52.169426931 +0000 UTC m=+4140.645993347"
Oct 31 02:12:57.097851 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:12:57.097809    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-7lvdn/must-gather-b779n" event={"ID":"de254cef-7fd4-43b7-a1a6-14ef5f72e542","Type":"ContainerStarted","Data":"828688601f6fcdfdb5f9f8f4b23a3118a6279d97ee6b76293f83e049a99e6b3b"}
Oct 31 02:12:57.098391 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:12:57.097861    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-7lvdn/must-gather-b779n" event={"ID":"de254cef-7fd4-43b7-a1a6-14ef5f72e542","Type":"ContainerStarted","Data":"928d0f7024ba7ef06f8419b206eee28d850a0aa265bf9d0326c47f8f4ddc1314"}
Oct 31 02:12:57.273398 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:12:57.273346    4964 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-7lvdn/must-gather-b779n" podStartSLOduration=2.940994861 podStartE2EDuration="9.273331256s" totalImagesPullingTime="6.332336395s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 02:12:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 02:12:57.270439282 +0000 UTC m=+4145.747005662" watchObservedRunningTime="2026-10-31 02:12:57.273331256 +0000 UTC m=+4145.749897626"
Oct 31 02:12:59.280651 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:12:59.280605    4964 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-zs5g7_aa4836a5-c71a-485d-82fc-f8b53dae7717/global-pull-secret-syncer/0.log"
Oct 31 02:12:59.849025 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:12:59.848996    4964 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-jm6bm_d719eda9-bc9f-4307-952a-99d99aa0f62b/connection-checker/0.log"
Oct 31 02:13:00.264671 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:13:00.264644    4964 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-q4z5f_0ab3954e-be64-4260-bca8-ff5d58bd6ea2/connection-checker/0.log"
Oct 31 02:13:00.360085 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:13:00.360052    4964 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-58f5dc44b9-c98xl_cf528aec-95b1-46dd-8f6c-da3ac6cc85d9/manager/0.log"
Oct 31 02:13:00.680040 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:13:00.679968    4964 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-vbbq6_da8395ef-7464-410e-96ee-b27378a03fed/connection-checker/0.log"
Oct 31 02:13:01.005053 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:13:01.005014    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 02:13:01.130688 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:13:01.130649    4964 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-58f5dc44b9-c98xl_cf528aec-95b1-46dd-8f6c-da3ac6cc85d9/manager/0.log"
Oct 31 02:13:01.169161 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:13:01.169126    4964 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log"
Oct 31 02:13:02.366257 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:13:02.366223    4964 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-133-197.us-west-1.compute.internal_8dea7006d41de627c8e06bf18f518df6/haproxy/0.log"
Oct 31 02:13:03.431674 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:13:03.431637    4964 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-z94np_baeff60c-ce67-45a0-a76b-52220ba1065f/manager/0.log"
Oct 31 02:13:04.083796 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:13:04.083759    4964 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-z94np_baeff60c-ce67-45a0-a76b-52220ba1065f/manager/0.log"
Oct 31 02:13:06.582398 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:13:06.582370    4964 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-58f5dc44b9-c98xl_cf528aec-95b1-46dd-8f6c-da3ac6cc85d9/manager/0.log"
Oct 31 02:13:07.032858 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:13:07.032825    4964 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-58f5dc44b9-c98xl_cf528aec-95b1-46dd-8f6c-da3ac6cc85d9/manager/0.log"
Oct 31 02:13:11.337615 ip-10-0-133-197 kubenswrapper[4964]: E1031 02:13:11.337496    4964 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q56x5" containerName="extractor"
Oct 31 02:13:11.453412 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:13:11.453376    4964 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-z94np_baeff60c-ce67-45a0-a76b-52220ba1065f/manager/0.log"
Oct 31 02:13:11.789741 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:13:11.787018    4964 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-z94np_baeff60c-ce67-45a0-a76b-52220ba1065f/manager/0.log"
Oct 31 02:13:14.781393 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:13:14.781348    4964 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-58f5dc44b9-c98xl_cf528aec-95b1-46dd-8f6c-da3ac6cc85d9/manager/0.log"
Oct 31 02:13:14.937869 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:13:14.937827    4964 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-58f5dc44b9-c98xl_cf528aec-95b1-46dd-8f6c-da3ac6cc85d9/manager/0.log"
Oct 31 02:13:18.350476 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:13:18.350434    4964 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-z94np_baeff60c-ce67-45a0-a76b-52220ba1065f/manager/0.log"
Oct 31 02:13:18.476253 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:13:18.476212    4964 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-z94np_baeff60c-ce67-45a0-a76b-52220ba1065f/manager/0.log"
Oct 31 02:13:19.318180 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:13:19.318147    4964 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-2nnh7_9b628e58-c287-4716-adbb-6bbdf0dad9fb/manager/0.log"
Oct 31 02:13:20.784267 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:13:20.784225    4964 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-9mrpk_0446b6be-06eb-4fde-858b-a36f492f2209/manager/0.log"
Oct 31 02:13:21.586745 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:13:21.586702    4964 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-svrmp_25e6b2b1-a331-483e-86e8-8585418a8e37/manager/0.log"
Oct 31 02:13:21.791837 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:13:21.791802    4964 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6b748b6585wg7kb_5b65b0b2-79a7-4116-a40e-78e1414c0071/istio-proxy/0.log"
Oct 31 02:13:22.071602 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:13:22.071565    4964 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6b748b6585wg7kb_5b65b0b2-79a7-4116-a40e-78e1414c0071/istio-proxy/0.log"
Oct 31 02:13:23.276861 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:13:23.276821    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 02:13:23.830165 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:13:23.830131    4964 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-tlb5m_c8461fdb-12e5-4691-8b95-139d5a901f0d/payload-processing/0.log"
Oct 31 02:13:24.052149 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:13:24.052118    4964 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-5964fff8f9-8k6pp_e49453a8-c7d9-4248-a8ef-4a819b76806c/router/0.log"
Oct 31 02:13:24.106080 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:13:24.105950    4964 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-tlb5m_c8461fdb-12e5-4691-8b95-139d5a901f0d/payload-processing/0.log"
Oct 31 02:13:24.321471 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:13:24.321062    4964 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-5964fff8f9-8k6pp_e49453a8-c7d9-4248-a8ef-4a819b76806c/router/0.log"
Oct 31 02:13:27.170466 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:13:27.170430    4964 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-884576c57-jccjk_b66619db-fd81-4435-be2a-a5ea6111e204/cluster-monitoring-operator/0.log"
Oct 31 02:13:28.265977 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:13:28.265932    4964 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-frc7g/perf-node-gather-daemonset-d965z"]
Oct 31 02:13:28.270490 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:13:28.270156    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-frc7g/perf-node-gather-daemonset-d965z"
Oct 31 02:13:28.327938 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:13:28.327903    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-frc7g/perf-node-gather-daemonset-d965z"]
Oct 31 02:13:28.376056 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:13:28.376024    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v5pzr\" (UniqueName: \"kubernetes.io/projected/a0aa27f2-0d42-4bd3-abd4-5b49ecb5412c-kube-api-access-v5pzr\") pod \"perf-node-gather-daemonset-d965z\" (UID: \"a0aa27f2-0d42-4bd3-abd4-5b49ecb5412c\") " pod="openshift-must-gather-frc7g/perf-node-gather-daemonset-d965z"
Oct 31 02:13:28.376241 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:13:28.376102    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a0aa27f2-0d42-4bd3-abd4-5b49ecb5412c-sys\") pod \"perf-node-gather-daemonset-d965z\" (UID: \"a0aa27f2-0d42-4bd3-abd4-5b49ecb5412c\") " pod="openshift-must-gather-frc7g/perf-node-gather-daemonset-d965z"
Oct 31 02:13:28.376241 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:13:28.376187    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/a0aa27f2-0d42-4bd3-abd4-5b49ecb5412c-proc\") pod \"perf-node-gather-daemonset-d965z\" (UID: \"a0aa27f2-0d42-4bd3-abd4-5b49ecb5412c\") " pod="openshift-must-gather-frc7g/perf-node-gather-daemonset-d965z"
Oct 31 02:13:28.376241 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:13:28.376219    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a0aa27f2-0d42-4bd3-abd4-5b49ecb5412c-lib-modules\") pod \"perf-node-gather-daemonset-d965z\" (UID: \"a0aa27f2-0d42-4bd3-abd4-5b49ecb5412c\") " pod="openshift-must-gather-frc7g/perf-node-gather-daemonset-d965z"
Oct 31 02:13:28.376444 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:13:28.376380    4964 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/a0aa27f2-0d42-4bd3-abd4-5b49ecb5412c-podres\") pod \"perf-node-gather-daemonset-d965z\" (UID: \"a0aa27f2-0d42-4bd3-abd4-5b49ecb5412c\") " pod="openshift-must-gather-frc7g/perf-node-gather-daemonset-d965z"
Oct 31 02:13:28.477284 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:13:28.477239    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a0aa27f2-0d42-4bd3-abd4-5b49ecb5412c-lib-modules\") pod \"perf-node-gather-daemonset-d965z\" (UID: \"a0aa27f2-0d42-4bd3-abd4-5b49ecb5412c\") " pod="openshift-must-gather-frc7g/perf-node-gather-daemonset-d965z"
Oct 31 02:13:28.477542 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:13:28.477305    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/a0aa27f2-0d42-4bd3-abd4-5b49ecb5412c-podres\") pod \"perf-node-gather-daemonset-d965z\" (UID: \"a0aa27f2-0d42-4bd3-abd4-5b49ecb5412c\") " pod="openshift-must-gather-frc7g/perf-node-gather-daemonset-d965z"
Oct 31 02:13:28.477542 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:13:28.477396    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a0aa27f2-0d42-4bd3-abd4-5b49ecb5412c-lib-modules\") pod \"perf-node-gather-daemonset-d965z\" (UID: \"a0aa27f2-0d42-4bd3-abd4-5b49ecb5412c\") " pod="openshift-must-gather-frc7g/perf-node-gather-daemonset-d965z"
Oct 31 02:13:28.477542 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:13:28.477436    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v5pzr\" (UniqueName: \"kubernetes.io/projected/a0aa27f2-0d42-4bd3-abd4-5b49ecb5412c-kube-api-access-v5pzr\") pod \"perf-node-gather-daemonset-d965z\" (UID: \"a0aa27f2-0d42-4bd3-abd4-5b49ecb5412c\") " pod="openshift-must-gather-frc7g/perf-node-gather-daemonset-d965z"
Oct 31 02:13:28.477542 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:13:28.477450    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/a0aa27f2-0d42-4bd3-abd4-5b49ecb5412c-podres\") pod \"perf-node-gather-daemonset-d965z\" (UID: \"a0aa27f2-0d42-4bd3-abd4-5b49ecb5412c\") " pod="openshift-must-gather-frc7g/perf-node-gather-daemonset-d965z"
Oct 31 02:13:28.477542 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:13:28.477530    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a0aa27f2-0d42-4bd3-abd4-5b49ecb5412c-sys\") pod \"perf-node-gather-daemonset-d965z\" (UID: \"a0aa27f2-0d42-4bd3-abd4-5b49ecb5412c\") " pod="openshift-must-gather-frc7g/perf-node-gather-daemonset-d965z"
Oct 31 02:13:28.477829 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:13:28.477562    4964 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/a0aa27f2-0d42-4bd3-abd4-5b49ecb5412c-proc\") pod \"perf-node-gather-daemonset-d965z\" (UID: \"a0aa27f2-0d42-4bd3-abd4-5b49ecb5412c\") " pod="openshift-must-gather-frc7g/perf-node-gather-daemonset-d965z"
Oct 31 02:13:28.477829 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:13:28.477652    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a0aa27f2-0d42-4bd3-abd4-5b49ecb5412c-sys\") pod \"perf-node-gather-daemonset-d965z\" (UID: \"a0aa27f2-0d42-4bd3-abd4-5b49ecb5412c\") " pod="openshift-must-gather-frc7g/perf-node-gather-daemonset-d965z"
Oct 31 02:13:28.477829 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:13:28.477657    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/a0aa27f2-0d42-4bd3-abd4-5b49ecb5412c-proc\") pod \"perf-node-gather-daemonset-d965z\" (UID: \"a0aa27f2-0d42-4bd3-abd4-5b49ecb5412c\") " pod="openshift-must-gather-frc7g/perf-node-gather-daemonset-d965z"
Oct 31 02:13:28.534491 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:13:28.534409    4964 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v5pzr\" (UniqueName: \"kubernetes.io/projected/a0aa27f2-0d42-4bd3-abd4-5b49ecb5412c-kube-api-access-v5pzr\") pod \"perf-node-gather-daemonset-d965z\" (UID: \"a0aa27f2-0d42-4bd3-abd4-5b49ecb5412c\") " pod="openshift-must-gather-frc7g/perf-node-gather-daemonset-d965z"
Oct 31 02:13:28.584525 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:13:28.584491    4964 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-frc7g/perf-node-gather-daemonset-d965z"
Oct 31 02:13:28.968453 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:13:28.968367    4964 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-frc7g/perf-node-gather-daemonset-d965z"]
Oct 31 02:13:28.975222 ip-10-0-133-197 kubenswrapper[4964]: W1031 02:13:28.975178    4964 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-poda0aa27f2_0d42_4bd3_abd4_5b49ecb5412c.slice/crio-7afc99d0bbec50163090291f151afbc0ff97d93d94e1c3fced9e1fc8738d577b WatchSource:0}: Error finding container 7afc99d0bbec50163090291f151afbc0ff97d93d94e1c3fced9e1fc8738d577b: Status 404 returned error can't find the container with id 7afc99d0bbec50163090291f151afbc0ff97d93d94e1c3fced9e1fc8738d577b
Oct 31 02:13:29.356617 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:13:29.356576    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-frc7g/perf-node-gather-daemonset-d965z" event={"ID":"a0aa27f2-0d42-4bd3-abd4-5b49ecb5412c","Type":"ContainerStarted","Data":"7204390c498988d2dc0a8c68b907af8e76e739a1da6b2247fe7644b60e2f7d32"}
Oct 31 02:13:29.357073 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:13:29.356651    4964 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-frc7g/perf-node-gather-daemonset-d965z" event={"ID":"a0aa27f2-0d42-4bd3-abd4-5b49ecb5412c","Type":"ContainerStarted","Data":"7afc99d0bbec50163090291f151afbc0ff97d93d94e1c3fced9e1fc8738d577b"}
Oct 31 02:13:29.357073 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:13:29.356689    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-frc7g/perf-node-gather-daemonset-d965z"
Oct 31 02:13:29.474168 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:13:29.474117    4964 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-frc7g/perf-node-gather-daemonset-d965z" podStartSLOduration=1.474100672 podStartE2EDuration="1.474100672s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 02:13:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 02:13:29.471439285 +0000 UTC m=+4177.948005665" watchObservedRunningTime="2026-10-31 02:13:29.474100672 +0000 UTC m=+4177.950667054"
Oct 31 02:13:29.583367 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:13:29.583307    4964 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-2nnh7_9b628e58-c287-4716-adbb-6bbdf0dad9fb/manager/0.log"
Oct 31 02:13:29.999532 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:13:29.999493    4964 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-2nnh7_9b628e58-c287-4716-adbb-6bbdf0dad9fb/manager/0.log"
Oct 31 02:13:30.688019 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:13:30.687978    4964 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-9mrpk_0446b6be-06eb-4fde-858b-a36f492f2209/manager/0.log"
Oct 31 02:13:31.003822 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:13:31.003783    4964 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-kwtlx_24af6b7c-9cc3-43e6-be11-192cd1a3678d/node-exporter/0.log"
Oct 31 02:13:31.184225 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:13:31.184189    4964 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-9mrpk_0446b6be-06eb-4fde-858b-a36f492f2209/manager/0.log"
Oct 31 02:13:31.196583 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:13:31.196547    4964 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-svrmp_25e6b2b1-a331-483e-86e8-8585418a8e37/manager/0.log"
Oct 31 02:13:31.441382 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:13:31.441286    4964 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-kwtlx_24af6b7c-9cc3-43e6-be11-192cd1a3678d/kube-rbac-proxy/0.log"
Oct 31 02:13:31.586913 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:13:31.586872    4964 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-svrmp_25e6b2b1-a331-483e-86e8-8585418a8e37/manager/0.log"
Oct 31 02:13:31.856665 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:13:31.856636    4964 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-kwtlx_24af6b7c-9cc3-43e6-be11-192cd1a3678d/init-textfile/0.log"
Oct 31 02:13:35.381828 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:13:35.381796    4964 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-frc7g/perf-node-gather-daemonset-d965z"
Oct 31 02:13:38.703592 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:13:38.703557    4964 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-6489d8bf89-xwkdt_b6953f8d-148b-4079-b4fa-365d83084e44/prometheus-operator-admission-webhook/0.log"
Oct 31 02:13:41.242538 ip-10-0-133-197 kubenswrapper[4964]: E1031 02:13:41.242502    4964 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q56x5" containerName="extractor"
Oct 31 02:13:45.373006 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:13:45.372973    4964 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-86c478bb88-wm4qt_9dfae6d6-b9a6-4d85-b594-9d947a9e6383/networking-console-plugin/0.log"
Oct 31 02:13:46.334632 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:13:46.334597    4964 log.go:25] "Finished parsing log file" path="/var/log/pods/cert-manager_cert-manager-cainjector-5c57bbd4fc-4mfd5_2ddf195b-447a-460c-8b02-ca432fda1fbc/cert-manager-cainjector/0.log"
Oct 31 02:13:47.211479 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:13:47.211448    4964 log.go:25] "Finished parsing log file" path="/var/log/pods/cert-manager_cert-manager-cainjector-5c57bbd4fc-4mfd5_2ddf195b-447a-460c-8b02-ca432fda1fbc/cert-manager-cainjector/0.log"
Oct 31 02:13:48.080273 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:13:48.080241    4964 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-9257n_f0c0ada7-7688-4dc2-97fc-813e4d0c4ef3/console-operator/2.log"
Oct 31 02:13:48.087886 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:13:48.087860    4964 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-9257n_f0c0ada7-7688-4dc2-97fc-813e4d0c4ef3/console-operator/3.log"
Oct 31 02:13:49.620947 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:13:49.620915    4964 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub-operator-system_opendatahub-operator-controller-manager-559fd7ff56-m5c2x_5cfd9ec8-fd50-421c-8744-18c0b03903d6/manager/0.log"
Oct 31 02:13:49.827845 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:13:49.827812    4964 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub-operator-system_opendatahub-operator-controller-manager-559fd7ff56-m5c2x_5cfd9ec8-fd50-421c-8744-18c0b03903d6/copy-manifests/0.log"
Oct 31 02:13:50.511392 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:13:50.511357    4964 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub-operator-system_opendatahub-operator-controller-manager-559fd7ff56-m5c2x_5cfd9ec8-fd50-421c-8744-18c0b03903d6/manager/0.log"
Oct 31 02:13:50.718623 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:13:50.718596    4964 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub-operator-system_opendatahub-operator-controller-manager-559fd7ff56-m5c2x_5cfd9ec8-fd50-421c-8744-18c0b03903d6/copy-manifests/0.log"
Oct 31 02:13:51.965480 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:13:51.965417    4964 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-58f5dc44b9-c98xl_cf528aec-95b1-46dd-8f6c-da3ac6cc85d9/manager/0.log"
Oct 31 02:13:52.416840 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:13:52.416811    4964 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-9257n_f0c0ada7-7688-4dc2-97fc-813e4d0c4ef3/console-operator/2.log"
Oct 31 02:13:52.434974 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:13:52.434947    4964 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-9257n_f0c0ada7-7688-4dc2-97fc-813e4d0c4ef3/console-operator/2.log"
Oct 31 02:13:52.939075 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:13:52.939040    4964 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-58f5dc44b9-c98xl_cf528aec-95b1-46dd-8f6c-da3ac6cc85d9/manager/0.log"
Oct 31 02:13:53.556944 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:13:53.556913    4964 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-58c9f5f46d-v585z_eafc69c5-15f1-4e4c-97ca-be586e55fbae/volume-data-source-validator/0.log"
Oct 31 02:13:54.956461 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:13:54.956403    4964 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-z94np_baeff60c-ce67-45a0-a76b-52220ba1065f/manager/0.log"
Oct 31 02:13:55.858257 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:13:55.858204    4964 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-z94np_baeff60c-ce67-45a0-a76b-52220ba1065f/manager/0.log"
Oct 31 02:13:57.318780 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:13:57.318751    4964 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6b748b6585wg7kb_5b65b0b2-79a7-4116-a40e-78e1414c0071/istio-proxy/0.log"
Oct 31 02:13:58.279144 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:13:58.279111    4964 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6b748b6585wg7kb_5b65b0b2-79a7-4116-a40e-78e1414c0071/istio-proxy/0.log"
Oct 31 02:13:59.255007 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:13:59.254961    4964 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-tlb5m_c8461fdb-12e5-4691-8b95-139d5a901f0d/payload-processing/0.log"
Oct 31 02:13:59.473386 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:13:59.473328    4964 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-5964fff8f9-8k6pp_e49453a8-c7d9-4248-a8ef-4a819b76806c/router/0.log"
Oct 31 02:13:59.968530 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:13:59.968499    4964 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-t7r6c_85bb69db-c69b-467e-adeb-950bf2947c94/dns/0.log"
Oct 31 02:14:00.166228 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:14:00.166195    4964 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-tlb5m_c8461fdb-12e5-4691-8b95-139d5a901f0d/payload-processing/0.log"
Oct 31 02:14:00.378085 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:14:00.378051    4964 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-t7r6c_85bb69db-c69b-467e-adeb-950bf2947c94/kube-rbac-proxy/0.log"
Oct 31 02:14:00.385354 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:14:00.385302    4964 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-5964fff8f9-8k6pp_e49453a8-c7d9-4248-a8ef-4a819b76806c/router/0.log"
Oct 31 02:14:01.639710 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:14:01.639680    4964 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-j46hv_57c2f196-0d8a-4bce-a5bc-3b6b5c0d808c/dns-node-resolver/0.log"
Oct 31 02:14:02.796272 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:14:02.796244    4964 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-jobset-operator_jobset-controller-manager-66db79789b-2jk8v_ec0a1197-55db-4519-9e0e-670c36ff799c/manager/0.log"
Oct 31 02:14:03.760712 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:14:03.760667    4964 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-jobset-operator_jobset-controller-manager-66db79789b-2jk8v_ec0a1197-55db-4519-9e0e-670c36ff799c/manager/0.log"
Oct 31 02:14:05.252593 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:14:05.252559    4964 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-6464b7c4f6-dfbph_e5089828-1c26-424e-b51b-d62ca3490b07/registry/0.log"
Oct 31 02:14:06.121715 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:14:06.121680    4964 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-rlxt2_dd8134f9-b2a6-4542-82ec-c10edc3fcd72/node-ca/0.log"
Oct 31 02:14:11.144194 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:14:11.144156    4964 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6b748b6585wg7kb_5b65b0b2-79a7-4116-a40e-78e1414c0071/istio-proxy/0.log"
Oct 31 02:14:11.237930 ip-10-0-133-197 kubenswrapper[4964]: E1031 02:14:11.237895    4964 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q56x5" containerName="extractor"
Oct 31 02:14:13.877667 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:14:13.877628    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 02:14:15.338391 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:14:15.338349    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024
Oct 31 02:14:15.354625 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:14:15.354595    4964 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-tlb5m_c8461fdb-12e5-4691-8b95-139d5a901f0d/payload-processing/0.log"
Oct 31 02:14:15.779868 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:14:15.779833    4964 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-5964fff8f9-8k6pp_e49453a8-c7d9-4248-a8ef-4a819b76806c/router/0.log"
Oct 31 02:14:19.556850 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:14:19.556816    4964 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-7dpwv_147bb9af-dfc7-4785-bb5e-906f25d65dd1/serve-healthcheck-canary/0.log"
Oct 31 02:14:22.906457 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:14:22.904407    4964 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-7c4986d449-jkk5x_c8e98df5-b2f7-49dd-a565-ed60534da496/insights-operator/0.log"
Oct 31 02:14:22.906983 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:14:22.906913    4964 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-7c4986d449-jkk5x_c8e98df5-b2f7-49dd-a565-ed60534da496/insights-operator/1.log"
Oct 31 02:14:23.141493 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:14:23.141458    4964 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6b748b6585wg7kb_5b65b0b2-79a7-4116-a40e-78e1414c0071/istio-proxy/0.log"
Oct 31 02:14:23.608249 ip-10-0-133-197 kubenswrapper[4964]: I1031 02:14:23.608121    4964 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78szg_f8855c2c-87b4-4b9e-b28a-227981205b11/konnectivity-agent/0.log" limit=1024